Skip to contentSkip to conversation

add zstd cluster compression

PATCH Superseded Denis Plotnikov 4 participants 14 messages Jul 3, 2019 – Jul 9, 2019 Block layer

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

cover add zstd cluster compression

change log:

v1:
* extend qcow2 header instead of adding a new incompatible extension header specification re-written accordingly
* enable zstd compression via config
* fix zstd (de)compression functions
* fix comments/description
* fix function naming

---
The goal of the patch-set is to enable qcow2 to use zstd compression for clusters. ZSTD provides better (de)compression performance than currently used ZLIB. Using it will improve perforamnce (reduce compression time) when the compressed clusters is used, e.g backup scenarios.

Also, the patch-set extends qcow2 specification by adding compression_type feature. The feature enables adding ZSTD and another compression algorithms in the future.

Here is some measurements ZSTD vs ZLIB:

The test:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

The results:
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0

Both ZLIB and ZSTD gave the same compression ratio: ~1.5 compressed image size in both cases: ~1.4G

Denis Plotnikov (3):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression

 block/qcow2.c             | 283 +++++++++++++++++++++++++++++++++++---
 block/qcow2.h             |  26 +++-
 configure                 |  32 +++++
 docs/interop/qcow2.txt    |  22 ++-
 include/block/block_int.h |   1 +
 qapi/block-core.json      |  23 +++-
 6 files changed, 357 insertions(+), 30 deletions(-)
Signature · 1 line
2.17.0
#1 lore raw

PATCH 1/3 qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly x2 faster than ZLIB providing a comparable compression ratio and therefore provide a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type is backward compatible with older qemu versions.

---
 block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
 block/qcow2.h             | 26 ++++++++---
 docs/interop/qcow2.txt    | 22 ++++++++-
 include/block/block_int.h |  1 +
 qapi/block-core.json      | 22 ++++++++-
 5 files changed, 155 insertions(+), 10 deletions(-)
block/qcow2.c +94 -0
index 3ace3b2209..921eb67b80 100644
@@ -1202,6 +1202,47 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
     return ret;
 }
 
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    bool is_set;
+    int ret = 0;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        if (errp) {
+            error_setg(errp, "qcow2: unknown compression type: %u",
+                       s->compression_type);
+            return -ENOTSUP;
+        }
+    }
+
+    /*
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set
+     */
+    is_set = s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE;
+
+    if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+        if (is_set) {
+            ret = -EINVAL;
+        }
+    } else {
+        if (!is_set) {
+            ret = -EINVAL;
+        }
+    }
+
+    if (ret && errp) {
+        error_setg(errp, "qcow2: Illegal compression type setting");
+    }
+
+    return ret;
+}
+
 /* Called with s->lock held.  */
 static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                       int flags, Error **errp)
@@ -1318,6 +1359,24 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
     s->compatible_features      = header.compatible_features;
     s->autoclear_features       = header.autoclear_features;
 
+    /* Handle compression type */
+    if (header.header_length > 104) {
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        /*
+         * older qcow2 images don't contain the compression type header
+         * field, distinguish them by the header length and use
+         * the only valid compression type in that case
+         */
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
     if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
         void *feature_table = NULL;
         qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2434,6 +2493,13 @@ int qcow2_update_header(BlockDriverState *bs)
     total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
     refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);
+
+    if (ret) {
+        goto fail;
+    }
+
+
     *header = (QCowHeader) {
         /* Version 2 fields */
         .magic                  = cpu_to_be32(QCOW_MAGIC),
@@ -2456,6 +2522,7 @@ int qcow2_update_header(BlockDriverState *bs)
         .autoclear_features     = cpu_to_be64(s->autoclear_features),
         .refcount_order         = cpu_to_be32(s->refcount_order),
         .header_length          = cpu_to_be32(header_length),
+        .compression_type       = cpu_to_be32(s->compression_type),
     };
 
     /* For older versions, write a shorter header */
@@ -2553,6 +2620,11 @@ int qcow2_update_header(BlockDriverState *bs)
                 .bit  = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
                 .name = "lazy refcounts",
             },
+            {
+                .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+                .bit  = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
+                .name = "compression type",
+            },
         };
 
         ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3107,6 +3179,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         .refcount_table_offset      = cpu_to_be64(cluster_size),
         .refcount_table_clusters    = cpu_to_be32(1),
         .refcount_order             = cpu_to_be32(refcount_order),
+        .compression_type           = cpu_to_be32(QCOW2_COMPRESSION_TYPE_ZLIB),
         .header_length              = cpu_to_be32(sizeof(*header)),
     };
 
@@ -3126,6 +3199,20 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
             cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
     }
 
+    if (qcow2_opts->has_compression_type &&
+        qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+        if (qcow2_opts->compression_type >= QCOW2_COMPRESSION_TYPE__MAX) {
+            error_setg_errno(errp, -EINVAL, "Unknown compression type");
+            goto out;
+        }
+
+        header->compression_type = cpu_to_be32(qcow2_opts->compression_type);
+
+        header->incompatible_features |=
+            cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
+    }
+
     ret = blk_pwrite(blk, 0, header, cluster_size, 0);
     g_free(header);
     if (ret < 0) {
@@ -3307,6 +3394,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
         { BLOCK_OPT_ENCRYPT,            BLOCK_OPT_ENCRYPT_FORMAT },
         { BLOCK_OPT_COMPAT_LEVEL,       "version" },
         { BLOCK_OPT_DATA_FILE_RAW,      "data-file-raw" },
+        { BLOCK_OPT_COMPRESSION_TYPE,   "compression-type" },
         { NULL, NULL },
     };
 
@@ -5239,6 +5327,12 @@ static QemuOptsList qcow2_create_opts = {
             .help = "Width of a reference count entry in bits",
             .def_value_str = "16"
         },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "0"
+        },
         { /* end of list */ }
     }
 };
block/qcow2.h +19 -7
index fdee297f33..b44344c78e 100644
@@ -135,6 +135,7 @@ typedef struct QCowHeader {
 
     uint32_t refcount_order;
     uint32_t header_length;
+    uint32_t compression_type;
 } QEMU_PACKED QCowHeader;
 
 typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -198,16 +199,20 @@ enum {
 
 /* Incompatible feature bits */
 enum {
-    QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
-    QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
-    QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
-    QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
-    QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
-    QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_DIRTY_BITNR            = 0,
+    QCOW2_INCOMPAT_CORRUPT_BITNR          = 1,
+    QCOW2_INCOMPAT_DATA_FILE_BITNR        = 2,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
+    QCOW2_INCOMPAT_DIRTY                  = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+    QCOW2_INCOMPAT_CORRUPT                = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+    QCOW2_INCOMPAT_DATA_FILE              = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION_TYPE       =
+        1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
 
     QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                     | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION_TYPE,
 };
 
 /* Compatible feature bits */
@@ -350,6 +355,13 @@ typedef struct BDRVQcow2State {
     int nb_compress_threads;
 
     BdrvChild *data_file;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set
+     */
+    uint32_t compression_type;
 } BDRVQcow2State;
 
 typedef struct Qcow2COWRegion {
docs/interop/qcow2.txt +21 -1
index af5711e533..5b8a8d15fe 100644
@@ -109,7 +109,12 @@ in the description of a field.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default: zlib.
+                                If the compression type is default the bit must
+                                be unset.
+
+                    Bits 4-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can
@@ -165,6 +170,21 @@ in the description of a field.
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
 
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.
+                    When the deafult compression type is used the compression
+                    type bit (incompatible feature bit 3) must be unset.
+                    When any other compression type is used the compression
+                    type bit must be set.
+                    Qemu versions older than 4.1 can use images created with
+                    the default compression type without any additional
+                    preparations and cannot use images created with any other
+                    compression type.
+
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
 
include/block/block_int.h +1 -0
index 01e855a066..814917baec 100644
@@ -58,6 +58,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 #define BLOCK_OPT_DATA_FILE         "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +20 -2
index 7ccbfff9d0..6aa8b99993 100644
@@ -78,6 +78,8 @@
 #
 # @bitmaps: A list of qcow2 bitmap details (since 4.0)
 #
+# @compression-type: the image cluster compression method (since 4.1)
+#
 # Since: 1.7
 ##
 { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
       '*corrupt': 'bool',
       'refcount-bits': 'int',
       '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'
   } }
 
 ##
@@ -4206,6 +4209,18 @@
   'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see <http://zlib.net/>
+#
+# Since: 4.1
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
 ##
 # @BlockdevCreateOptionsQcow2:
 #
@@ -4228,6 +4243,8 @@
 # @preallocation    Preallocation mode for the new image (default: off)
 # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
 # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.1)
 #
 # Since: 2.12
 ##
@@ -4243,7 +4260,8 @@
             '*cluster-size':    'size',
             '*preallocation':   'PreallocMode',
             '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }
 
 ##
 # @BlockdevCreateOptionsQed:
Signature · 1 line
2.17.0
#10 lore raw

PATCH 3/3 qcow2: add zstd cluster compression

In reply to Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

---
 block/qcow2.c        | 96 ++++++++++++++++++++++++++++++++++++++++++++
 configure            | 32 +++++++++++++++
 qapi/block-core.json |  3 +-
 3 files changed, 130 insertions(+), 1 deletion(-)
block/qcow2.c +96 -0
index 37a563a671..caa04b0beb 100644
@@ -27,6 +27,11 @@
 #define ZLIB_CONST
 #include <zlib.h>
 
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
 #include "block/block_int.h"
 #include "block/qdict.h"
 #include "sysemu/block-backend.h"
@@ -1209,6 +1214,9 @@ static int check_compression_type(BDRVQcow2State *s, Error **errp)
 
     switch (s->compression_type) {
     case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
         break;
 
     default:
@@ -4092,6 +4100,84 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
     return ret;
 }
 
+#ifdef CONFIG_ZSTD
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ *          a negative error code on fail
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+
+    if (dest_size < sizeof(*c_size)) {
+        return -ENOMEM;
+    }
+
+    dest_size -= sizeof(*c_size);
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+    if (ZSTD_isError(ret)) {
+        if (ret == ZSTD_error_dstSize_tooSmall) {
+            return -ENOMEM;
+        } else {
+            return -EIO;
+        }
+    }
+
+    /* store the compressed chunk size in the very beginning of the buffer */
+    *c_size = ret;
+
+    return ret + sizeof(ret);
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          -EIO on fail
+ */
+
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{
+    ssize_t ret;
+    /*
+     * zstd decompress wants to know the exact lenght of the data
+     * for that purpose, on the compression the length is stored in
+     * the very beginning of the compressed buffer
+     */
+    const uint32_t *s_size = src;
+    const char *s_buf = ((char *) src) + sizeof(*s_size);
+
+    ret = ZSTD_decompress(dest, dest_size, s_buf, *s_size);
+
+    if (ZSTD_isError(ret)) {
+        return -EIO;
+    }
+
+    return 0;
+}
+#endif
+
 #define MAX_COMPRESS_THREADS 4
 
 typedef ssize_t (*Qcow2CompressFunc)(void *dest, size_t dest_size,
@@ -4180,6 +4266,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
         fn = qcow2_zlib_compress;
         break;
 
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+        fn = qcow2_zstd_compress;
+        break;
+#endif
     default:
         return -ENOTSUP;
     }
@@ -4212,6 +4303,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
         fn = qcow2_zlib_decompress;
         break;
 
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+        fn = qcow2_zstd_decompress;
+        break;
+#endif
     default:
         return -ENOTSUP;
     }
configure +32 -0
index 1c563a7027..57a80e38e7 100755
@@ -433,6 +433,7 @@ opengl_dmabuf="no"
 cpuid_h="no"
 avx2_opt=""
 zlib="yes"
+zstd=""
 capstone=""
 lzo=""
 snappy=""
@@ -1333,6 +1334,10 @@ for opt do
   ;;
   --disable-lzfse) lzfse="no"
   ;;
+  --enable-zstd) zstd="yes"
+  ;;
+  --disable-zstd) zstd="no"
+  ;;
   --enable-guest-agent) guest_agent="yes"
   ;;
   --disable-guest-agent) guest_agent="no"
@@ -1788,6 +1793,7 @@ disabled with --disable-FEATURE, default is enabled if available:
                   (for reading bzip2-compressed dmg images)
   lzfse           support of lzfse compression library
                   (for reading lzfse-compressed dmg images)
+  zstd            support of zstd compression library
   seccomp         seccomp support
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
@@ -2374,6 +2380,29 @@ EOF
     fi
 fi
 
+#########################################
+# zstd check
+
+if test "$zstd" != "no" ; then
+    if $pkg_config --exists libzstd; then
+        zstd_cflags=$($pkg_config --cflags libzstd)
+        zstd_libs=$($pkg_config --libs libzstd)
+        QEMU_CFLAGS="$zstd_cflags $QEMU_CFLAGS"
+        LIBS="$zstd_libs $LIBS"
+    else
+        cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+        if compile_prog "" "-lzstd" ; then
+            LIBS="$LIBS -lzstd"
+        else
+            error_exit "zstd check failed" \
+                "Make sure to have the zstd libs and headers installed."
+        fi
+    fi
+fi
+
 ##########################################
 # libseccomp check
 
@@ -7253,6 +7282,9 @@ fi
 if test "$sheepdog" = "yes" ; then
   echo "CONFIG_SHEEPDOG=y" >> $config_host_mak
 fi
+if test "$zstd" = "yes" ; then
+  echo "CONFIG_ZSTD=y" >> $config_host_mak
+fi
 
 if test "$tcg_interpreter" = "yes"; then
   QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
qapi/block-core.json +2 -1
index 6aa8b99993..2604f201ee 100644
@@ -4215,11 +4215,12 @@
 # Compression type used in qcow2 image file
 #
 # @zlib:  zlib compression, see <http://zlib.net/>
+# @zstd:  zstd compression, see <http://github.com/facebook/zstd>
 #
 # Since: 4.1
 ##
 { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', 'zstd' ] }
 
 ##
 # @BlockdevCreateOptionsQcow2:
Signature · 1 line
2.17.0
#9 lore raw

PATCH 2/3 qcow2: rework the cluster compression routine

In reply to Denis Plotnikov

The patch allow to process image compression type defined in the image header and choose an appropriate method for image clusters (de)compression.

---
 block/qcow2.c | 93 ++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 73 insertions(+), 20 deletions(-)
block/qcow2.c +73 -20
index 921eb67b80..37a563a671 100644
@@ -4005,8 +4005,11 @@ fail:
 }
 
 /*
- * qcow2_compress()
+ * qcow2_zlib_compress()
  *
+ * Compress @src_size bytes of data using zlib compression method
+ *
+ * @dest_size bytes.
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
  *
@@ -4014,8 +4017,8 @@ fail:
  *          -1 destination buffer is not enough to store compressed data
  *          -2 on any other error
  */
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
-                              const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
 {
     ssize_t ret;
     z_stream strm;
@@ -4025,7 +4028,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
     ret = deflateInit2(&strm, Z_DEFAULT_COMPRESSION, Z_DEFLATED,
                        -12, 9, Z_DEFAULT_STRATEGY);
     if (ret != Z_OK) {
-        return -2;
+        return -EIO;
     }
 
     /* strm.next_in is not const in old zlib versions, such as those used on
@@ -4039,7 +4042,7 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
     if (ret == Z_STREAM_END) {
         ret = dest_size - strm.avail_out;
     } else {
-        ret = (ret == Z_OK ? -1 : -2);
+        ret = (ret == Z_OK ? -ENOMEM : -EIO);
     }
 
     deflateEnd(&strm);
@@ -4048,10 +4051,10 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
 }
 
 /*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
  *
  * Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -4059,8 +4062,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
  * Returns: 0 on success
  *          -1 on fail
  */
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-                                const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
 {
     int ret = 0;
     z_stream strm;
@@ -4073,7 +4076,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
 
     ret = inflateInit2(&strm, -12);
     if (ret != Z_OK) {
-        return -1;
+        return -EIO;
     }
 
     ret = inflate(&strm, Z_FINISH);
@@ -4081,7 +4084,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
         /* We approve Z_BUF_ERROR because we need @dest buffer to be filled, but
          * @src buffer may be processed partly (because in qcow2 we know size of
          * compressed data with precision of one sector) */
-        ret = -1;
+        ret = -EIO;
     }
 
     inflateEnd(&strm);
@@ -4153,20 +4156,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
     return arg.ret;
 }
 
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
 static ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
                   const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_compress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_compress;
+        break;
+
+    default:
+        return -ENOTSUP;
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on fail
+ */
 static ssize_t coroutine_fn
 qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                     const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_decompress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_decompress;
+        break;
+
+    default:
+        return -ENOTSUP;
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
 /* XXX: put compressed sectors first, then all the cluster aligned
@@ -4178,7 +4228,7 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
     BDRVQcow2State *s = bs->opaque;
     QEMUIOVector hd_qiov;
     int ret;
-    size_t out_len;
+    ssize_t out_len;
     uint8_t *buf, *out_buf;
     uint64_t cluster_offset;
 
@@ -4217,16 +4267,19 @@ qcow2_co_pwritev_compressed(BlockDriverState *bs, uint64_t offset,
 
     out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
                                 buf, s->cluster_size);
-    if (out_len == -2) {
-        ret = -EINVAL;
-        goto fail;
-    } else if (out_len == -1) {
+    if (out_len == -ENOMEM) {
         /* could not compress: write normal cluster */
         ret = qcow2_co_pwritev(bs, offset, bytes, qiov, 0);
         if (ret < 0) {
             goto fail;
         }
         goto success;
+    } else if (out_len < 0) {
+        /*
+         * encounter other compression issues propagate error to the upper level
+         */
+        ret = out_len;
+        goto fail;
     }
 
     qemu_co_mutex_lock(&s->lock);
Signature · 1 line
2.17.0
#2 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

In reply to Denis Plotnikov
Quoted 11 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly x2 faster than ZLIB providing a comparable compression ratio

2x

Quoted 4 lines

and therefore provide a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type is backward compatible with older qemu versions.

s/is/are/

Quoted 19 lines
---
 block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
 block/qcow2.h             | 26 ++++++++---
 docs/interop/qcow2.txt    | 22 ++++++++-
 include/block/block_int.h |  1 +
 qapi/block-core.json      | 22 ++++++++-
 5 files changed, 155 insertions(+), 10 deletions(-)
block/qcow2.c +1 -0
index 3ace3b2209..921eb67b80 100644
@@ -1202,6 +1202,47 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
     return ret;
 }
 
+static int check_compression_type(BDRVQcow2State *s, Error **errp)

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1215

Quoted 10 lines
block/qcow2.c +10 -0
+{
+    bool is_set;
+    int ret = 0;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        if (errp) {

Useless check for errp being non-NULL. What's worse, if the caller passes in NULL because they don't care about the error, then your code behaves differently. Just call error_setg() and return -ENOTSUP unconditionally.

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 10 lines
block/qcow2.c +10 -0
+            error_setg(errp, "qcow2: unknown compression type: %u",
+                       s->compression_type);
+            return -ENOTSUP;
+        }
+    }
+
+    /*
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set

Is there a strong reason for forbid the incompatible feature flag with ZLIB? Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1237

Quoted 12 lines
block/qcow2.c +12 -0
+     */
+    is_set = s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE;
+
+    if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+        if (is_set) {
+            ret = -EINVAL;
+        }
+    } else {
+        if (!is_set) {
+            ret = -EINVAL;
+        }
+    }

More compact as:

if ((s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) == is_set)

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1240

Quoted 3 lines
block/qcow2.c +3 -0
+
+    if (ret && errp) {
+        error_setg(errp, "qcow2: Illegal compression type setting");

s/Illegal/Invalid/ (the user isn't breaking a law)

Also, don't check whether errp is non-NULL. Just blindly call error_setg() if ret is non-zero.

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1243

Quoted 3 lines
block/qcow2.c +3 -0
+    }
+
+    return ret;

Or even shorter (psuedocode):

if ((compression == ZLIB) != is_set) {
    error_setg();
    return -EINVAL;
}

return 0;

block/qcow2.c static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, · line 1363

Quoted 11 lines
block/qcow2.c +4 -0
+}
+
 /* Called with s->lock held.  */
 static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                       int flags, Error **errp)
@@ -1318,6 +1359,24 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
     s->compatible_features      = header.compatible_features;
     s->autoclear_features       = header.autoclear_features;
 
+    /* Handle compression type */
+    if (header.header_length > 104) {

Magic number. Please spell this as header.header_length > offsetof (xxx, compression_type) for the appropriate xxx type. Also, do you need to sanity check for an image using a length of 105-107 (invalid) vs. an image of 108 or larger?

Quoted 20 lines
block/qcow2.c +16 -0
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        /*
+         * older qcow2 images don't contain the compression type header
+         * field, distinguish them by the header length and use
+         * the only valid compression type in that case
+         */
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
     if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
         void *feature_table = NULL;
         qcow2_read_extensions(bs, header.header_length, ext_end,

block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2496

Quoted 4 lines
block/qcow2.c +1 -0
@@ -2434,6 +2493,13 @@ int qcow2_update_header(BlockDriverState *bs)
     total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
     refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);

Why are you ignoring the error here?

block/qcow2.c static QemuOptsList qcow2_create_opts = { · line 5334

Quoted 9 lines
block/qcow2.c +5 -0
@@ -5239,6 +5327,12 @@ static QemuOptsList qcow2_create_opts = {
             .help = "Width of a reference count entry in bits",
             .def_value_str = "16"
         },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "0"

Eww. Why are we exposing an integer rather than an enum value as the default value? This should probably be "zlib".

block/qcow2.c in the description of a field. · line 113

Quoted 8 lines
block/qcow2.c +3 -1
+++ b/docs/interop/qcow2.txt
@@ -109,7 +109,12 @@ in the description of a field.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default: zlib.

Maybe: "differs from the default of zlib".

Up to here is okay.

block/qcow2.c in the description of a field. · line 115

Quoted 2 lines
block/qcow2.c +2 -0
+                                If the compression type is default the bit must
+                                be unset.

Is this restriction necessary?

Quoted 6 lines
block/qcow2.c +2 -0
+
+                    Bits 4-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can

block/qcow2.c in the description of a field. · line 178

Quoted 9 lines
block/qcow2.c +6 -0
@@ -165,6 +170,21 @@ in the description of a field.
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
 
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.

Where is the documentation that a value of 0 corresponds to zlib?

block/qcow2.c in the description of a field. · line 179

Quoted 1 line
block/qcow2.c +1 -0
+                    When the deafult compression type is used the compression

default

block/qcow2.c in the description of a field. · line 180

Quoted 1 line
block/qcow2.c +1 -0
+                    type bit (incompatible feature bit 3) must be unset.

Is this restriction necessary?

Quoted 36 lines
block/qcow2.c +7 -0
+                    When any other compression type is used the compression
+                    type bit must be set.
+                    Qemu versions older than 4.1 can use images created with
+                    the default compression type without any additional
+                    preparations and cannot use images created with any other
+                    compression type.
+
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
 
include/block/block_int.h +1 -0
index 01e855a066..814917baec 100644
@@ -58,6 +58,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 #define BLOCK_OPT_DATA_FILE         "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +2 -0
index 7ccbfff9d0..6aa8b99993 100644
@@ -78,6 +78,8 @@
 #
 # @bitmaps: A list of qcow2 bitmap details (since 4.0)
 #
+# @compression-type: the image cluster compression method (since 4.1)
+#
 # Since: 1.7
 ##
 { 'struct': 'ImageInfoSpecificQCow2',

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
@@ -89,7 +91,8 @@
       '*corrupt': 'bool',
       'refcount-bits': 'int',
       '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Quoted 32 lines
qapi/block-core.json +14 -0
   } }
 
 ##
@@ -4206,6 +4209,18 @@
   'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see <http://zlib.net/>
+#
+# Since: 4.1
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
 ##
 # @BlockdevCreateOptionsQcow2:
 #
@@ -4228,6 +4243,8 @@
 # @preallocation    Preallocation mode for the new image (default: off)
 # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
 # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.1)
 #
 # Since: 2.12
 ##

qapi/block-core.json line 4264

Quoted 6 lines
qapi/block-core.json +2 -1
@@ -4243,7 +4260,8 @@
             '*cluster-size':    'size',
             '*preallocation':   'PreallocMode',
             '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }

But this one must indeed be optional.

Quoted 4 lines
qapi/block-core.json +0 -0
 ##
 # @BlockdevCreateOptionsQed:
 
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#11 lore raw

Re: [Qemu-devel] [PATCH v1 3/3] qcow2: add zstd cluster compression

In reply to Denis Plotnikov
Quoted 7 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available.

Quoted 5 lines
---
 block/qcow2.c        | 96 ++++++++++++++++++++++++++++++++++++++++++++
 configure            | 32 +++++++++++++++
 qapi/block-core.json |  3 +-
 3 files changed, 130 insertions(+), 1 deletion(-)

Where is the change to docs/interop/qcow2.txt to describe this new compression format?

Quoted 8 lines
block/qcow2.c +0 -0
index 37a563a671..caa04b0beb 100644
@@ -27,6 +27,11 @@
 #define ZLIB_CONST
 #include <zlib.h>

block/qcow2.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 4123

Quoted 10 lines
block/qcow2.c +8 -0
 
 
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+

Do you always want exactly 4 bytes for the compressed size? Or is it worth some sort of variable-length encoding, since we're already dealing with non-cacheline-aligned data? You can represent all sizes up to 4M using a maximum of 3 bytes (set the high bit if the integer continues, then sizes 0-127 take 1 byte [7 bits], 128-32767 take 2 bytes [15 bits], and 32768-4194303 take 3 bytes [22 bits]).

block/qcow2.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 4130

Quoted 7 lines
block/qcow2.c +7 -0
+    if (dest_size < sizeof(*c_size)) {
+        return -ENOMEM;
+    }
+
+    dest_size -= sizeof(*c_size);
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);

The fact that you are storing the size separate from the data passed to zstd MUST be documented in the qcow2 spec, for the next person to produce/consume the same data.

block/qcow2.c line 4223

Quoted 12 lines

+++ b/qapi/block-core.json

block/qcow2.c +2 -1
@@ -4215,11 +4215,12 @@
 # Compression type used in qcow2 image file
 #
 # @zlib:  zlib compression, see <http://zlib.net/>
+# @zstd:  zstd compression, see <http://github.com/facebook/zstd>
 #
 # Since: 4.1
 ##
 { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', 'zstd' ] }

Since you patched configure so that linking against zstd is optional, this should use { 'name':'zstd', 'if':'CONDITIONAL' } so that during introspection, the enum only advertises zstd on a build that linked against the library.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

In reply to Eric Blake
Quoted 12 lines · Eric Blake
Quoted 11 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that*all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly x2 faster than ZLIB providing a comparable compression ratio

Quoted 7 lines

2x

Quoted 4 lines

and therefore provide a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type is backward compatible with older qemu versions.

s/is/are/

grammar matters, will be changed

Quoted 18 lines
Quoted 18 lines
---
  block/qcow2.c             | 94 +++++++++++++++++++++++++++++++++++++++
  block/qcow2.h             | 26 ++++++++---
  docs/interop/qcow2.txt    | 22 ++++++++-
  include/block/block_int.h |  1 +
  qapi/block-core.json      | 22 ++++++++-
  5 files changed, 155 insertions(+), 10 deletions(-)
block/qcow2.c +0 -0
index 3ace3b2209..921eb67b80 100644
@@ -1202,6 +1202,47 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
      return ret;
  }
 
Quoted 15 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1215

Quoted 11 lines
block/qcow2.c +11 -0
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    bool is_set;
+    int ret = 0;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        if (errp) {

Useless check for errp being non-NULL. What's worse, if the caller passes in NULL because they don't care about the error, then your code behaves differently. Just call error_setg() and return -ENOTSUP unconditionally.

ok

Quoted 11 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 11 lines
block/qcow2.c +10 -0
+            error_setg(errp, "qcow2: unknown compression type: %u",
+                       s->compression_type);
+            return -ENOTSUP;
+        }
+    }
+
+    /*
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set
Quoted 2 lines

Is there a strong reason for forbid the incompatible feature flag with ZLIB?

The main reason is to guarantee image opening for older qemu if compression type is zlib.

Quoted 4 lines

Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

I just can't imagine when and why we would want to set incompatible feature flag with zlib. Suppose we have zlib with the flag set, then older qemu can't open the image although it is able to work with the zlib compression type. For now, I don't understand why we should make such an artificial restriction.

Quoted 13 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1237

Quoted 13 lines
block/qcow2.c +12 -0
+     */
+    is_set = s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE;
+
+    if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+        if (is_set) {
+            ret = -EINVAL;
+        }
+    } else {
+        if (!is_set) {
+            ret = -EINVAL;
+        }
+    }
Quoted 3 lines

More compact as:

if ((s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) == is_set)

ok

Quoted 5 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1240

Quoted 4 lines
block/qcow2.c +3 -0
+
+    if (ret && errp) {
+        error_setg(errp, "qcow2: Illegal compression type setting");

s/Illegal/Invalid/ (the user isn't breaking a law)

Don't know, may be they do :) but, I'll change it

Quoted 3 lines

Also, don't check whether errp is non-NULL. Just blindly call error_setg() if ret is non-zero.

ok

Quoted 21 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1243

Quoted 4 lines
block/qcow2.c +3 -0
+    }
+
+    return ret;

Or even shorter (psuedocode):

if ((compression == ZLIB) != is_set) {
     error_setg();
     return -EINVAL;
}

return 0;

Quoted 9 lines
block/qcow2.c +2 -0
+}
+
  /* Called with s->lock held.  */
  static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
                                        int flags, Error **errp)
@@ -1318,6 +1359,24 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
      s->compatible_features      = header.compatible_features;
      s->autoclear_features       = header.autoclear_features;
 
Quoted 6 lines

block/qcow2.c static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, · line 1363

Quoted 2 lines
block/qcow2.c +2 -0
+    /* Handle compression type */
+    if (header.header_length > 104) {

Magic number. Please spell this as header.header_length > offsetof (xxx, compression_type) for the appropriate xxx type. Also, do you need to sanity check for an image using a length of 105-107 (invalid) vs. an image of 108 or larger?

Nice, will change it

Quoted 24 lines
Quoted 24 lines
block/qcow2.c +16 -0
+        header.compression_type = be32_to_cpu(header.compression_type);
+        s->compression_type = header.compression_type;
+    } else {
+        /*
+         * older qcow2 images don't contain the compression type header
+         * field, distinguish them by the header length and use
+         * the only valid compression type in that case
+         */
+        s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+    }
+
+    ret = check_compression_type(s, errp);
+    if (ret) {
+        goto fail;
+    }
+
      if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
          void *feature_table = NULL;
          qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2434,6 +2493,13 @@ int qcow2_update_header(BlockDriverState *bs)
      total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
      refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
Quoted 2 lines

block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2496

Quoted 1 line
block/qcow2.c +1 -0
+    ret = check_compression_type(s, NULL);

Why are you ignoring the error here?

qcow2_update_header() doesn't use the error and just returns an error code or 0

Quoted 13 lines

block/qcow2.c static QemuOptsList qcow2_create_opts = { · line 5334

Quoted 11 lines
block/qcow2.c +5 -0
@@ -5239,6 +5327,12 @@ static QemuOptsList qcow2_create_opts = {
              .help = "Width of a reference count entry in bits",
              .def_value_str = "16"
          },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",
+            .def_value_str = "0"

Eww. Why are we exposing an integer rather than an enum value as the default value? This should probably be "zlib".

Sure

Quoted 12 lines

block/qcow2.c in the description of a field. · line 113

Quoted 10 lines
block/qcow2.c +3 -1
+++ b/docs/interop/qcow2.txt
@@ -109,7 +109,12 @@ in the description of a field.
                                  An External Data File Name header extension may
                                  be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default: zlib.

Maybe: "differs from the default of zlib".

Quoted 5 lines

Up to here is okay.

block/qcow2.c in the description of a field. · line 115

Quoted 2 lines
block/qcow2.c +2 -0
+                                If the compression type is default the bit must
+                                be unset.

Is this restriction necessary?

Please, see above about older qemu usage

Quoted 10 lines
Quoted 10 lines
block/qcow2.c +2 -0
+
+                    Bits 4-63:  Reserved (set to 0)
 
           80 -  87:  compatible_features
                      Bitmask of compatible features. An implementation can
@@ -165,6 +170,21 @@ in the description of a field.
                      Length of the header structure in bytes. For version 2
                      images, the length is always assumed to be 72 bytes.
 
Quoted 7 lines

block/qcow2.c in the description of a field. · line 178

Quoted 6 lines
block/qcow2.c +6 -0
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.

Where is the documentation that a value of 0 corresponds to zlib?

Should I do it right here or it's better to add a separate chapter in the docs/interop/qcow2.txt ?

Quoted 43 lines

block/qcow2.c in the description of a field. · line 179

Quoted 2 lines
block/qcow2.c +1 -0
+                    When the deafult compression type is used the compression

default

block/qcow2.c in the description of a field. · line 180

Quoted 1 line
block/qcow2.c +1 -0
+                    type bit (incompatible feature bit 3) must be unset.

Is this restriction necessary?

Quoted 36 lines
block/qcow2.c +7 -0
+                    When any other compression type is used the compression
+                    type bit must be set.
+                    Qemu versions older than 4.1 can use images created with
+                    the default compression type without any additional
+                    preparations and cannot use images created with any other
+                    compression type.
+
  Directly after the image header, optional sections called header extensions can
  be stored. Each extension has a structure like the following:
 
include/block/block_int.h +1 -0
index 01e855a066..814917baec 100644
@@ -58,6 +58,7 @@
  #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
  #define BLOCK_OPT_DATA_FILE         "data_file"
  #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
  #define BLOCK_PROBE_BUF_SIZE        512
 
qapi/block-core.json +2 -0
index 7ccbfff9d0..6aa8b99993 100644
@@ -78,6 +78,8 @@
  #
  # @bitmaps: A list of qcow2 bitmap details (since 4.0)
  #
+# @compression-type: the image cluster compression method (since 4.1)
+#
  # Since: 1.7
  ##
  { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
Quoted 8 lines

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
        '*corrupt': 'bool',
        'refcount-bits': 'int',
        '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Why we should if we don't care ?

Quoted 48 lines
Quoted 33 lines
qapi/block-core.json +14 -0
    } }
 
  ##
@@ -4206,6 +4209,18 @@
    'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib:  zlib compression, see<http://zlib.net/>
+#
+# Since: 4.1
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
  ##
  # @BlockdevCreateOptionsQcow2:
  #
@@ -4228,6 +4243,8 @@
  # @preallocation    Preallocation mode for the new image (default: off)
  # @lazy-refcounts   True if refcounts may be updated lazily (default: off)
  # @refcount-bits    Width of reference counts in bits (default: 16)
+# @compression-type The image cluster compression method
+#                   (default: zlib, since 4.1)
  #
  # Since: 2.12
  ##

qapi/block-core.json line 4264

Quoted 6 lines
qapi/block-core.json +2 -1
@@ -4243,7 +4260,8 @@
              '*cluster-size':    'size',
              '*preallocation':   'PreallocMode',
              '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type': 'Qcow2CompressionType' } }

But this one must indeed be optional.

qapi/block-core.json

Quoted 4 lines
qapi/block-core.json +0 -0
  ##
  # @BlockdevCreateOptionsQed:
 

-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org

Signature · 2 lines
Best,
Denis
#12 lore raw

Re: [Qemu-devel] [PATCH v1 3/3] qcow2: add zstd cluster compression

In reply to Eric Blake
Quoted 8 lines · Eric Blake
Quoted 6 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available.

Quoted 9 lines
Quoted 5 lines
---
  block/qcow2.c        | 96 ++++++++++++++++++++++++++++++++++++++++++++
  configure            | 32 +++++++++++++++
  qapi/block-core.json |  3 +-
  3 files changed, 130 insertions(+), 1 deletion(-)

Where is the change to docs/interop/qcow2.txt to describe this new compression format?

ok

Quoted 10 lines
Quoted 9 lines
block/qcow2.c +0 -0
index 37a563a671..caa04b0beb 100644
@@ -27,6 +27,11 @@
  #define ZLIB_CONST
  #include <zlib.h>
 
Quoted 15 lines

block/qcow2.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 4123

Quoted 8 lines
block/qcow2.c +8 -0
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+

Do you always want exactly 4 bytes for the compressed size? Or is it worth some sort of variable-length encoding, since we're already dealing with non-cacheline-aligned data? You can represent all sizes up to 4M using a maximum of 3 bytes (set the high bit if the integer continues, then sizes 0-127 take 1 byte [7 bits], 128-32767 take 2 bytes [15 bits], and 32768-4194303 take 3 bytes [22 bits]).

Is it really worth to do that? I liked Kevin's comment that the size should complain with some variable size

Quoted 12 lines

block/qcow2.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 4130

Quoted 8 lines
block/qcow2.c +7 -0
+    if (dest_size < sizeof(*c_size)) {
+        return -ENOMEM;
+    }
+
+    dest_size -= sizeof(*c_size);
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);

The fact that you are storing the size separate from the data passed to zstd MUST be documented in the qcow2 spec, for the next person to produce/consume the same data.

Should I add a separate chapter there?

Quoted 11 lines
Quoted 11 lines

+++ b/qapi/block-core.json

block/qcow2.c +1 -0
@@ -4215,11 +4215,12 @@
  # Compression type used in qcow2 image file
  #
  # @zlib:  zlib compression, see <http://zlib.net/>
+# @zstd:  zstd compression, see <http://github.com/facebook/zstd>
  #
  # Since: 4.1
  ##
Quoted 8 lines

block/qcow2.c line 4223

Quoted 3 lines
block/qcow2.c +1 -1
  { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', 'zstd' ] }

Since you patched configure so that linking against zstd is optional, this should use { 'name':'zstd', 'if':'CONDITIONAL' } so that during introspection, the enum only advertises zstd on a build that linked against the library.

Sure, will change it

Quoted 1 line
Signature · 2 lines
Best,
Denis
#4 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

In reply to Denis Plotnikov
Quoted 11 lines · Denis Plotnikov
Quoted 5 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 3 lines
block/qcow2.c +3 -0
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set

Is there a strong reason for forbid the incompatible feature flag with ZLIB?

The main reason is to guarantee image opening for older qemu if compression type is zlib.

Quoted 4 lines

Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

Quoted 5 lines

I just can't imagine when and why we would want to set incompatible feature flag with zlib. Suppose we have zlib with the flag set, then older qemu can't open the image although it is able to work with the zlib compression type. For now, I don't understand why we should make such an artificial restriction.

We have an artificial restriction one way or the other.

Method 1 - artificial restriction that incompatible bit must NOT be set when compression type is zlib

Method 2 - artificial restriction that older qcow2 programs can't open a zlib image with incompatible bit set, even though removing the bit makes the image more portable.

It's desirable that qemu should NOT set the incompatible bit when it does not need to (for the sake of portability to more apps), but MANDATING that it must not set the bit for zlib is a stronger spec restriction.

I tend to lean for the spec being looser unless there is a strong reason for why it must be strict; just because qemu won't be setting the incompatible bit does not necessarily mean that all other implementations will try to be that careful; they may have valid reasons for setting the bit even when using zlib, but only if the spec permits them to do so.

Quoted 8 lines
Quoted 6 lines

block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2496

Quoted 5 lines
block/qcow2.c +1 -0
@@ -2434,6 +2493,13 @@ int qcow2_update_header(BlockDriverState *bs)
      total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
      refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);

Why are you ignoring the error here?

qcow2_update_header() doesn't use the error and just returns an error code or 0

Are we potentially losing a valuable error message (in which case qcow2_update_header should maybe be first patched to take an errp parameter), or is it always going to succeed (in which case &error_abort would document our intention that we know it can't fail), or is it really a case where it may fail, but we don't care about losing the message? Passing NULL is not wrong (as you say, we aren't plumbed to pass the message back up to the caller), but does raise enough suspicions as to be worth auditing the code while in the area.

Quoted 9 lines
Quoted 7 lines

docs/interop/qcow2.txt in the description of a field. · line 178

Quoted 6 lines
docs/interop/qcow2.txt +6 -0
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.

Where is the documentation that a value of 0 corresponds to zlib?

Should I do it right here or it's better to add a separate chapter in the docs/interop/qcow2.txt ?

Right here.

Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
qapi/block-core.json +3 -0
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
  #
  # @bitmaps: A list of qcow2 bitmap details (since 4.0)
  #
+# @compression-type: the image cluster compression method (since 4.1)
+#
  # Since: 1.7
  ##
  { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
Quoted 8 lines

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
        '*corrupt': 'bool',
        'refcount-bits': 'int',
        '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Quoted 1 line

Why we should if we don't care ?

Because it shows that we are running a new enough qemu that knows about the compression field (even when it is absent).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#5 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

Quoted 33 lines · Eric Blake
Quoted 11 lines · Denis Plotnikov
Quoted 5 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 3 lines
block/qcow2.c +3 -0
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set

Is there a strong reason for forbid the incompatible feature flag with ZLIB?

The main reason is to guarantee image opening for older qemu if compression type is zlib.

Quoted 4 lines

Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

Quoted 5 lines

I just can't imagine when and why we would want to set incompatible feature flag with zlib. Suppose we have zlib with the flag set, then older qemu can't open the image although it is able to work with the zlib compression type. For now, I don't understand why we should make such an artificial restriction.

We have an artificial restriction one way or the other.

Method 1 - artificial restriction that incompatible bit must NOT be set when compression type is zlib

Method 2 - artificial restriction that older qcow2 programs can't open a zlib image with incompatible bit set, even though removing the bit makes the image more portable.

It's desirable that qemu should NOT set the incompatible bit when it does not need to (for the sake of portability to more apps), but MANDATING that it must not set the bit for zlib is a stronger spec restriction.

Quoted 6 lines

I tend to lean for the spec being looser unless there is a strong reason for why it must be strict; just because qemu won't be setting the incompatible bit does not necessarily mean that all other implementations will try to be that careful; they may have valid reasons for setting the bit even when using zlib, but only if the spec permits them to do so.

So, how I should implement that? Method 1 or Method 2? How we can decide? Ask what other maintainers think about that?

Quoted 11 lines
Quoted 10 lines
Quoted 8 lines

block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2496

Quoted 7 lines
block/qcow2.c +1 -0
@@ -2434,6 +2493,13 @@ int qcow2_update_header(BlockDriverState *bs)
       total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
       refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
 
+    ret = check_compression_type(s, NULL);

Why are you ignoring the error here?

qcow2_update_header() doesn't use the error and just returns an error code or 0

Quoted 8 lines

Are we potentially losing a valuable error message (in which case qcow2_update_header should maybe be first patched to take an errp parameter), or is it always going to succeed (in which case &error_abort would document our intention that we know it can't fail), or is it really a case where it may fail, but we don't care about losing the message? Passing NULL is not wrong (as you say, we aren't plumbed to pass the message back up to the caller), but does raise enough suspicions as to be worth auditing the code while in the area.

Could we do it after adding this series since it already implemented without the error?

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
docs/interop/qcow2.txt +6 -0
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Where is the documentation that a value of 0 corresponds to zlib?

Should I do it right here or it's better to add a separate chapter in the docs/interop/qcow2.txt ?

Right here.

ok

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 13 lines
qapi/block-core.json +3 -0
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
   #
   # @bitmaps: A list of qcow2 bitmap details (since 4.0)
   #
+# @compression-type: the image cluster compression method (since 4.1)
+#
   # Since: 1.7
   ##
   { 'struct': 'ImageInfoSpecificQCow2',

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
@@ -89,7 +91,8 @@
         '*corrupt': 'bool',
         'refcount-bits': 'int',
         '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Quoted 4 lines
Quoted 1 line

Why we should if we don't care ?

Because it shows that we are running a new enough qemu that knows about the compression field (even when it is absent).

ok, if everybody agree with that I'll implement whatever is better

Quoted 1 line
Signature · 2 lines
Best,
Denis
#6 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

In reply to Denis Plotnikov

Am 03.07.2019 um 17:01 hat Denis Plotnikov geschrieben:

Quoted 45 lines · Eric Blake
Quoted 9 lines · Denis Plotnikov
Quoted 8 lines
block/qcow2.c +0 -0
index 3ace3b2209..921eb67b80 100644
@@ -1202,6 +1202,47 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
      return ret;
  }
 
Quoted 15 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1215

Quoted 11 lines
block/qcow2.c +11 -0
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    bool is_set;
+    int ret = 0;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        if (errp) {

Useless check for errp being non-NULL. What's worse, if the caller passes in NULL because they don't care about the error, then your code behaves differently. Just call error_setg() and return -ENOTSUP unconditionally.

ok

Quoted 11 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 11 lines
block/qcow2.c +10 -0
+            error_setg(errp, "qcow2: unknown compression type: %u",
+                       s->compression_type);
+            return -ENOTSUP;
+        }
+    }
+
+    /*
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set
Quoted 2 lines

Is there a strong reason for forbid the incompatible feature flag with ZLIB?

The main reason is to guarantee image opening for older qemu if compression type is zlib.

Quoted 4 lines

Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

Quoted 5 lines

I just can't imagine when and why we would want to set incompatible feature flag with zlib. Suppose we have zlib with the flag set, then older qemu can't open the image although it is able to work with the zlib compression type. For now, I don't understand why we should make such an artificial restriction.

We don't want to create such images, but we want to keep our code as simple as possible.

As your patch shows, forbidding the case is more work than just allowing it. So if external software can create such images, and it would just automatically work in QEMU, then why do the extra work to articifially forbid this?

(Actually, it's likely that on the first header update, QEMU would just end up dropping the useless flag, so we even "fix" such images.)

Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
qapi/block-core.json +2 -0
index 7ccbfff9d0..6aa8b99993 100644
@@ -78,6 +78,8 @@
  #
  # @bitmaps: A list of qcow2 bitmap details (since 4.0)
  #
+# @compression-type: the image cluster compression method (since 4.1)
+#
  # Since: 1.7
  ##
  { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
Quoted 8 lines

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
        '*corrupt': 'bool',
        'refcount-bits': 'int',
        '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Quoted 1 line

Why we should if we don't care ?

I was trying too check what the condition is under which the field will be present in the output, but I couldn't find any code for it.

So it looks like this patch never makes use of the field and it's dead code?

Kevin

#7 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

Quoted 55 lines · Kevin Wolf

Am 03.07.2019 um 17:01 hat Denis Plotnikov geschrieben:

Quoted 45 lines · Eric Blake
Quoted 9 lines · Denis Plotnikov
Quoted 8 lines
block/qcow2.c +0 -0
index 3ace3b2209..921eb67b80 100644
@@ -1202,6 +1202,47 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
       return ret;
   }
 
Quoted 15 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1215

Quoted 11 lines
block/qcow2.c +11 -0
+static int check_compression_type(BDRVQcow2State *s, Error **errp)
+{
+    bool is_set;
+    int ret = 0;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        break;
+
+    default:
+        if (errp) {

Useless check for errp being non-NULL. What's worse, if the caller passes in NULL because they don't care about the error, then your code behaves differently. Just call error_setg() and return -ENOTSUP unconditionally.

ok

Quoted 11 lines

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1225

Quoted 11 lines
block/qcow2.c +10 -0
+            error_setg(errp, "qcow2: unknown compression type: %u",
+                       s->compression_type);
+            return -ENOTSUP;
+        }
+    }
+
+    /*
+     * with QCOW2_COMPRESSION_TYPE_ZLIB the corresponding incompatible
+     * feature flag must be absent, with other compression types the
+     * incompatible feature flag must be set
Quoted 2 lines

Is there a strong reason for forbid the incompatible feature flag with ZLIB?

The main reason is to guarantee image opening for older qemu if compression type is zlib.

Quoted 4 lines

Sure, it makes the image impossible to open with older qemu, but it doesn't get in the way of newer qemu opening it. I'll have to see how your spec changes documented this, to see if you could instead word it as 'for ZLIB, the incompatible feature flag may be absent'.

Quoted 5 lines

I just can't imagine when and why we would want to set incompatible feature flag with zlib. Suppose we have zlib with the flag set, then older qemu can't open the image although it is able to work with the zlib compression type. For now, I don't understand why we should make such an artificial restriction.

We don't want to create such images, but we want to keep our code as simple as possible.

Quoted 7 lines

As your patch shows, forbidding the case is more work than just allowing it. So if external software can create such images, and it would just automatically work in QEMU, then why do the extra work to articifially forbid this?

(Actually, it's likely that on the first header update, QEMU would just end up dropping the useless flag, so we even "fix" such images.)

ok, removing the restriction

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Quoted 15 lines
qapi/block-core.json +2 -0
index 7ccbfff9d0..6aa8b99993 100644
@@ -78,6 +78,8 @@
   #
   # @bitmaps: A list of qcow2 bitmap details (since 4.0)
   #
+# @compression-type: the image cluster compression method (since 4.1)
+#
   # Since: 1.7
   ##
   { 'struct': 'ImageInfoSpecificQCow2',

qapi/block-core.json line 95

Quoted 6 lines
qapi/block-core.json +2 -1
@@ -89,7 +91,8 @@
         '*corrupt': 'bool',
         'refcount-bits': 'int',
         '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      '*compression-type': 'Qcow2CompressionType'

Why is this field optional? Can't we always populate it, even for older images?

Quoted 7 lines
Quoted 1 line

Why we should if we don't care ?

I was trying too check what the condition is under which the field will be present in the output, but I couldn't find any code for it.

So it looks like this patch never makes use of the field and it's dead code?

ok

Quoted 3 lines

Kevin

Signature · 2 lines
Best,
Denis
#13 lore raw

Re: [Qemu-devel] [PATCH v1 3/3] qcow2: add zstd cluster compression

In reply to Eric Blake

Am 03.07.2019 um 16:36 hat Eric Blake geschrieben:

Quoted 27 lines · Denis Plotnikov
Quoted 7 lines

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of compression ratio in comparison with zlib, which, by the moment, has been the only compression method available.

Quoted 5 lines
---
 block/qcow2.c        | 96 ++++++++++++++++++++++++++++++++++++++++++++
 configure            | 32 +++++++++++++++
 qapi/block-core.json |  3 +-
 3 files changed, 130 insertions(+), 1 deletion(-)

Where is the change to docs/interop/qcow2.txt to describe this new compression format?

Quoted 9 lines
block/qcow2.c +0 -0
index 37a563a671..caa04b0beb 100644
@@ -27,6 +27,11 @@
 #define ZLIB_CONST
 #include <zlib.h>
 
Quoted 15 lines

block/qcow2.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 4123

Quoted 8 lines
block/qcow2.c +8 -0
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;
+    uint32_t *c_size = dest;
+    /* steal some bytes to store compressed chunk size */
+    char *d_buf = ((char *) dest) + sizeof(*c_size);
+

Do you always want exactly 4 bytes for the compressed size? Or is it worth some sort of variable-length encoding, since we're already dealing with non-cacheline-aligned data? You can represent all sizes up to 4M using a maximum of 3 bytes (set the high bit if the integer continues, then sizes 0-127 take 1 byte [7 bits], 128-32767 take 2 bytes [15 bits], and 32768-4194303 take 3 bytes [22 bits]).

I don't think the additional complexity is worth it. The combination of endianness conversions (which are missing here, this is a bug!) and variable lengths is almost guaranteed to not only give us headaches, but also to result in corner case bugs.

Also, are we sure that we will always keep 2M as the maximum cluster size?

Kevin

#8 lore raw

Re: [Qemu-devel] [PATCH v1 1/3] qcow2: introduce compression type feature

In reply to Denis Plotnikov

I'd like to recommend a few commas.

Quoted 17 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to QCOW2 header that indicates that *all* compressed clusters must be (de)compressed using a certain compression type.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image.

The goal of the feature is to add support of other compression algorithms to qcow2. For example, ZSTD which is more effective on compression than ZLIB. It works roughly x2 faster than ZLIB providing a comparable compression ratio and therefore provide a performance advantage in backup scenarios.

The default compression is ZLIB. Images created with ZLIB compression type is backward compatible with older qemu versions.

[...]

Quoted 19 lines
docs/interop/qcow2.txt +6 -1
index af5711e533..5b8a8d15fe 100644
@@ -109,7 +109,12 @@ in the description of a field.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit. The bit must be set if
+                                the compression type differs from default: zlib.
+                                If the compression type is default the bit must
+                                be unset.
+
+                    Bits 4-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can

docs/interop/qcow2.txt in the description of a field. · line 179

Quoted 10 lines
docs/interop/qcow2.txt +7 -0
@@ -165,6 +170,21 @@ in the description of a field.
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
 
+        104 - 107:  compression_type
+                    Defines the compression method used for compressed clusters.
+                    A single compression type is applied to all compressed image
+                    clusters.
+                    The compression type is set on image creation only.
+                    The default compression type is zlib.
+                    When the deafult compression type is used the compression

Comma after used, please.

docs/interop/qcow2.txt in the description of a field. · line 181

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+                    type bit (incompatible feature bit 3) must be unset.
+                    When any other compression type is used the compression

Likewise.

docs/interop/qcow2.txt in the description of a field. · line 185

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+                    type bit must be set.
+                    Qemu versions older than 4.1 can use images created with
+                    the default compression type without any additional
+                    preparations and cannot use images created with any other

Comma after preparations, please.

Quoted 5 lines
docs/interop/qcow2.txt +2 -0
+                    compression type.
+
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
 

[...]