Skip to contentSkip to conversation

implement zstd cluster compression method

PATCH Superseded Denis Plotnikov 4 participants 21 messages Apr 28, 2020 – May 4, 2020 Block layer

v22 of 24 Superseded Latest is v24 →

lore
Share & tools
lore
#3 lore raw

PATCH v22 3/4 qcow2: add zstd cluster compression

In reply to Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
 docs/interop/qcow2.txt |   1 +
 configure              |   2 +-
 qapi/block-core.json   |   3 +-
 block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          |   7 ++
 slirp                  |   2 +-
 6 files changed, 181 insertions(+), 3 deletions(-)
docs/interop/qcow2.txt +1 -0
index 640e0eca40..18a77f737e 100644
@@ -209,6 +209,7 @@ version 2.
 
                     Available compression type values:
                         0: zlib <https://www.zlib.net/>
+                        1: zstd <http://github.com/facebook/zstd>
 
 
 === Header padding ===
configure +1 -1
index 23b5e93752..4e3a1690ea 100755
@@ -1861,7 +1861,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   lzfse           support of lzfse compression library
                   (for reading lzfse-compressed dmg images)
   zstd            support for zstd compression library
-                  (for migration compression)
+                  (for migration compression and qcow2 cluster compression)
   seccomp         seccomp support
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
qapi/block-core.json +2 -1
index 1522e2983f..6fbacddab2 100644
@@ -4293,11 +4293,12 @@
 # Compression type used in qcow2 image file
 #
 # @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
 #
 # Since: 5.1
 ##
 { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
 
 ##
 # @BlockdevCreateOptionsQcow2:
block/qcow2-threads.c +169 -0

193 lines not on this page · raw message

block/qcow2.c +7 -0
index 9c0b20c912..21f281b7f8 100644
@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
 {
     switch (s->compression_type) {
     case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
         break;
 
     default:
@@ -3478,6 +3481,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         }
 
         switch (qcow2_opts->compression_type) {
+#ifdef CONFIG_ZSTD
+        case QCOW2_COMPRESSION_TYPE_ZSTD:
+            break;
+#endif
         default:
             error_setg(errp, "Unknown compression type");
             goto out;
slirp +1 -1
index 2faae0f778..55ab21c9a3 160000
@@ -1 +1 @@
-Subproject commit 2faae0f778f818fadc873308f983289df697eb93
+Subproject commit 55ab21c9a36852915b81f1b41ebaf3b6509dd8ba
Signature · 1 line
2.17.0
#2 lore raw

PATCH v22 2/4 qcow2: rework the cluster compression routine

In reply to Denis Plotnikov

The patch enables processing the image compression type defined for the image and chooses an appropriate method for image clusters (de)compression.

---
 block/qcow2-threads.c | 71 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 60 insertions(+), 11 deletions(-)
block/qcow2-threads.c +60 -11
index a68126f291..7dbaf53489 100644
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
 } Qcow2CompressData;
 
 /*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
  *          -ENOMEM destination buffer is not enough to store compressed data
  *          -EIO    on any other error
  */
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
-                              const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
 {
     ssize_t ret;
     z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
 }
 
 /*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
  *
  * Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
  * Returns: 0 on success
  *          -EIO on fail
  */
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-                                const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
 {
     int ret;
     z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
     return arg.ret;
 }
 
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ *          a negative error code on failure
+ */
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
                   const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_compress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_compress;
+        break;
+
+    default:
+        abort();
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on failure
+ */
 ssize_t coroutine_fn
 qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                     const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_decompress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_decompress;
+        break;
+
+    default:
+        abort();
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
 
Signature · 1 line
2.17.0
#1 lore raw

PATCH v22 1/4 qcow2: introduce compression type feature

In reply to Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.

The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.

The tests are fixed in the following ways:

    * filter out compression_type for many tests
    * fix header size, feature table size and backing file offset
      affected tests: 031, 036, 061, 080
      header_size +=8: 1 byte compression type
                       7 bytes padding
      feature_table += 48: incompatible feature compression type
      backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
    * add "compression type" for test output matching when it isn't filtered
      affected tests: 049, 060, 061, 065, 144, 182, 242, 255

QAPI part:

---
 qapi/block-core.json             |  22 +++++-
 block/qcow2.h                    |  20 +++++-
 include/block/block_int.h        |   1 +
 block/qcow2.c                    | 113 +++++++++++++++++++++++++++++++
 tests/qemu-iotests/031.out       |  14 ++--
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++++++++++++++--------------
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  34 ++++++----
 tests/qemu-iotests/065           |  28 +++++---
 tests/qemu-iotests/080           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/242.out       |   5 ++
 tests/qemu-iotests/255.out       |   8 +--
 tests/qemu-iotests/common.filter |   3 +-
 16 files changed, 267 insertions(+), 96 deletions(-)
qapi/block-core.json +20 -2
index 943df1926a..1522e2983f 100644
@@ -78,6 +78,8 @@
 #
 # @bitmaps: A list of qcow2 bitmap details (since 4.0)
 #
+# @compression-type: the image cluster compression method (since 5.1)
+#
 # Since: 1.7
 ##
 { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
       '*corrupt': 'bool',
       'refcount-bits': 'int',
       '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      'compression-type': 'Qcow2CompressionType'
   } }
 
 ##
@@ -4284,6 +4287,18 @@
   'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.1
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
 ##
 # @BlockdevCreateOptionsQcow2:
 #
@@ -4307,6 +4322,8 @@
 #                 allowed values: off, falloc, full, metadata)
 # @lazy-refcounts: True if refcounts may be updated lazily (default: off)
 # @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#                    (default: zlib, since 5.1)
 #
 # Since: 2.12
 ##
@@ -4322,7 +4339,8 @@
             '*cluster-size':    'size',
             '*preallocation':   'PreallocMode',
             '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type':'Qcow2CompressionType' } }
 
 ##
 # @BlockdevCreateOptionsQed:
block/qcow2.h +19 -1
index f4de0a27d5..6a8b82e6cc 100644
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
 
     uint32_t refcount_order;
     uint32_t header_length;
+
+    /* Additional fields */
+    uint8_t compression_type;
+
+    /* header must be a multiple of 8 */
+    uint8_t padding[7];
 } QEMU_PACKED QCowHeader;
 
+QEMU_BUILD_BUG_ON(!QEMU_IS_ALIGNED(sizeof(QCowHeader), 8));
+
 typedef struct QEMU_PACKED QCowSnapshotHeader {
     /* header is 8 byte aligned */
     uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {
     QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
     QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
     QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
+    QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
     QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
     QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
     QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION      = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
 
     QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                     | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION,
 };
 
 /* Compatible feature bits */
@@ -366,6 +377,13 @@ typedef struct BDRVQcow2State {
 
     bool metadata_preallocation_checked;
     bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * For now, the only way to change the compression type
+     * is to convert the image with the desired compression type set.
+     */
+    Qcow2CompressionType compression_type;
 } BDRVQcow2State;
 
 typedef struct Qcow2COWRegion {
include/block/block_int.h +1 -0
index 4c3587ea19..bc91b69f46 100644
@@ -57,6 +57,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 #define BLOCK_OPT_DATA_FILE         "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
block/qcow2.c +113 -0

191 lines not on this page · raw message

tests/qemu-iotests/031.out +7 -7
index 46f97c5a4e..5a4beda6a2 100644
@@ -113,11 +113,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 Header extension:
@@ -146,11 +146,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
 
 magic                     0x514649fb
 version                   3
-backing_file_offset       0x1d8
+backing_file_offset       0x210
 backing_file_size         0x17
 cluster_bits              16
 size                      67108864
@@ -179,7 +179,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0xe2792aca
@@ -188,7 +188,7 @@ data                      'host_device'
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 Header extension:
tests/qemu-iotests/036.out +2 -2
index 23b699ce06..e409acf60e 100644
@@ -26,7 +26,7 @@ compatible_features       []
 autoclear_features        [63]
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 
@@ -38,7 +38,7 @@ compatible_features       []
 autoclear_features        []
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 *** done
tests/qemu-iotests/049.out +51 -51

225 lines not on this page · raw message

tests/qemu-iotests/060.out +1 -0
index 09caaea865..be5f8707a3 100644
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     corrupt: true
tests/qemu-iotests/061.out +20 -14
index 413cc4e0f4..45c029228c 100644
@@ -22,11 +22,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 magic                     0x514649fb
@@ -80,11 +80,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 magic                     0x514649fb
@@ -136,11 +136,11 @@ incompatible_features     [0]
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features     []
 compatible_features       [42]
 autoclear_features        [42]
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 magic                     0x514649fb
@@ -260,11 +260,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features     [0]
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    288
+length                    336
 data                      <binary>
 
 read 131072/131072 bytes at offset 0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
tests/qemu-iotests/065 +18 -10
index 6426474271..18dc488c7a 100755
@@ -88,24 +88,30 @@ class TestQMP(TestImageInfoSpecific):
 class TestQCow2(TestQemuImgInfo):
     '''Testing a qcow2 version 2 image'''
     img_options = 'compat=0.10'
-    json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
-    human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
+    json_compare = { 'compat': '0.10', 'refcount-bits': 16,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 0.10', 'compression type: zlib',
+                      'refcount bits: 16' ]
 
 class TestQCow3NotLazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
     img_options = 'compat=1.1,lazy_refcounts=off'
     json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
-                     'refcount-bits': 16, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
-                      'refcount bits: 16', 'corrupt: false' ]
+                     'refcount-bits': 16, 'corrupt': False,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 1.1', 'compression type: zlib',
+                      'lazy refcounts: false', 'refcount bits: 16',
+                      'corrupt: false' ]
 
 class TestQCow3Lazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
     img_options = 'compat=1.1,lazy_refcounts=on'
     json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
-                     'refcount-bits': 16, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
-                      'refcount bits: 16', 'corrupt: false' ]
+                     'refcount-bits': 16, 'corrupt': False,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 1.1', 'compression type: zlib',
+                      'lazy refcounts: true', 'refcount bits: 16',
+                      'corrupt: false' ]
 
 class TestQCow3NotLazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
@@ -113,7 +119,8 @@ class TestQCow3NotLazyQMP(TestQMP):
     img_options = 'compat=1.1,lazy_refcounts=off'
     qemu_options = 'lazy-refcounts=on'
     compare = { 'compat': '1.1', 'lazy-refcounts': False,
-                'refcount-bits': 16, 'corrupt': False }
+                'refcount-bits': 16, 'corrupt': False,
+                'compression-type': 'zlib' }
 
 
 class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +129,8 @@ class TestQCow3LazyQMP(TestQMP):
     img_options = 'compat=1.1,lazy_refcounts=on'
     qemu_options = 'lazy-refcounts=off'
     compare = { 'compat': '1.1', 'lazy-refcounts': True,
-                'refcount-bits': 16, 'corrupt': False }
+                'refcount-bits': 16, 'corrupt': False,
+                'compression-type': 'zlib' }
 
 TestImageInfoSpecific = None
 TestQemuImgInfo = None
tests/qemu-iotests/080 +1 -1
index a3d13c414e..7588c63b6c 100755
@@ -45,7 +45,7 @@ _supported_os Linux
 # - This is generally a test for compat=1.1 images
 _unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10'
 
-header_size=104
+header_size=112
 
 offset_backing_file_offset=8
 offset_backing_file_size=16
tests/qemu-iotests/144.out +2 -2
index c7aa2e4820..885a8874a5 100644
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
 { 'execute': 'qmp_capabilities' }
 {"return": {}}
 { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 
 === Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
 === Performing Live Snapshot 2 ===
 
 { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 *** done
tests/qemu-iotests/182.out +1 -1
index a8eea166c3..ae43654d32 100644
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
 {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
 {"return": {}}
 {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
-Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
 {"return": {}}
tests/qemu-iotests/242.out +5 -0
index 7ac8404d11..091b9126ce 100644
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
tests/qemu-iotests/255.out +4 -4
index 348909fdef..a3c99fd62e 100644
@@ -3,9 +3,9 @@ Finishing a commit job with background reads
 
 === Create backing chain and start VM ===
 
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 === Start background read requests ===
 
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
 
 === Create images and start VM ===
 
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 wrote 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/common.filter +2 -1
index 3f8ee3e5f7..279e0bbb0d 100644
@@ -152,7 +152,8 @@ _filter_img_create()
         -e "s# refcount_bits=[0-9]\\+##g" \
         -e "s# key-secret=[a-zA-Z0-9]\\+##g" \
         -e "s# iter-time=[0-9]\\+##g" \
-        -e "s# force_size=\\(on\\|off\\)##g"
+        -e "s# force_size=\\(on\\|off\\)##g" \
+        -e "s# compression_type=[a-zA-Z0-9]\\+##g"
 }
 
 _filter_img_info()
Signature · 1 line
2.17.0
#17 lore raw

PATCH v22 4/4 iotests: 287: add qcow2 compression type test

In reply to Denis Plotnikov

The test checks fulfilling qcow2 requirements for the compression type feature and zstd compression type operability.

---
 tests/qemu-iotests/287     | 152 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/287.out |  67 ++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 220 insertions(+)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +152 -0
new file mode 100755
index 0000000000..21fe1f19f5
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+_unsupported_imgopts 'compat=0.10' data_file
+
+COMPR_IMG="$TEST_IMG.compressed"
+RAND_FILE="$TEST_DIR/rand_data"
+
+_cleanup()
+{
+	_rm_test_img
+	rm -f "$COMPR_IMG"
+	rm -f "$RAND_FILE"
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+if IMGOPTS='compression_type=zstd' _make_test_img 64M |
+    grep "Invalid parameter 'zstd'"; then
+    _notrun "ZSTD is disabled"
+fi
+
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+_make_test_img -o compression_type=zlib 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+_make_test_img -o compression_type=zstd 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+_make_test_img -o compression_type=zlib 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
+    echo "Error: The image opened successfully. The image must not be opened."
+fi
+
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+_make_test_img -o compression_type=zstd 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
+    echo "Error: The image opened successfully. The image must not be opened."
+fi
+
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+_make_test_img -o compression_type=zlib 64M
+peek_file_be "$TEST_IMG" 104 1
+echo
+
+# zstd=1
+_make_test_img -o compression_type=zstd 64M
+peek_file_be "$TEST_IMG" 104 1
+echo
+
+echo
+echo "=== Testing simple reading and writing with zstd ==="
+echo
+_make_test_img -o compression_type=zstd 64M
+$QEMU_IO -c "write -c -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+# read on the cluster boundaries
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo "=== Testing adjacent clusters reading and writing with zstd ==="
+echo
+_make_test_img -o compression_type=zstd 64M
+$QEMU_IO -c "write -c -P 0xAB 0 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "write -c -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "write -c -P 0xAD 128K 64K " "$TEST_IMG" | _filter_qemu_io
+
+$QEMU_IO -c "read -P 0xAB 0 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 64K 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAD 128K 64k " "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo "=== Testing incompressible cluster processing with zstd ==="
+echo
+# create a 2M image and fill it with 1M likely incompressible data
+# and 1M compressible data
+dd if=/dev/urandom of="$RAND_FILE" bs=1M count=1 seek=1
+QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" \
+$QEMU_IO -f raw -c "write -P 0xFA 0 1M" "$RAND_FILE" | _filter_qemu_io
+
+$QEMU_IMG convert -f raw -O $IMGFMT -c \
+-o "$(_optstr_add "$IMGOPTS" "compression_type=zlib")" "$RAND_FILE" \
+"$TEST_IMG" | _filter_qemu_io
+
+$QEMU_IMG convert -O $IMGFMT -c \
+-o "$(_optstr_add "$IMGOPTS" "compression_type=zstd")" "$TEST_IMG" \
+"$COMPR_IMG" | _filter_qemu_io
+
+$QEMU_IMG compare "$TEST_IMG" "$COMPR_IMG"
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/287.out new file +67 -0

67 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 435dccd5af..3bbe02c88d 100644
@@ -294,6 +294,7 @@
 283 auto quick
 284 rw
 286 rw quick
+287 auto quick
 288 quick
 289 rw quick
 290 rw auto quick
Signature · 1 line
2.17.0
#0 lore raw

v22 cover implement zstd cluster compression method

v22:
03: remove assignemnt in if condition

v21:
03:
* remove the loop on compression [Max]
* use designated initializers [Max]
04:
* don't erase user's options [Max]
* use _rm_test_img [Max]
* add unsupported qcow2 options [Max]

v20:
04: fix a number of flaws [Vladimir]
* don't use $RAND_FILE passing to qemu-io,
so check $TEST_DIR is redundant
* re-arrage $RAND_FILE writing
* fix a typo

v19:
04: fix a number of flaws [Eric]
* remove rudundant test case descriptions
* fix stdout redirect
* don't use (())
* use peek_file_be instead of od
* check $TEST_DIR for spaces and other before using
* use $RAND_FILE safer

v18:
* 04: add quotes to all file name variables [Vladimir]
* 04: add Vladimir's comment according to "qemu-io write -s" option issue.

v17:
* 03: remove incorrect comment in zstd decompress [Vladimir]
* 03: remove "paraniod" and rewrite the comment on decompress [Vladimir]
* 03: fix dead assignment [Vladimir]
* 04: add and remove quotes [Vladimir]
* 04: replace long offset form with the short one [Vladimir]

v16:
* 03: ssize_t for ret, size_t for zstd_ret [Vladimir]
* 04: small fixes according to the comments [Vladimir]

v15:
* 01: aiming qemu 5.1 [Eric]
* 03: change zstd_res definition place [Vladimir]
* 04: add two new test cases [Eric]
1. test adjacent cluster compression with zstd
2. test incompressible cluster processing
* 03, 04: many rewording and gramma fixing [Eric]

v14:
* fix bug on compression - looping until compress == 0 [Me]
* apply reworked Vladimir's suggestions:
1. not mixing ssize_t with size_t
2. safe check for ENOMEM in compression part - avoid overflow
3. tolerate sanity check allow zstd to make progress only on one of the buffers
v13:
* 03: add progress sanity check to decompression loop [Vladimir] 03: add successful decompression check [Me]

v12:
* 03: again, rework compression and decompression loops
to make them more correct [Vladimir]
03: move assert in compression to more appropriate place
[Vladimir]
v11:
* 03: the loops don't need "do{}while" form anymore and
the they were buggy (missed "do" in the beginning)
replace them with usual "while(){}" loops [Vladimir]
v10:
* 03: fix zstd (de)compressed loops for multi-frame
cases [Vladimir]
v9:
* 01: fix error checking and reporting in qcow2_amend compression type part [Vladimir]
* 03: replace asserts with -EIO in qcow2_zstd_decompression [Vladimir, Alberto]
* 03: reword/amend/add comments, fix typos [Vladimir]

v8:
* 03: switch zstd API from simple to stream [Eric] No need to state a special cluster layout for zstd compressed clusters.
v7:
* use qapi_enum_parse instead of the open-coding [Eric]
* fix wording, typos and spelling [Eric]

v6:
* "block/qcow2-threads: fix qcow2_decompress" is removed from the series since it has been accepted by Max already
* add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto]
* report error on qcow2 amending when the compression type is actually chnged [Max]
* remove the extra space and the extra new line [Max]
* re-arrange acks and signed-off-s [Vladimir]

v5:
* replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir]
* set cluster size for all test cases in the beginning of the 287 test

v4:
* the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress"
* 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate

v3:
* remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything)
* prevent compression type changing on "qemu-img amend" [Vladimir]
* remove zstd config setting, since it has been added already by "migration" patches [Vladimir]
* change the compression type error message [Vladimir]
* fix alignment and 80-chars exceeding [Vladimir]

v2:
* rework compression type setting [Vladimir]
* squash iotest changes to the compression type introduction patch [Vladimir, Eric]
* fix zstd availability checking in zstd iotest [Vladimir]
* remove unnecessry casting [Eric]
* remove rudundant checks [Eric]
* fix compressed cluster layout in qcow2 spec [Vladimir]
* fix wording [Eric, Vladimir]
* fix compression type filtering in iotests [Eric]

v1:
the initial series

Denis Plotnikov (4):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: 287: add qcow2 compression type test

 docs/interop/qcow2.txt           |   1 +
 configure                        |   2 +-
 qapi/block-core.json             |  23 ++-
 block/qcow2.h                    |  20 ++-
 include/block/block_int.h        |   1 +
 block/qcow2-threads.c            | 240 +++++++++++++++++++++++++++++--
 block/qcow2.c                    | 120 ++++++++++++++++
 slirp                            |   2 +-
 tests/qemu-iotests/031.out       |  14 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++++++-------
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  34 +++--
 tests/qemu-iotests/065           |  28 ++--
 tests/qemu-iotests/080           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/287           | 152 ++++++++++++++++++++
 tests/qemu-iotests/287.out       |  67 +++++++++
 tests/qemu-iotests/common.filter |   3 +-
 tests/qemu-iotests/group         |   1 +
 23 files changed, 727 insertions(+), 109 deletions(-)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
Signature · 1 line
2.17.0
#4 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Denis Plotnikov
Quoted 41 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---

block/qcow2-threads.c +4 -0
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    ssize_t ret;

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 237

Quoted 10 lines
block/qcow2-threads.c +9 -0
+    size_t zstd_ret;
+    ZSTD_outBuffer output = {
+        .dst = dest,
+        .size = dest_size,
+        .pos = 0
+    };
 
+
+    /* make sure we can safely return compressed buffer size with ssize_t */
+    assert(output.pos <= SSIZE_MAX);

Seems rather vague, since we know that .pos won't exceed .size which was initialized by dest_size which is <= 2M. A tighter assertion:

assert(output.pos <= dest_size)

seems like it is more realistic to your real constraint (namely, that zstd did not overflow dest).

block/qcow2-threads.c line 1

Quoted 11 lines
block/qcow2-threads.c +8 -1
+    ret = output.pos;
+out:
+    ZSTD_freeCCtx(cctx);
+    return ret;
+}
+
 
+++ b/slirp
@@ -1 +1 @@
-Subproject commit 2faae0f778f818fadc873308f983289df697eb93
+Subproject commit 55ab21c9a36852915b81f1b41ebaf3b6509dd8ba

Umm, you definitely don't want that.

A maintainer could touch up both of those, but I'm not sure which block maintainer will be accepting this series. Maybe wait for that question to be answered before trying to post a v23.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Reviewed-by on v22 4/4
#18 lore raw

Re: [PATCH v22 4/4] iotests: 287: add qcow2 compression type test

In reply to Denis Plotnikov
Quoted 7 lines · Denis Plotnikov

The test checks fulfilling qcow2 requirements for the compression type feature and zstd compression type operability.

---

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#5 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Denis Plotnikov
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
 docs/interop/qcow2.txt |   1 +
 configure              |   2 +-
 qapi/block-core.json   |   3 +-
 block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          |   7 ++
 slirp                  |   2 +-
 6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,

Quoted 3 lines
block/qcow2-threads.c +3 -0
+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

Max

#19 lore raw

Re: [PATCH v22 4/4] iotests: 287: add qcow2 compression type test

In reply to Denis Plotnikov
Quoted 54 lines · Denis Plotnikov

The test checks fulfilling qcow2 requirements for the compression type feature and zstd compression type operability.

---
 tests/qemu-iotests/287     | 152 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/287.out |  67 ++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 220 insertions(+)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +34 -0
new file mode 100755
index 0000000000..21fe1f19f5
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2

tests/qemu-iotests/287 line 44

Quoted 10 lines
tests/qemu-iotests/287 new file +10 -0
+_supported_proto file
+_supported_os Linux
+_unsupported_imgopts 'compat=0.10' data_file
+
+COMPR_IMG="$TEST_IMG.compressed"
+RAND_FILE="$TEST_DIR/rand_data"
+
+_cleanup()
+{
+	_rm_test_img

_rm_test_img needs an argument (it basically replaces “rm”). What I thus meant was to keep the _cleanup_test_img here (that was completely correct), but...

tests/qemu-iotests/287 line 45

Quoted 1 line
tests/qemu-iotests/287 new file +1 -0
+	rm -f "$COMPR_IMG"

...to use “_rm_test_img "$COMPR_IMG"” here instead of rm.

Max

#6 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Max Reitz
Quoted 60 lines · Max Reitz
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
  docs/interop/qcow2.txt |   1 +
  configure              |   2 +-
  qapi/block-core.json   |   3 +-
  block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++
  block/qcow2.c          |   7 ++
  slirp                  |   2 +-
  6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 5 lines

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,

Quoted 3 lines
block/qcow2-threads.c +3 -0
+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification. Hmm. If at some point we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will allowing multiple frames help? I don't know actually, but I think better not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups) we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Signature · 2 lines
Best regards,
Vladimir
#15 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Max Reitz
Quoted 57 lines · Max Reitz
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
  docs/interop/qcow2.txt |   1 +
  configure              |   2 +-
  qapi/block-core.json   |   3 +-
  block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++
  block/qcow2.c          |   7 ++
  slirp                  |   2 +-
  6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 4 lines

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,

Quoted 3 lines
block/qcow2-threads.c +3 -0
+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

Potentially, it can, if another implemention of qcow2 saves a couple of frames for some reason.

Denis

Quoted 3 lines

Max

#20 lore raw

Re: [PATCH v22 4/4] iotests: 287: add qcow2 compression type test

In reply to Max Reitz
Quoted 69 lines · Max Reitz
Quoted 54 lines · Denis Plotnikov

The test checks fulfilling qcow2 requirements for the compression type feature and zstd compression type operability.

---
  tests/qemu-iotests/287     | 152 +++++++++++++++++++++++++++++++++++++
  tests/qemu-iotests/287.out |  67 ++++++++++++++++
  tests/qemu-iotests/group   |   1 +
  3 files changed, 220 insertions(+)

create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out

tests/qemu-iotests/287 new file +34 -0
new file mode 100755
index 0000000000..21fe1f19f5
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2

tests/qemu-iotests/287 line 44

Quoted 10 lines
tests/qemu-iotests/287 new file +10 -0
+_supported_proto file
+_supported_os Linux
+_unsupported_imgopts 'compat=0.10' data_file
+
+COMPR_IMG="$TEST_IMG.compressed"
+RAND_FILE="$TEST_DIR/rand_data"
+
+_cleanup()
+{
+	_rm_test_img

_rm_test_img needs an argument (it basically replaces “rm”). What I thus meant was to keep the _cleanup_test_img here (that was completely correct), but...

Quoted 4 lines

tests/qemu-iotests/287 line 45

Quoted 1 line
tests/qemu-iotests/287 new file +1 -0
+	rm -f "$COMPR_IMG"

...to use “_rm_test_img "$COMPR_IMG"” here instead of rm.

Max

ok, I got it.

Denis

#7 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Vladimir Sementsov-Ogievskiy
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 60 lines · Max Reitz
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
   time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
   time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

            compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
  docs/interop/qcow2.txt |   1 +
  configure              |   2 +-
  qapi/block-core.json   |   3 +-
  block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c          |   7 ++
  slirp                  |   2 +-

  6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

+    /*
+     * The compressed stream from the input buffer may consist of more +     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 8 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will allowing multiple frames help? I don't know actually, but I think better not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups) we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Is it really that important? Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once. So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Max

#16 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Denis Plotnikov
Quoted 60 lines · Denis Plotnikov
Quoted 57 lines · Max Reitz
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
   time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
   time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

            compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
  docs/interop/qcow2.txt |   1 +
  configure              |   2 +-
  qapi/block-core.json   |   3 +-
  block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c          |   7 ++
  slirp                  |   2 +-

  6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

+    /*
+     * The compressed stream from the input buffer may consist of more +     * than one zstd frame.

Can it?

Potentially, it can, if another implemention of qcow2 saves a couple of frames for some reason.

Well, it can’t do that if qemu cannot decode it. Maybe there should be a note in the specification on the precise format to expect.

If we decide that it might make sense for someone to encode a cluster with multiple frames, then OK.

Max

#8 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Max Reitz
Quoted 82 lines · Max Reitz
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 60 lines · Max Reitz
Quoted 43 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
   time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img
decompress cmd
   time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

            compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
   docs/interop/qcow2.txt |   1 +
   configure              |   2 +-
   qapi/block-core.json   |   3 +-
   block/qcow2-threads.c  | 169 +++++++++++++++++++++++++++++++++++++++++
   block/qcow2.c          |   7 ++
   slirp                  |   2 +-
   6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 3 lines
block/qcow2-threads.c +3 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

+    /*
+     * The compressed stream from the input buffer may consist of more +     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 8 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will allowing multiple frames help? I don't know actually, but I think better not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups) we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 2 lines

Is it really that important? Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Quoted 6 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once. So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Yes it works in this way already :)

===

So, we don't know do we want one frame restriction or not. Do you have a preference?

Signature · 2 lines
Best regards,
Vladimir
#9 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

So, we don't know do we want one frame restriction or not. Do you have a preference?

I'm a fan of 'be strict in what you produce, liberal in what you accept'. While our implementation always produces only a single frame of compressed data, I think our decoder should be prepared to see more than one frame from other implementations, as that is more liberal than tightening the specification to require that encoding must produce exactly one frame.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#10 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

In reply to Vladimir Sementsov-Ogievskiy
Quoted 88 lines · Vladimir Sementsov-Ogievskiy
Quoted 87 lines · Max Reitz
Quoted 64 lines · Vladimir Sementsov-Ogievskiy
Quoted 63 lines · Max Reitz
Quoted 45 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
    time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
                    src.img [zlib|zstd]_compressed.img decompress cmd
    time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

             compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
   docs/interop/qcow2.txt |   1 +
   configure              |   2 +-
   qapi/block-core.json   |   3 +-
   block/qcow2-threads.c  | 169
+++++++++++++++++++++++++++++++++++++++++
   block/qcow2.c          |   7 ++
   slirp                  |   2 +-

   6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 4 lines

+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size, +                                     const void *src, size_t src_size)
+{

[...]

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 8 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will allowing multiple frames help? I don't know actually, but I think better not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups) we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 5 lines
Quoted 2 lines

Is it really that important?  Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Ah, good. I forgot.

Quoted 7 lines
Quoted 5 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once.  So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Yes it works in this way already :)

Well, OK then.

Quoted 2 lines

So, we don't know do we want one frame restriction or not. Do you have a preference?

*shrug*

Seems like it would be preferential to allow multiple frames still. A note in the spec would be nice (i.e., streaming format, multiple frames per cluster possible).

Max

#11 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

Quoted 96 lines · Max Reitz
Quoted 81 lines · Vladimir Sementsov-Ogievskiy
Quoted 80 lines · Max Reitz
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 60 lines · Max Reitz
Quoted 45 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
    time ./qemu-img convert -O qcow2 -c -o
compression_type=[zlib|zstd]
                    src.img [zlib|zstd]_compressed.img
decompress cmd
    time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

             compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
    docs/interop/qcow2.txt |   1 +
    configure              |   2 +-
    qapi/block-core.json   |   3 +-
    block/qcow2-threads.c  | 169
+++++++++++++++++++++++++++++++++++++++++
    block/qcow2.c          |   7 ++
    slirp                  |   2 +-
    6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 4 lines

+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size, +                                     const void *src, size_t src_size)
+{

[...]

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 8 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will allowing multiple frames help? I don't know actually, but I think better not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups) we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 4 lines
Quoted 2 lines

Is it really that important?  Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Ah, good. I forgot.

Quoted 6 lines
Quoted 5 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once.  So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Yes it works in this way already :)

Well, OK then.

Quoted 7 lines
Quoted 2 lines

So, we don't know do we want one frame restriction or not. Do you have a preference?

*shrug*

Seems like it would be preferential to allow multiple frames still. A note in the spec would be nice (i.e., streaming format, multiple frames per cluster possible).

We don't mention anything about zlib compressing details in the spec.

If we mention anything about ZSTD compressing details we'll have to do it for zlib as well. So, I think we have two possibilities for the spec:
1. mention for both
2. don't mention at all

I think the 2nd is better. It gives more degree of freedom for the future improvements. and we've already covered both cases (one frame, may frames) in the code. I'm note sure I'm right. Any other opinions?

Denis

Quoted 3 lines

Max

#12 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

Quoted 104 lines · Denis Plotnikov
Quoted 101 lines · Max Reitz
Quoted 84 lines · Vladimir Sementsov-Ogievskiy
Quoted 83 lines · Max Reitz
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 60 lines · Max Reitz
Quoted 45 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
     time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
                     src.img [zlib|zstd]_compressed.img decompress cmd
     time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

              compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
    docs/interop/qcow2.txt |   1 +
    configure              |   2 +-
    qapi/block-core.json   |   3 +-
    block/qcow2-threads.c  | 169
+++++++++++++++++++++++++++++++++++++++++
    block/qcow2.c          |   7 ++
    slirp                  |   2 +-

    6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 4 lines

+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size, +                                     const void *src, size_t src_size)
+{

[...]

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 11 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will
allowing multiple frames help? I don't know actually, but I think better
not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups)
we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 4 lines
Quoted 2 lines

Is it really that important?  Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Ah, good.  I forgot.

Quoted 6 lines
Quoted 5 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once.  So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Yes it works in this way already :)

Well, OK then.

Quoted 2 lines

So, we don't know do we want one frame restriction or not. Do you have a preference?

Quoted 7 lines
Quoted 5 lines

*shrug*

Seems like it would be preferential to allow multiple frames still.  A note in the spec would be nice (i.e., streaming format, multiple frames per cluster possible).

We don't mention anything about zlib compressing details in the spec.

Yep. Which is bad enough.

Quoted 3 lines

If we mention anything about ZSTD compressing details we'll have to do it for
zlib as well.

I personally don’t like “If you can’t make it perfect, you shouldn’t do it at all”. Mentioning it for zstd would still be an improvement.

Also, I believe that “zlib compression” is pretty much unambiguous, considering all the code does is call deflate()/inflate().

I’m not saying we need to amend the spec in this series, but I don’t see a good argument against doing so at some point.

Quoted 6 lines

So, I think we have two possibilities for the spec:
1. mention for both
2. don't mention at all

I think the 2nd is better. It gives more degree of freedom for the future improvements.

No, it absolutely doesn’t. There is a de-facto format, namely what qemu accepts. Changing that would be an incompatible change. Just because we don’t write what’s allowed into the spec doesn’t change that fact.

Quoted 1 line

and we've already covered both cases (one frame, may frames) in the code.

There are still different zstd formats, namely streaming or not streaming. That’s what should be mentioned.

Max

#13 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

Quoted 109 lines · Max Reitz
Quoted 108 lines · Denis Plotnikov
Quoted 106 lines · Max Reitz
Quoted 84 lines · Vladimir Sementsov-Ogievskiy
Quoted 83 lines · Max Reitz
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 60 lines · Max Reitz
Quoted 45 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
     time ./qemu-img convert -O qcow2 -c -o
compression_type=[zlib|zstd]
                     src.img [zlib|zstd]_compressed.img
decompress cmd
     time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

              compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
     docs/interop/qcow2.txt |   1 +
     configure              |   2 +-
     qapi/block-core.json   |   3 +-
     block/qcow2-threads.c  | 169
+++++++++++++++++++++++++++++++++++++++++
     block/qcow2.c          |   7 ++
     slirp                  |   2 +-
     6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 4 lines

+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size, +                                     const void *src, size_t src_size)
+{

[...]

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 11 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could this be implemented with zstd lib, if multiple frames are allowed, will
allowing multiple frames help? I don't know actually, but I think better
not to forbid it. On the other hand, I don't see any benefit in large compressed clusters. At least, in our scenarios (for compressed backups)
we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 4 lines
Quoted 2 lines

Is it really that important?  Naïvely, it sounds rather complicated to introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Ah, good.  I forgot.

Quoted 6 lines
Quoted 5 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once.  So parallelism on the cluster level should sufficient to get high usage, and it would benefit all compression types and cluster sizes.)

Yes it works in this way already :)

Well, OK then.

Quoted 2 lines

So, we don't know do we want one frame restriction or not. Do you have a preference?

*shrug*

Seems like it would be preferential to allow multiple frames still.  A note in the spec would be nice (i.e., streaming format, multiple frames per cluster possible).

Quoted 8 lines
Quoted 1 line

We don't mention anything about zlib compressing details in the spec.

Yep. Which is bad enough.

Quoted 3 lines

If we mention anything about ZSTD compressing details we'll have to do it for
zlib as well.

I personally don’t like “If you can’t make it perfect, you shouldn’t do it at all”. Mentioning it for zstd would still be an improvement.

Good approach. I like it. But I'm trying to be cautious.

Quoted 13 lines

Also, I believe that “zlib compression” is pretty much unambiguous, considering all the code does is call deflate()/inflate().

I’m not saying we need to amend the spec in this series, but I don’t see a good argument against doing so at some point.

Quoted 6 lines

So, I think we have two possibilities for the spec:
1. mention for both
2. don't mention at all

I think the 2nd is better. It gives more degree of freedom for the future improvements.

Quoted 7 lines

No, it absolutely doesn’t. There is a de-facto format, namely what qemu accepts. Changing that would be an incompatible change. Just because we don’t write what’s allowed into the spec doesn’t change that fact.

Quoted 1 line

and we've already covered both cases (one frame, may frames) in the code.

There are still different zstd formats, namely streaming or not streaming. That’s what should be mentioned.

It looks like the ZSTD format is always the same.
The ZSTD interface is different: for some reason the simple zstd_decompress() wants to know the size of data to decompress

From zstd doc:
*size_t ZSTD_decompress( void* dst, size_t dstCapacity, const void* src, size_t compressedSize); *

compressedSize : must be the _exact_ size of some number of compressed and/or skippable frames.

I made a test (based on Vladimir's code) to check it:

// compile with gcc -lzstd -g test_zstd.c -o test_zstd

#include <stdio.h> #include <assert.h> #include <zstd.h> #include <zstd_errors.h> int main() { char buf1[] = "erbebewbwe"; char buf2[100]; char buf3[100]; int ret; ZSTD_outBuffer output; ZSTD_inBuffer input; ret = ZSTD_compress(buf2, sizeof(buf2), buf1, sizeof(buf1), 5); printf("ret: %d\n", ret); ZSTD_DCtx *dctx = ZSTD_createDCtx(); input = (ZSTD_inBuffer){buf2, ret, 0}; output = (ZSTD_outBuffer){buf3, sizeof(buf3), 0}; ret = ZSTD_decompressStream(dctx, &output, &input); printf("ret: %d, input.pos: %ld, output.pos: %ld\n", ret, input.pos, output.pos); printf("To compress: %s\n", buf1); buf3[output.pos] = 0; printf("Uncompressed: %s\n", buf3); return 0; }

And it works fine.

We use streaming semantics just to be consistent with the interfaces (stream_compress/stream_decompress), otherwise

we could use simple_compress/stream_decompress

Denis

Quoted 3 lines

Max

#14 lore raw

Re: [PATCH v22 3/4] qcow2: add zstd cluster compression

Quoted 155 lines · Denis Plotnikov
Quoted 115 lines · Max Reitz
Quoted 114 lines · Denis Plotnikov
Quoted 112 lines · Max Reitz
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 62 lines · Vladimir Sementsov-Ogievskiy
Quoted 61 lines · Max Reitz
Quoted 45 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
      time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
                      src.img [zlib|zstd]_compressed.img decompress cmd
      time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.img

               compression               decompression zlib       zstd           zlib         zstd
------------------------------------------------------------
real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %) user     65.0       15.8            5.3          2.5 sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
     docs/interop/qcow2.txt |   1 +
     configure              |   2 +-
     qapi/block-core.json   |   3 +-
     block/qcow2-threads.c  | 169
+++++++++++++++++++++++++++++++++++++++++
     block/qcow2.c          |   7 ++
     slirp                  |   2 +-

     6 files changed, 181 insertions(+), 3 deletions(-)

[...]

Quoted 4 lines
block/qcow2-threads.c +0 -0
index 7dbaf53489..a0b12e1b15 100644

[...]

Quoted 5 lines

+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{

[...]

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

+    /*
+     * The compressed stream from the input buffer may consist of more
+     * than one zstd frame.

Can it?

If not, we must require it in the specification.

Actually, now that you mention it, it would make sense anyway to add some note to the specification on what exactly compressed with zstd means.

Quoted 13 lines

Hmm. If at some point
we'll want multi-threaded compression of one big (2M) cluster.. Could
this be implemented with zstd lib, if multiple frames are allowed, will
allowing multiple frames help? I don't know actually, but I think better
not to forbid it. On the other hand, I don't see any benefit in large
compressed clusters. At least, in our scenarios (for compressed backups)
we use 64k compressed clusters, for good granularity of incremental backups (when for running vm we use 1M clusters).

Quoted 5 lines
Quoted 3 lines

Is it really that important?  Naïvely, it sounds rather complicated to
introduce multithreading into block drivers.

It is already here: compression and encryption already multithreaded. But of course, one cluster is handled in one thread.

Ah, good.  I forgot.

Quoted 7 lines
Quoted 6 lines

(Also, as for compression, it can only be used in backup scenarios anyway, where you write many clusters at once.  So parallelism on the cluster level should sufficient to get high usage, and it would benefit
all compression types and cluster sizes.)

Yes it works in this way already :)

Well, OK then.

Quoted 3 lines

So, we don't know do we want one frame restriction or not. Do you have a
preference?

*shrug*

Seems like it would be preferential to allow multiple frames still.  A note in the spec would be nice (i.e., streaming format, multiple frames per cluster possible).

Quoted 8 lines
Quoted 1 line

We don't mention anything about zlib compressing details in the spec.

Yep.  Which is bad enough.

Quoted 3 lines

If we mention anything about ZSTD compressing details we'll have to do it for
zlib as well.

I personally don’t like “If you can’t make it perfect, you shouldn’t do it at all”.  Mentioning it for zstd would still be an improvement.

Good approach. I like it. But I'm trying to be cautious.

Quoted 13 lines

Also, I believe that “zlib compression” is pretty much unambiguous, considering all the code does is call deflate()/inflate().

I’m not saying we need to amend the spec in this series, but I don’t see a good argument against doing so at some point.

Quoted 6 lines

So, I think we have two possibilities for the spec:
1. mention for both
2. don't mention at all

I think the 2nd is better. It gives more degree of freedom for the future improvements.

Quoted 8 lines

No, it absolutely doesn’t.  There is a de-facto format, namely what qemu accepts.  Changing that would be an incompatible change.  Just because we don’t write what’s allowed into the spec doesn’t change that fact.

Quoted 2 lines

and we've already covered both cases (one frame, may frames) in the code.

There are still different zstd formats, namely streaming or not streaming.  That’s what should be mentioned.

It looks like the ZSTD format is always the same. The ZSTD interface is different: for some reason the simple zstd_decompress()
wants to know the size of data to decompress

Quoted 6 lines

From zstd doc:
*size_t ZSTD_decompress( void* dst, size_t dstCapacity, const void* src, size_t compressedSize); *

compressedSize : must be the _exact_ size of some number of compressed and/or skippable frames.

That’s... interesting.

Quoted 15 lines

I made a test (based on Vladimir's code) to check it:

// compile with gcc -lzstd -g test_zstd.c -o test_zstd

#include <stdio.h> #include <assert.h> #include <zstd.h> #include <zstd_errors.h> int main() { char buf1[] = "erbebewbwe"; char buf2[100]; char buf3[100]; int ret; ZSTD_outBuffer output; ZSTD_inBuffer input; ret = ZSTD_compress(buf2, sizeof(buf2), buf1, sizeof(buf1), 5); printf("ret:
%d\n", ret); ZSTD_DCtx *dctx = ZSTD_createDCtx(); input = (ZSTD_inBuffer){buf2, ret, 0}; output = (ZSTD_outBuffer){buf3, sizeof(buf3), 0}; ret = ZSTD_decompressStream(dctx, &output, &input); printf("ret: %d, input.pos: %ld, output.pos: %ld\n", ret, input.pos, output.pos); printf("To compress: %s\n", buf1); buf3[output.pos] = 0; printf("Uncompressed: %s\n", buf3); return 0; }

Quoted 6 lines

And it works fine.

We use streaming semantics just to be consistent with the interfaces (stream_compress/stream_decompress), otherwise

we could use simple_compress/stream_decompress

OK, if there’s just a single format and we accept multiple frames, then there’s indeed no need to document it.

Max