Skip to contentSkip to conversation

qcow2: Implement zstd cluster compression method

PATCH Superseded Denis Plotnikov 2 participants 9 messages Mar 12, 2020 – Mar 16, 2020 Block layer

v6 of 24 Superseded Latest is v24 →

lore
Share & tools
lore
#0 lore raw

v6 cover qcow2: Implement zstd cluster compression method

v6:
* "block/qcow2-threads: fix qcow2_decompress" is removed from the series since it has been accepted by Max already
* add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto]
* report error on qcow2 amending when the compression type is actually chnged [Max]
* remove the extra space and the extra new line [Max]
* re-arrange acks and signed-off-s [Vladimir]

v5:
* replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir]
* set cluster size for all test cases in the beginning of the 287 test

v4:
* the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress"
* 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate

v3:
* remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything)
* prevent compression type changing on "qemu-img amend" [Vladimir]
* remove zstd config setting, since it has been added already by "migration" patches [Vladimir]
* change the compression type error message [Vladimir]
* fix alignment and 80-chars exceeding [Vladimir]

v2:
* rework compression type setting [Vladimir]
* squash iotest changes to the compression type introduction patch [Vladimir, Eric]
* fix zstd availability checking in zstd iotest [Vladimir]
* remove unnecessry casting [Eric]
* remove rudundant checks [Eric]
* fix compressed cluster layout in qcow2 spec [Vladimir]
* fix wording [Eric, Vladimir]
* fix compression type filtering in iotests [Eric]

v1:
the initial series

Denis Plotnikov (4):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: 287: add qcow2 compression type test

 docs/interop/qcow2.txt           |  20 ++++
 configure                        |   2 +-
 qapi/block-core.json             |  23 +++-
 block/qcow2.h                    |  20 +++-
 include/block/block_int.h        |   1 +
 block/qcow2-threads.c            | 195 +++++++++++++++++++++++++++++--
 block/qcow2.c                    | 132 +++++++++++++++++++++
 tests/qemu-iotests/031.out       |  14 +--
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++++++++--------
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  34 +++---
 tests/qemu-iotests/065           |  28 +++--
 tests/qemu-iotests/080           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/287           | 128 ++++++++++++++++++++
 tests/qemu-iotests/287.out       |  43 +++++++
 tests/qemu-iotests/common.filter |   3 +-
 tests/qemu-iotests/group         |   1 +
 22 files changed, 664 insertions(+), 108 deletions(-)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
Signature · 1 line
2.17.0
#1 lore raw

PATCH v6 1/4 qcow2: introduce compression type feature

The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.

The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.

The tests are fixed in the following ways:

    * filter out compression_type for all the tests
    * fix header size, feature table size and backing file offset
      affected tests: 031, 036, 061, 080
      header_size +=8: 1 byte compression type
                       7 bytes padding
      feature_table += 48: incompatible feture compression type
      backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
    * add "compression type" for test output matching when it isn't filtered
      affected tests: 049, 060, 061, 065, 144, 182, 242, 255
---
 qapi/block-core.json             |  22 +++++-
 block/qcow2.h                    |  20 ++++-
 include/block/block_int.h        |   1 +
 block/qcow2.c                    | 121 +++++++++++++++++++++++++++++++
 tests/qemu-iotests/031.out       |  14 ++--
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 +++++++++++++-------------
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061.out       |  34 +++++----
 tests/qemu-iotests/065           |  28 ++++---
 tests/qemu-iotests/080           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/242.out       |   5 ++
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/common.filter |   3 +-
 16 files changed, 275 insertions(+), 96 deletions(-)
qapi/block-core.json +20 -2
index 85e27bb61f..a306484973 100644
@@ -78,6 +78,8 @@
 #
 # @bitmaps: A list of qcow2 bitmap details (since 4.0)
 #
+# @compression-type: the image cluster compression method (since 5.0)
+#
 # Since: 1.7
 ##
 { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
       '*corrupt': 'bool',
       'refcount-bits': 'int',
       '*encrypt': 'ImageInfoSpecificQCow2Encryption',
-      '*bitmaps': ['Qcow2BitmapInfo']
+      '*bitmaps': ['Qcow2BitmapInfo'],
+      'compression-type': 'Qcow2CompressionType'
   } }
 
 ##
@@ -4392,6 +4395,18 @@
   'data': [ 'v2', 'v3' ] }
 
 
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+  'data': [ 'zlib' ] }
+
 ##
 # @BlockdevCreateOptionsQcow2:
 #
@@ -4415,6 +4430,8 @@
 #                 allowed values: off, falloc, full, metadata)
 # @lazy-refcounts: True if refcounts may be updated lazily (default: off)
 # @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+#                    (default: zlib, since 5.0)
 #
 # Since: 2.12
 ##
@@ -4430,7 +4447,8 @@
             '*cluster-size':    'size',
             '*preallocation':   'PreallocMode',
             '*lazy-refcounts':  'bool',
-            '*refcount-bits':   'int' } }
+            '*refcount-bits':   'int',
+            '*compression-type':'Qcow2CompressionType' } }
 
 ##
 # @BlockdevCreateOptionsQed:
block/qcow2.h +19 -1
index 0942126232..e434cff9ff 100644
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
 
     uint32_t refcount_order;
     uint32_t header_length;
+
+    /* Additional fields */
+    uint8_t  compression_type;
+
+    /* header must be a multiple of 8 */
+    uint8_t  padding[7];
 } QEMU_PACKED QCowHeader;
 
+QEMU_BUILD_BUG_ON(sizeof(QCowHeader) % 8 != 0);
+
 typedef struct QEMU_PACKED QCowSnapshotHeader {
     /* header is 8 byte aligned */
     uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {
     QCOW2_INCOMPAT_DIRTY_BITNR      = 0,
     QCOW2_INCOMPAT_CORRUPT_BITNR    = 1,
     QCOW2_INCOMPAT_DATA_FILE_BITNR  = 2,
+    QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
     QCOW2_INCOMPAT_DIRTY            = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
     QCOW2_INCOMPAT_CORRUPT          = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
     QCOW2_INCOMPAT_DATA_FILE        = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+    QCOW2_INCOMPAT_COMPRESSION      = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
 
     QCOW2_INCOMPAT_MASK             = QCOW2_INCOMPAT_DIRTY
                                     | QCOW2_INCOMPAT_CORRUPT
-                                    | QCOW2_INCOMPAT_DATA_FILE,
+                                    | QCOW2_INCOMPAT_DATA_FILE
+                                    | QCOW2_INCOMPAT_COMPRESSION,
 };
 
 /* Compatible feature bits */
@@ -369,6 +380,13 @@ typedef struct BDRVQcow2State {
 
     bool metadata_preallocation_checked;
     bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set
+     */
+    Qcow2CompressionType compression_type;
 } BDRVQcow2State;
 
 typedef struct Qcow2COWRegion {
include/block/block_int.h +1 -0
index 6f9fd5e20e..2c6bb9dc99 100644
@@ -57,6 +57,7 @@
 #define BLOCK_OPT_REFCOUNT_BITS     "refcount_bits"
 #define BLOCK_OPT_DATA_FILE         "data_file"
 #define BLOCK_OPT_DATA_FILE_RAW     "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE  "compression_type"
 
 #define BLOCK_PROBE_BUF_SIZE        512
 
block/qcow2.c +121 -0

199 lines not on this page · raw message

tests/qemu-iotests/031.out +7 -7
index d535e407bc..ed51afe9ce 100644
@@ -113,11 +113,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 Header extension:
@@ -146,11 +146,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
 
 magic                     0x514649fb
 version                   3
-backing_file_offset       0x178
+backing_file_offset       0x1b0
 backing_file_size         0x17
 cluster_bits              16
 size                      67108864
@@ -179,7 +179,7 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0xe2792aca
@@ -188,7 +188,7 @@ data                      'host_device'
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 Header extension:
tests/qemu-iotests/036.out +2 -2
index 0b52b934e1..fb509f6357 100644
@@ -26,7 +26,7 @@ compatible_features       []
 autoclear_features        [63]
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 
@@ -38,7 +38,7 @@ compatible_features       []
 autoclear_features        []
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 *** done
tests/qemu-iotests/049.out +51 -51

225 lines not on this page · raw message

tests/qemu-iotests/060.out +1 -0
index d27692a33c..3e47f537e8 100644
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     corrupt: true
tests/qemu-iotests/061.out +20 -14
index 8b3091a412..c913f02ad6 100644
@@ -22,11 +22,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 magic                     0x514649fb
@@ -80,11 +80,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 magic                     0x514649fb
@@ -136,11 +136,11 @@ incompatible_features     [0]
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features     []
 compatible_features       [42]
 autoclear_features        [42]
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 magic                     0x514649fb
@@ -260,11 +260,11 @@ incompatible_features     []
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features     [0]
 compatible_features       [0]
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features     []
 compatible_features       []
 autoclear_features        []
 refcount_order            4
-header_length             104
+header_length             112
 
 Header extension:
 magic                     0x6803f857
-length                    192
+length                    240
 data                      <binary>
 
 read 131072/131072 bytes at offset 0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     data file: TEST_DIR/t.IMGFMT.data
tests/qemu-iotests/065 +18 -10
index 6426474271..18dc488c7a 100755
@@ -88,24 +88,30 @@ class TestQMP(TestImageInfoSpecific):
 class TestQCow2(TestQemuImgInfo):
     '''Testing a qcow2 version 2 image'''
     img_options = 'compat=0.10'
-    json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
-    human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
+    json_compare = { 'compat': '0.10', 'refcount-bits': 16,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 0.10', 'compression type: zlib',
+                      'refcount bits: 16' ]
 
 class TestQCow3NotLazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
     img_options = 'compat=1.1,lazy_refcounts=off'
     json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
-                     'refcount-bits': 16, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
-                      'refcount bits: 16', 'corrupt: false' ]
+                     'refcount-bits': 16, 'corrupt': False,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 1.1', 'compression type: zlib',
+                      'lazy refcounts: false', 'refcount bits: 16',
+                      'corrupt: false' ]
 
 class TestQCow3Lazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
     img_options = 'compat=1.1,lazy_refcounts=on'
     json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
-                     'refcount-bits': 16, 'corrupt': False }
-    human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
-                      'refcount bits: 16', 'corrupt: false' ]
+                     'refcount-bits': 16, 'corrupt': False,
+                     'compression-type': 'zlib' }
+    human_compare = [ 'compat: 1.1', 'compression type: zlib',
+                      'lazy refcounts: true', 'refcount bits: 16',
+                      'corrupt: false' ]
 
 class TestQCow3NotLazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
@@ -113,7 +119,8 @@ class TestQCow3NotLazyQMP(TestQMP):
     img_options = 'compat=1.1,lazy_refcounts=off'
     qemu_options = 'lazy-refcounts=on'
     compare = { 'compat': '1.1', 'lazy-refcounts': False,
-                'refcount-bits': 16, 'corrupt': False }
+                'refcount-bits': 16, 'corrupt': False,
+                'compression-type': 'zlib' }
 
 
 class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +129,8 @@ class TestQCow3LazyQMP(TestQMP):
     img_options = 'compat=1.1,lazy_refcounts=on'
     qemu_options = 'lazy-refcounts=off'
     compare = { 'compat': '1.1', 'lazy-refcounts': True,
-                'refcount-bits': 16, 'corrupt': False }
+                'refcount-bits': 16, 'corrupt': False,
+                'compression-type': 'zlib' }
 
 TestImageInfoSpecific = None
 TestQemuImgInfo = None
tests/qemu-iotests/080 +1 -1
index a3d13c414e..7588c63b6c 100755
@@ -45,7 +45,7 @@ _supported_os Linux
 # - This is generally a test for compat=1.1 images
 _unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10'
 
-header_size=104
+header_size=112
 
 offset_backing_file_offset=8
 offset_backing_file_size=16
tests/qemu-iotests/144.out +2 -2
index c7aa2e4820..885a8874a5 100644
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
 { 'execute': 'qmp_capabilities' }
 {"return": {}}
 { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 
 === Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
 === Performing Live Snapshot 2 ===
 
 { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 *** done
tests/qemu-iotests/182.out +1 -1
index a8eea166c3..ae43654d32 100644
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
 {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
 {"return": {}}
 {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
-Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 {"return": {}}
 {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
 {"return": {}}
tests/qemu-iotests/242.out +5 -0
index 7ac8404d11..091b9126ce 100644
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     refcount bits: 16
     corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
 cluster_size: 65536
 Format specific information:
     compat: 1.1
+    compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
tests/qemu-iotests/255.out +4 -4
index 348909fdef..a3c99fd62e 100644
@@ -3,9 +3,9 @@ Finishing a commit job with background reads
 
 === Create backing chain and start VM ===
 
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 === Start background read requests ===
 
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
 
 === Create images and start VM ===
 
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
 
 wrote 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/common.filter +2 -1
index 3f8ee3e5f7..279e0bbb0d 100644
@@ -152,7 +152,8 @@ _filter_img_create()
         -e "s# refcount_bits=[0-9]\\+##g" \
         -e "s# key-secret=[a-zA-Z0-9]\\+##g" \
         -e "s# iter-time=[0-9]\\+##g" \
-        -e "s# force_size=\\(on\\|off\\)##g"
+        -e "s# force_size=\\(on\\|off\\)##g" \
+        -e "s# compression_type=[a-zA-Z0-9]\\+##g"
 }
 
 _filter_img_info()
Signature · 1 line
2.17.0
#2 lore raw

Re: [PATCH v6 1/4] qcow2: introduce compression type feature

Quoted 15 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.

The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

Quoted 6 lines

Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.

The tests are fixed in the following ways:
* filter out compression_type for all the tests

Presumably this filter is optional, and we will not use it on the specific new tests that prove zstd compression works - but that should be later in the series, so for this patch it is okay.

Quoted 5 lines
     * fix header size, feature table size and backing file offset
       affected tests: 031, 036, 061, 080
       header_size +=8: 1 byte compression type
                        7 bytes padding
       feature_table += 48: incompatible feture compression type

feature

Quoted 1 line
       backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)

feature

(interesting that you have two different changed spellings ;)

Quoted 2 lines

* add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255

Or maybe the comment above should be changed to "many tests" rather than "all the tests".

Quoted 25 lines
---
  qapi/block-core.json             |  22 +++++-
  block/qcow2.h                    |  20 ++++-
  include/block/block_int.h        |   1 +
  block/qcow2.c                    | 121 +++++++++++++++++++++++++++++++
  tests/qemu-iotests/031.out       |  14 ++--
  tests/qemu-iotests/036.out       |   4 +-
  tests/qemu-iotests/049.out       | 102 +++++++++++++-------------
  tests/qemu-iotests/060.out       |   1 +
  tests/qemu-iotests/061.out       |  34 +++++----
  tests/qemu-iotests/065           |  28 ++++---
  tests/qemu-iotests/080           |   2 +-
  tests/qemu-iotests/144.out       |   4 +-
  tests/qemu-iotests/182.out       |   2 +-
  tests/qemu-iotests/242.out       |   5 ++
  tests/qemu-iotests/255.out       |   8 +-
  tests/qemu-iotests/common.filter |   3 +-
  16 files changed, 275 insertions(+), 96 deletions(-)

+++ b/block/qcow2.h

block/qcow2.h typedef struct QCowHeader { · line 154

Quoted 9 lines
block/qcow2.h +6 -0
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
 
      uint32_t refcount_order;
      uint32_t header_length;
+
+    /* Additional fields */
+    uint8_t  compression_type;
+
+    /* header must be a multiple of 8 */
+    uint8_t  padding[7];

Why two spaces after uint8_t (twice)?

block/qcow2.h typedef struct BDRVQcow2State { · line 387

Quoted 9 lines
block/qcow2.h +5 -0
@@ -369,6 +380,13 @@ typedef struct BDRVQcow2State {
 
      bool metadata_preallocation_checked;
      bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the image
+     * with the desired compression type set

Missing trailing '.'. Maybe someday we can get 'qemu-img amend' to also adjust the compression type in-place; if that's something we think we might do, then this could be better worded as "For now, the only way to change...".

Quoted 5 lines
block/qcow2.c +1 -0
+++ b/block/qcow2.c
@@ -1242,6 +1242,48 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
      return ret;
  }
 

block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1254

Quoted 10 lines
block/qcow2.c +9 -0
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
 
+
+static int qcow2_compression_type_from_format(const char *ct)
+{
+    if (g_str_equal(ct, "zlib")) {
+        return QCOW2_COMPRESSION_TYPE_ZLIB;
+    } else {
+        return -EINVAL;
+    }

Why are you open-coding this?

qapi_enum_parse(&Qcow2CompressionType_lookup, ct, -1, errp)

should do what you use this for, and automatically updates itself when you add zstd to the qapi enum later.

block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3496

Quoted 5 lines
block/qcow2.c +1 -0
@@ -3401,6 +3493,8 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
          .refcount_table_offset      = cpu_to_be64(cluster_size),
          .refcount_table_clusters    = cpu_to_be32(1),
          .refcount_order             = cpu_to_be32(refcount_order),
+        /* don't deal with endians since compression_type is 1 byte long */

endianness

block/qcow2.c static QemuOptsList qcow2_create_opts = { · line 5637

Quoted 12 lines
block/qcow2.c +5 -0
+        .compression_type           = compression_type,
          .header_length              = cpu_to_be32(sizeof(*header)),
      };
 
@@ -5516,6 +5631,12 @@ static QemuOptsList qcow2_create_opts = {
              .help = "Width of a reference count entry in bits",
              .def_value_str = "16"
          },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",

s/clusters/cluster/

Overall, getting closer. I assume this would go through Max's tree, if he has time to include it before soft freeze...

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [PATCH v6 1/4] qcow2: introduce compression type feature

Thanks for the comments.
I'll make the fixes accordingly and re-sent the series shortly.

Denis

Quoted 143 lines · Eric Blake
Quoted 15 lines · Denis Plotnikov

The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.

It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.

The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.

The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.

Quoted 6 lines

Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.

The tests are fixed in the following ways:
     * filter out compression_type for all the tests

Presumably this filter is optional, and we will not use it on the specific new tests that prove zstd compression works - but that should be later in the series, so for this patch it is okay.

Quoted 5 lines

     * fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080
       header_size +=8: 1 byte compression type
                        7 bytes padding
       feature_table += 48: incompatible feture compression type

feature

Quoted 1 line

       backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)

feature

(interesting that you have two different changed spellings ;)

Quoted 2 lines

     * add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255

Or maybe the comment above should be changed to "many tests" rather than "all the tests".

Quoted 25 lines

---
  qapi/block-core.json             |  22 +++++- block/qcow2.h                    |  20 ++++- include/block/block_int.h        |   1 +
  block/qcow2.c                    | 121 +++++++++++++++++++++++++++++++

  tests/qemu-iotests/031.out       |  14 ++--
  tests/qemu-iotests/036.out       |   4 +-
  tests/qemu-iotests/049.out       | 102 +++++++++++++-------------
  tests/qemu-iotests/060.out       |   1 +
  tests/qemu-iotests/061.out       |  34 +++++----
  tests/qemu-iotests/065           |  28 ++++---
  tests/qemu-iotests/080           |   2 +-
  tests/qemu-iotests/144.out       |   4 +-
  tests/qemu-iotests/182.out       |   2 +-
  tests/qemu-iotests/242.out       |   5 ++
  tests/qemu-iotests/255.out       |   8 +-
  tests/qemu-iotests/common.filter |   3 +-
  16 files changed, 275 insertions(+), 96 deletions(-)

+++ b/block/qcow2.h

block/qcow2.h typedef struct QCowHeader { · line 153

Quoted 8 lines
block/qcow2.h +6 -0
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
         uint32_t refcount_order;
       uint32_t header_length;
+
+    /* Additional fields */
+    uint8_t  compression_type;
+
+    /* header must be a multiple of 8 */
+    uint8_t  padding[7];

Why two spaces after uint8_t (twice)?

Quoted 9 lines
block/qcow2.h +4 -0
@@ -369,6 +380,13 @@ typedef struct BDRVQcow2State {
         bool metadata_preallocation_checked;
       bool metadata_preallocation;
+    /*
+     * Compression type used for the image. Default: 0 - ZLIB
+     * The image compression type is set on image creation.
+     * The only way to change the compression type is to convert the 

image
+     * with the desired compression type set

Missing trailing '.'.  Maybe someday we can get 'qemu-img amend' to also adjust the compression type in-place; if that's something we think we might do, then this could be better worded as "For now, the only way to change...".

Quoted 5 lines

+++ b/block/qcow2.c

block/qcow2.c +0 -0
@@ -1242,6 +1242,48 @@ static int 
 qcow2_update_options(BlockDriverState *bs, QDict *options,
       return ret;
   }

block/qcow2.c static int · line 1254

Quoted 10 lines
block/qcow2.c +8 -0
   +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
 
+
+static int qcow2_compression_type_from_format(const char *ct)
+{
+    if (g_str_equal(ct, "zlib")) {
+        return QCOW2_COMPRESSION_TYPE_ZLIB;
+    } else {
+        return -EINVAL;
+    }

Why are you open-coding this?

qapi_enum_parse(&Qcow2CompressionType_lookup, ct, -1, errp)

should do what you use this for, and automatically updates itself when you add zstd to the qapi enum later.

Quoted 7 lines
block/qcow2.c +1 -0
@@ -3401,6 +3493,8 @@ qcow2_co_create(BlockdevCreateOptions 
 *create_options, Error **errp)
           .refcount_table_offset      = cpu_to_be64(cluster_size),
           .refcount_table_clusters    = cpu_to_be32(1),
           .refcount_order             = cpu_to_be32(refcount_order),
+        /* don't deal with endians since compression_type is 1 byte 

long */

endianness

Quoted 4 lines

+        .compression_type           = compression_type, .header_length              = cpu_to_be32(sizeof(*header)), };
  @@ -5516,6 +5631,12 @@ static QemuOptsList qcow2_create_opts = {

Quoted 8 lines

              .help = "Width of a reference count entry in bits", .def_value_str = "16"
          },
+        {
+            .name = BLOCK_OPT_COMPRESSION_TYPE,
+            .type = QEMU_OPT_STRING,
+            .help = "Compression method used for image clusters compression",

s/clusters/cluster/

Overall, getting closer. I assume this would go through Max's tree, if he has time to include it before soft freeze...

#4 lore raw

PATCH v6 2/4 qcow2: rework the cluster compression routine

In reply to Denis Plotnikov

The patch enables processing the image compression type defined for the image and chooses an appropriate method for image clusters (de)compression.

---
 block/qcow2-threads.c | 71 ++++++++++++++++++++++++++++++++++++-------
 1 file changed, 60 insertions(+), 11 deletions(-)
block/qcow2-threads.c +60 -11
index a68126f291..7dbaf53489 100644
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
 } Qcow2CompressData;
 
 /*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
  *          -ENOMEM destination buffer is not enough to store compressed data
  *          -EIO    on any other error
  */
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
-                              const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
 {
     ssize_t ret;
     z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
 }
 
 /*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
  *
  * Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
  *
  * @dest - destination buffer, @dest_size bytes
  * @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
  * Returns: 0 on success
  *          -EIO on fail
  */
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
-                                const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
 {
     int ret;
     z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
     return arg.ret;
 }
 
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ *          a negative error code on failure
+ */
 ssize_t coroutine_fn
 qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
                   const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_compress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_compress;
+        break;
+
+    default:
+        abort();
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          a negative error code on failure
+ */
 ssize_t coroutine_fn
 qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
                     const void *src, size_t src_size)
 {
-    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
-                                qcow2_decompress);
+    BDRVQcow2State *s = bs->opaque;
+    Qcow2CompressFunc fn;
+
+    switch (s->compression_type) {
+    case QCOW2_COMPRESSION_TYPE_ZLIB:
+        fn = qcow2_zlib_decompress;
+        break;
+
+    default:
+        abort();
+    }
+
+    return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
 }
 
 
Signature · 1 line
2.17.0
#5 lore raw

PATCH v6 3/4 qcow2: add zstd cluster compression

In reply to Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G

The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.

compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img

compression decompression
zlib zstd zlib zstd
------------------------------------------------------------

real     65.5       16.3 (-75 %)    1.9          1.6 (-16 %)
user     65.0       15.8            5.3          2.5
sys       3.3        0.2            2.0          2.0

Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G

QAPI part:

---
 docs/interop/qcow2.txt |  20 +++++++
 configure              |   2 +-
 qapi/block-core.json   |   3 +-
 block/qcow2-threads.c  | 124 +++++++++++++++++++++++++++++++++++++++++
 block/qcow2.c          |  11 ++++
 5 files changed, 158 insertions(+), 2 deletions(-)
docs/interop/qcow2.txt +20 -0
index 5597e24474..9048114445 100644
@@ -208,6 +208,7 @@ version 2.
 
                     Available compression type values:
                         0: zlib <https://www.zlib.net/>
+                        1: zstd <http://github.com/facebook/zstd>
 
 
 === Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
                     Another compressed cluster may map to the tail of the final
                     sector used by this compressed cluster.
 
+                    The layout of the compressed data depends on the compression
+                    type used for the image (see compressed cluster layout).
+
 If a cluster is unallocated, read requests shall read the data from the backing
 file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
 no backing file or the backing file is smaller than the image, they shall read
 zeros for all parts that are not covered by the backing file.
 
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1
+
+    0:  (default)  zlib <http://zlib.net/>:
+            Byte  0 -  x:     the compressed data content
+                              all the space provided used for compressed data
+    1:  zstd <http://github.com/facebook/zstd>:
+            Byte  0 -  3:     the length of compressed data in bytes
+                  4 -  x:     the compressed data content
 
 == Snapshots ==
 
configure +1 -1
index caa65f5883..b2a0aa241a 100755
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if available:
   lzfse           support of lzfse compression library
                   (for reading lzfse-compressed dmg images)
   zstd            support for zstd compression library
-                  (for migration compression)
+                  (for migration compression and qcow2 cluster compression)
   seccomp         seccomp support
   coroutine-pool  coroutine freelist (better performance)
   glusterfs       GlusterFS backend
qapi/block-core.json +2 -1
index a306484973..8953451818 100644
@@ -4401,11 +4401,12 @@
 # Compression type used in qcow2 image file
 #
 # @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
 #
 # Since: 5.0
 ##
 { 'enum': 'Qcow2CompressionType',
-  'data': [ 'zlib' ] }
+  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
 
 ##
 # @BlockdevCreateOptionsQcow2:
block/qcow2-threads.c +124 -0

148 lines not on this page · raw message

block/qcow2.c +11 -0
index 21231adb63..53f65502f1 100644
@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
 {
     switch (s->compression_type) {
     case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+    case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
         break;
 
     default:
@@ -1279,6 +1282,10 @@ static int qcow2_compression_type_from_format(const char *ct)
 {
     if (g_str_equal(ct, "zlib")) {
         return QCOW2_COMPRESSION_TYPE_ZLIB;
+#ifdef CONFIG_ZSTD
+    } else if (g_str_equal(ct, "zstd")) {
+        return QCOW2_COMPRESSION_TYPE_ZSTD;
+#endif
     } else {
         return -EINVAL;
     }
@@ -3463,6 +3470,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
         }
 
         switch (qcow2_opts->compression_type) {
+#ifdef CONFIG_ZSTD
+        case QCOW2_COMPRESSION_TYPE_ZSTD:
+            break;
+#endif
         default:
             error_setg(errp, "Unknown compression type");
             goto out;
Signature · 1 line
2.17.0
#6 lore raw

Re: [PATCH v6 3/4] qcow2: add zstd cluster compression

Quoted 25 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

+++ b/docs/interop/qcow2.txt

docs/interop/qcow2.txt +4 -0
@@ -208,6 +208,7 @@ version 2.
 
                      Available compression type values:
                          0: zlib <https://www.zlib.net/>
+                        1: zstd <http://github.com/facebook/zstd>
 
 
  === Header padding ===
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
                      Another compressed cluster may map to the tail of the final
                      sector used by this compressed cluster.
 
+                    The layout of the compressed data depends on the compression
+                    type used for the image (see compressed cluster layout).
+
  If a cluster is unallocated, read requests shall read the data from the backing
  file (except if bit 0 in the Standard Cluster Descriptor is set). If there is

docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 593

Quoted 10 lines
docs/interop/qcow2.txt +7 -0
  no backing file or the backing file is smaller than the image, they shall read
  zeros for all parts that are not covered by the backing file.
 
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a compressed cluster.

length

docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 595

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1

If I understand correctly, data_space_length is really an upper bounds on the length available, because it is computed by rounding UP to the next 512-byte boundary (that is, the L2 descriptor lists the number of additional sectors used in storing the compressed data). Which really means that we have the following, where + is cluster boundaries, S and E are the start and end of the compressed data, and D is the offset determined by data_space_length:

+-------+-------+------+

      S============E...D

docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 599

Quoted 4 lines
docs/interop/qcow2.txt +4 -0
+
+    0:  (default)  zlib <http://zlib.net/>:
+            Byte  0 -  x:     the compressed data content
+                              all the space provided used for compressed data

For zlib, we have byte 0-E are compressed data, and bytes (E+1)-D (if any) are ignored. There is no way to tell how many bytes between E and D exist, because zlib doesn't care (the compression stream itself ensures that decompression stops when input reaches E because the output reached a cluster boundary at that point).

docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 602

Quoted 3 lines
docs/interop/qcow2.txt +3 -0
+    1:  zstd <http://github.com/facebook/zstd>:
+            Byte  0 -  3:     the length of compressed data in bytes
+                  4 -  x:     the compressed data content

Whereas for zstd, the decompression MUST know the actual location of E, rather than passing in the slop between E and D; bytes 0-3 give us that information.

But your description is not very accurate: if 'x' is point E, then it is NOT data_space_length - 1, but rather data_space_length - slop, where slop can be up to 511 bytes (the number of bytes from (E+1) to D). And if 'x' is point E, then the real layout for zlib is:

byte 0 - E: the compressed data content
byte E+1 - x: ignored slop (E is implied solely by the compressed data)

and for zstd is:

byte 0 - 3: the length of the compressed data byte 4 - E: the compressed data (E computed from byte 0-3) byte E+1 - x: ignored

I'm not sure what the best way is to document this.

Quoted 14 lines
block/qcow2-threads.c +13 -0
+++ b/block/qcow2-threads.c
 
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    size_t ret;
+
+    /*
+     * steal ZSTD_LEN_BUF bytes in the very beginning of the buffer
+     * to store compressed chunk size
+     */
+    char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+    /*

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 212

Quoted 10 lines
block/qcow2-threads.c +10 -0
+     * sanity check that we can store the compressed data length,
+     * and there is some space left for the compressor buffer
+     */
+    if (dest_size <= ZSTD_LEN_BUF) {
+        return -ENOMEM;
+    }
+
+    dest_size -= ZSTD_LEN_BUF;
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);

Where does the magic number 5 come from?

Quoted 6 lines
block/qcow2-threads.c +6 -0
+
+    if (ZSTD_isError(ret)) {
+        if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+            return -ENOMEM;
+        } else {
+            return -EIO;

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 228

Quoted 10 lines
block/qcow2-threads.c +10 -0
+        }
+    }
+
+    /*
+     * paranoid sanity check that we can store
+     * the compressed size in the first 4 bytes
+     */
+    if (ret > UINT32_MAX) {
+        return -ENOMEM;
+    }

The if is awkward. I'd prefer to change this to:

    /*
     * Our largest cluster is 2M, and we insist that compression
     * actually compressed things.
     */
    assert(ret < UINT32_MAX);

or even tighten to assert(ret <= dest_size)

Quoted 37 lines
block/qcow2-threads.c +37 -0
+
+    /* store the compressed chunk size in the very beginning of the buffer */
+    stl_be_p(dest, ret);
+
+    return ret + ZSTD_LEN_BUF;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          -EIO on any error
+ */
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{
+    /*
+     * zstd decompress wants to know the exact length of the data.
+     * For that purpose, on compression, the length is stored in
+     * the very beginning of the compressed buffer
+     */
+    size_t s_size;
+    const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
+
+    /*
+     * sanity check that we can read 4 byte the content length and
+     * and there is some content to decompress
+     */
+    if (src_size <= ZSTD_LEN_BUF) {
+        return -EIO;
+    }

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 275

Quoted 10 lines
block/qcow2-threads.c +10 -0
+
+    s_size = ldl_be_p(src);
+
+    /* sanity check that the buffer is big enough to read the content from */
+    if (src_size - ZSTD_LEN_BUF < s_size) {
+        return -EIO;
+    }
+
+    if (ZSTD_isError(
+            ZSTD_decompress(dest, dest_size, s_buf, s_size))) {

You are correct that ZSTD_decompress() is picky that it must be given the exact size of the compressed buffer it is decompressing. But the ZSTD manual mentions that if an exact size is not known in advance, that the streaming API can be used instead:

https://facebook.github.io/zstd/zstd_manual.html#Chapter9

In other words, would it be possible to NOT have to prepend four bytes of exact size information, by instead setting up decompression via the streaming API where the input is (usually) oversized, but the output buffer limited to exactly one cluster is sufficient to consume the exact compressed data and ignore the slop, just as we do in zlib?

The rest of this patch looks okay.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#7 lore raw

Re: [PATCH v6 3/4] qcow2: add zstd cluster compression

Quoted 90 lines · Eric Blake
Quoted 32 lines · Denis Plotnikov

zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.

+++ b/docs/interop/qcow2.txt

(quoted diff) +0 -0
@@ -208,6 +208,7 @@ version 2.

                        Available compression type values:
                          0: zlib <https://www.zlib.net/>
+                        1: zstd <http://github.com/facebook/zstd>
      === Header padding ===

(quoted diff) +0 -0
@@ -575,11 +576,30 @@ Compressed Clusters Descriptor (x = 62 - 

(cluster_bits - 8)):
                      Another compressed cluster may map to the tail of the final sector used by this compressed cluster.
  +                    The layout of the compressed data depends on the compression +                    type used for the image (see compressed cluster layout).
+
  If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file.
  +=== Compressed Cluster Layout ===

docs/interop/qcow2.txt +6 -0
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+data_space_lenght - data chunk length available to store a 
Quoted 1 line

compressed cluster.

length

docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 595

Quoted 2 lines
docs/interop/qcow2.txt +2 -0
+(for more details see "Compressed Clusters Descriptor")
+x = data_space_length - 1

If I understand correctly, data_space_length is really an upper bounds on the length available, because it is computed by rounding UP to the next 512-byte boundary (that is, the L2 descriptor lists the number of additional sectors used in storing the compressed data).  Which really means that we have the following, where + is cluster boundaries, S and E are the start and end of the compressed data, and D is the offset determined by data_space_length:

+-------+-------+------+
      S============E...D

Quoted 5 lines

+
+    0:  (default)  zlib <http://zlib.net/>:
+            Byte  0 -  x:     the compressed data content +                              all the space provided used for compressed data

For zlib, we have byte 0-E are compressed data, and bytes (E+1)-D (if any) are ignored.  There is no way to tell how many bytes between E and D exist, because zlib doesn't care (the compression stream itself ensures that decompression stops when input reaches E because the output reached a cluster boundary at that point).

Quoted 3 lines

+    1:  zstd <http://github.com/facebook/zstd>:
+            Byte  0 -  3:     the length of compressed data in bytes +                  4 -  x:     the compressed data content

Whereas for zstd, the decompression MUST know the actual location of E, rather than passing in the slop between E and D; bytes 0-3 give us that information.

But your description is not very accurate:  if 'x' is point E, then it is NOT data_space_length - 1, but rather data_space_length - slop, where slop can be up to 511 bytes (the number of bytes from (E+1) to D).  And if 'x' is point E, then the real layout for zlib is:

byte 0 - E: the compressed data content
byte E+1 - x: ignored slop (E is implied solely by the compressed data)

and for zstd is:

byte 0 - 3: the length of the compressed data byte 4 - E: the compressed data (E computed from byte 0-3) byte E+1 - x: ignored

I'm not sure what the best way is to document this.

Quoted 18 lines

+++ b/block/qcow2-threads.c

block/qcow2-threads.c +16 -0
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+                                   const void *src, size_t src_size)
+{
+    size_t ret;
+
+    /*
+     * steal ZSTD_LEN_BUF bytes in the very beginning of the buffer
+     * to store compressed chunk size
+     */
+    char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+    /*
+     * sanity check that we can store the compressed data length,
+     * and there is some space left for the compressor buffer
+     */
+    if (dest_size <= ZSTD_LEN_BUF) {
Quoted 8 lines

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 212

Quoted 6 lines

+        return -ENOMEM;
+    }

block/qcow2-threads.c +4 -0
+
+    dest_size -= ZSTD_LEN_BUF;
+
+    ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);

Where does the magic number 5 come from?

I did some tests to get the same compression ratio as zlib but do it faster than zlib.
ZLIB also used hardcoded "compression ratio". Changing of the compression ratios in both compression types is something that can be changed with later patches.

Quoted 82 lines
Quoted 8 lines
block/qcow2-threads.c +6 -0
+
+    if (ZSTD_isError(ret)) {
+        if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+            return -ENOMEM;
+        } else {
+            return -EIO;

block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 228

Quoted 10 lines
block/qcow2-threads.c +10 -0
+        }
+    }
+
+    /*
+     * paranoid sanity check that we can store
+     * the compressed size in the first 4 bytes
+     */
+    if (ret > UINT32_MAX) {
+        return -ENOMEM;
+    }

The if is awkward.  I'd prefer to change this to:

    /*
     * Our largest cluster is 2M, and we insist that compression
     * actually compressed things.
     */
    assert(ret < UINT32_MAX);

or even tighten to assert(ret <= dest_size)

Quoted 43 lines

+
+    /* store the compressed chunk size in the very beginning of the buffer */
+    stl_be_p(dest, ret);

block/qcow2-threads.c +8 -0
+
+    return ret + ZSTD_LEN_BUF;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce 

exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *          -EIO on any error
+ */

block/qcow2-threads.c +22 -0
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+                                     const void *src, size_t src_size)
+{
+    /*
+     * zstd decompress wants to know the exact length of the data.
+     * For that purpose, on compression, the length is stored in
+     * the very beginning of the compressed buffer
+     */
+    size_t s_size;
+    const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
+
+    /*
+     * sanity check that we can read 4 byte the content length and
+     * and there is some content to decompress
+     */
+    if (src_size <= ZSTD_LEN_BUF) {
+        return -EIO;
+    }
+
+    s_size = ldl_be_p(src);
+
+    /* sanity check that the buffer is big enough to read the 
Quoted 7 lines

content from */
+    if (src_size - ZSTD_LEN_BUF < s_size) {
+        return -EIO;
+    }
+
+    if (ZSTD_isError(
+            ZSTD_decompress(dest, dest_size, s_buf, s_size))) {

You are correct that ZSTD_decompress() is picky that it must be given the exact size of the compressed buffer it is decompressing.  But the ZSTD manual mentions that if an exact size is not known in advance, that the streaming API can be used instead:

Quoted 1 line

https://facebook.github.io/zstd/zstd_manual.html#Chapter9

To be honest, I didn't find where they mentioned that explicitly. Could you please point where exactly?

But I found the following:

Calling ZSTD_compressStream2() with ZSTD_e_end instructs to finish a frame.
 It will perform a flush and write frame epilogue.
The epilogue is required for decoders to consider a frame completed.
flush operation is the same, and follows same rules as calling ZSTD_compressStream2() with ZSTD_e_flush. You must continue calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, at which point you are free to start a new frame

I think in the epilogue they store the same information that I did and potentially (I didn't check) some more to finish the frame. So we didn't win any space. Additionally, using streaming API will make the code more complex.

So I decided to stick with more simple version.

Quoted 5 lines

In other words, would it be possible to NOT have to prepend four bytes of exact size information, by instead setting up decompression via the streaming API where the input is (usually) oversized, but the output buffer limited to exactly one cluster is sufficient to consume the exact compressed data and ignore the slop, just as we do in zlib?

The rest of this patch looks okay.

#8 lore raw

PATCH v6 4/4 iotests: 287: add qcow2 compression type test

In reply to Denis Plotnikov

The test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.

---
 tests/qemu-iotests/287     | 128 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/287.out |  43 +++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 172 insertions(+)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +128 -0
new file mode 100755
index 0000000000..49d15b3d43
@@ -0,0 +1,128 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null
+
+ZSTD_SUPPORTED=$?
+
+if (($ZSTD_SUPPORTED==0)); then
+    _notrun "ZSTD is disabled"
+fi
+
+# Test: when compression is zlib the incompatible bit is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+#       incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set  ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+    echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+#       incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset  ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+    echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values  ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/287.out new file +43 -0

43 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 0317667695..5edbadef40 100644
@@ -293,3 +293,4 @@
 283 auto quick
 284 rw
 286 rw quick
+287 auto quick
Signature · 1 line
2.17.0