qcow2: Implement zstd cluster compression method
v8 cover qcow2: Implement zstd cluster compression method
v8:
* 03: switch zstd API from simple to stream [Eric] No need to state a special cluster layout for zstd compressed clusters.
v7:
* use qapi_enum_parse instead of the open-coding [Eric]
* fix wording, typos and spelling [Eric]
v6:
* "block/qcow2-threads: fix qcow2_decompress" is removed from the series since it has been accepted by Max already
* add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto]
* report error on qcow2 amending when the compression type is actually chnged [Max]
* remove the extra space and the extra new line [Max]
* re-arrange acks and signed-off-s [Vladimir]
v5:
* replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir]
* set cluster size for all test cases in the beginning of the 287 test
v4:
* the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress"
* 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate
v3:
* remove redundant max compression type value check [Vladimir, Eric] (the switch below checks everything)
* prevent compression type changing on "qemu-img amend" [Vladimir]
* remove zstd config setting, since it has been added already by "migration" patches [Vladimir]
* change the compression type error message [Vladimir]
* fix alignment and 80-chars exceeding [Vladimir]
v2:
* rework compression type setting [Vladimir]
* squash iotest changes to the compression type introduction patch [Vladimir, Eric]
* fix zstd availability checking in zstd iotest [Vladimir]
* remove unnecessry casting [Eric]
* remove rudundant checks [Eric]
* fix compressed cluster layout in qcow2 spec [Vladimir]
* fix wording [Eric, Vladimir]
* fix compression type filtering in iotests [Eric]
v1:
the initial series
Denis Plotnikov (4):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: 287: add qcow2 compression type test
docs/interop/qcow2.txt | 1 + configure | 2 +- qapi/block-core.json | 23 +++- block/qcow2.h | 20 +++- include/block/block_int.h | 1 + block/qcow2-threads.c | 200 +++++++++++++++++++++++++++++-- block/qcow2.c | 120 +++++++++++++++++++ tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 ++++++++-------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 28 +++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 128 ++++++++++++++++++++ tests/qemu-iotests/287.out | 43 +++++++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 638 insertions(+), 108 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
Signature · 1 line
2.17.0
PATCH v8 1/4 qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255--- qapi/block-core.json | 22 +++++- block/qcow2.h | 20 +++++- include/block/block_int.h | 1 + block/qcow2.c | 113 +++++++++++++++++++++++++++++++ tests/qemu-iotests/031.out | 14 ++-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 ++++++++++++++-------------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 ++++++---- tests/qemu-iotests/065 | 28 +++++--- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- tests/qemu-iotests/common.filter | 3 +- 16 files changed, 267 insertions(+), 96 deletions(-)
qapi/block-core.json +20 -2
@@ -78,6 +78,8 @@ # # @bitmaps: A list of qcow2 bitmap details (since 4.0) # +# @compression-type: the image cluster compression method (since 5.0) +# # Since: 1.7 ## { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@ '*corrupt': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', - '*bitmaps': ['Qcow2BitmapInfo'] + '*bitmaps': ['Qcow2BitmapInfo'], + 'compression-type': 'Qcow2CompressionType' } } ##
@@ -4392,6 +4395,18 @@ 'data': [ 'v2', 'v3' ] } +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib: zlib compression, see <http://zlib.net/> +# +# Since: 5.0 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib' ] } + ## # @BlockdevCreateOptionsQcow2: #
@@ -4415,6 +4430,8 @@ # allowed values: off, falloc, full, metadata) # @lazy-refcounts: True if refcounts may be updated lazily (default: off) # @refcount-bits: Width of reference counts in bits (default: 16) +# @compression-type: The image cluster compression method +# (default: zlib, since 5.0) # # Since: 2.12 ##
@@ -4430,7 +4447,8 @@ '*cluster-size': 'size', '*preallocation': 'PreallocMode', '*lazy-refcounts': 'bool', - '*refcount-bits': 'int' } } + '*refcount-bits': 'int', + '*compression-type':'Qcow2CompressionType' } } ## # @BlockdevCreateOptionsQed:
block/qcow2.h +19 -1
@@ -146,8 +146,16 @@ typedef struct QCowHeader {uint32_t refcount_order;uint32_t header_length; + +/* Additional fields */ +uint8_t compression_type; + +/* header must be a multiple of 8 */ +uint8_t padding[7 ]; }QEMU_PACKED QCowHeader; +QEMU_BUILD_BUG_ON (sizeof (QCowHeader) %8 !=0 ); + typedef struct QEMU_PACKED QCowSnapshotHeader {/* header is 8 byte aligned */ uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {QCOW2_INCOMPAT_DIRTY_BITNR =0 ,QCOW2_INCOMPAT_CORRUPT_BITNR =1 ,QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , +QCOW2_INCOMPAT_COMPRESSION_BITNR =3 ,QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR ,QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR ,QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_COMPRESSION =1 <<QCOW2_INCOMPAT_COMPRESSION_BITNR ,QCOW2_INCOMPAT_MASK =QCOW2_INCOMPAT_DIRTY |QCOW2_INCOMPAT_CORRUPT - | QCOW2_INCOMPAT_DATA_FILE, + |QCOW2_INCOMPAT_DATA_FILE + | QCOW2_INCOMPAT_COMPRESSION, };/* Compatible feature bits */
@@ -369,6 +380,13 @@ typedef struct BDRVQcow2State {bool metadata_preallocation_checked;bool metadata_preallocation; +/* +* Compression type used for the image. Default: 0 - ZLIB +* The image compression type is set on image creation. +* For now, the only way to change the compression type +* is to convert the image with the desired compression type set. +*/ +Qcow2CompressionType compression_type; } BDRVQcow2State; typedef struct Qcow2COWRegion {
include/block/block_int.h +1 -0
@@ -57,6 +57,7 @@#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"#define BLOCK_OPT_DATA_FILE "data_file"#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"#define BLOCK_PROBE_BUF_SIZE 512
block/qcow2.c +113 -0
191 lines not on this page · raw message
tests/qemu-iotests/031.out +7 -7
@@ -113,11 +113,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
@@ -146,11 +146,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x178 +backing_file_offset 0x1b0 backing_file_size 0x17 cluster_bits 16 size 67108864
@@ -179,7 +179,7 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0xe2792aca
@@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
tests/qemu-iotests/036.out +2 -2
@@ -26,7 +26,7 @@ compatible_features [] autoclear_features [63] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary>
@@ -38,7 +38,7 @@ compatible_features [] autoclear_features [] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> *** done
tests/qemu-iotests/049.out +51 -51
225 lines not on this page · raw message
tests/qemu-iotests/060.out +1 -0
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: true
tests/qemu-iotests/061.out +20 -14
@@ -22,11 +22,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -80,11 +80,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -136,11 +136,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features [] compatible_features [42] autoclear_features [42] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -260,11 +260,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 131072/131072 bytes at offset 0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
tests/qemu-iotests/065 +18 -10
@@ -88,24 +88,30 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10', 'refcount-bits': 16 } - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ] + json_compare = { 'compat': '0.10', 'refcount-bits': 16, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 0.10', 'compression type: zlib', + 'refcount bits: 16' ] class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' json_compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', - 'refcount bits: 16', 'corrupt: false' ] + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', + 'lazy refcounts: false', 'refcount bits: 16', + 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' json_compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', - 'refcount bits: 16', 'corrupt: false' ] + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', + 'lazy refcounts: true', 'refcount bits: 16', + 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP): '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
@@ -113,7 +119,8 @@ class TestQCow3NotLazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +129,8 @@ class TestQCow3LazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } TestImageInfoSpecific = None TestQemuImgInfo = None
tests/qemu-iotests/080 +1 -1
@@ -45,7 +45,7 @@ _supported_os Linux # - This is generally a test for compat=1.1 images _unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10' -header_size=104 +header_size=112 offset_backing_file_offset=8 offset_backing_file_size=16
tests/qemu-iotests/144.out +2 -2
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912 { 'execute': 'qmp_capabilities' } {"return": {}} { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} === Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/ === Performing Live Snapshot 2 === { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} *** done
tests/qemu-iotests/182.out +1 -1
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]? {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}} {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } } -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}
tests/qemu-iotests/242.out +5 -0
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
tests/qemu-iotests/255.out +4 -4
@@ -3,9 +3,9 @@ Finishing a commit job with background reads === Create backing chain and start VM === -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib === Start background read requests ===
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled === Create images and start VM === -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib wrote 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/common.filter +2 -1
@@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=[a-zA-Z0-9]\\+##g" } _filter_img_info()
Signature · 1 line
2.17.0
Re: [PATCH v8 1/4] qcow2: introduce compression type feature
Quoted 33 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests * fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080 header_size +=8: 1 byte compression type 7 bytes padding feature_table += 48: incompatible feature compression type backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change) * add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>---
[...]
Quoted 9 lines
block/qcow2.c +1
-0@@ -4859,6 +4949,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; }else {/* if this assertion fails, this probably means a new version was
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 5348
Quoted 10 lines
block/qcow2.c +7
-0@@ -5248,6 +5339,22 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, "images");return -EINVAL ; } + }else if (!strcmp (desc->name,BLOCK_OPT_COMPRESSION_TYPE )) { +int compression_type = +qapi_enum_parse (&Qcow2CompressionType_lookup, +qemu_opt_get (opts,BLOCK_OPT_COMPRESSION_TYPE ), +-1 , errp); + +if (compression_type == -EINVAL ) {
You should compare to -1, as qapi_enum_parse returns given default on error.
Review static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 5349
Quoted 1 line
(quoted diff) +1
-0+ error_setg(errp, "Unknown compression type");
and errp is already set (ofcourse, if qemu_opt_get returned non NULL, but I hope it is guaranteed by if (!strcmp(desc->name, BLOCK_OPT_COMPRESSION_TYPE)) condition
Quoted 24 lines
block/qcow2.c +14
-0+return -ENOTSUP ; + } + +if (compression_type != s->compression_type) { +error_setg (errp,"Changing the compression type " +"is not supported" ); +return -ENOTSUP ; + } }else {/* if this point is reached, this probably means a new option was * added without having it covered here */ @@ -5516,6 +5623,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits", .def_value_str = "16" }, + { + .name = BLOCK_OPT_COMPRESSION_TYPE, + .type = QEMU_OPT_STRING, + .help = "Compression method used for image cluster compression", + .def_value_str = "zlib" + }, {/* end of list */ } } };
[...]
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 1/4] qcow2: introduce compression type feature
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 33 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter outcompression_typefor many tests
* fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080
header_size+=8: 1 byte compression type
7 bytes padding
feature_table+= 48: incompatible feature compression typebacking_file_offset+= 56 (8 + 48 ->header_change+feature_table_change)
* add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>---
[...]
Quoted 16 lines
block/qcow2.c +1
-0@@ -4859,6 +4949,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; } else {/* if this assertion fails, this probably means a new version was
block/qcow2.c +2
-0@@ -5248,6 +5339,22 @@ static intqcow2_amend_options (BlockDriverState *bs, QemuOpts *opts, "images"); return -EINVAL; } + } else if (!strcmp(desc->name, BLOCK_OPT_COMPRESSION_TYPE)) { + int compression_type =
Quoted 8 lines
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 5348
Quoted 6 lines
block/qcow2.c +5
-0+qapi_enum_parse (&Qcow2CompressionType_lookup, + qemu_opt_get(opts,BLOCK_OPT_COMPRESSION_TYPE ), + -1, errp); + + if (compression_type == -EINVAL) {You should compare to -1, as
qapi_enum_parsereturns given default on error.
ok
Quoted 4 lines
Review static int · line 5351
Quoted 2 lines
(quoted diff) +1
-0+ error_setg(errp, "Unknown compression type");and errp is already set (ofcourse, if
qemu_opt_getreturned non NULL, but I hope it is guaranteed by if (!strcmp(desc->name,BLOCK_OPT_COMPRESSION_TYPE)) condition
I wouldn't propagate the error from qapi_enum_parse because it looks like "invalid parameter value: foo". I think it's better to print "Unknown compression type: foo"
Quoted 31 lines
Quoted 26 lines
block/qcow2.c +8
-0+ return -ENOTSUP; + } + + if (compression_type != s->compression_type) { + error_setg(errp, "Changing the compression type " + "is not supported"); + return -ENOTSUP; + } } else {/* if this point is reached, this probably means a new option was
* added without having it covered here */block/qcow2.c +0
-0@@ -5516,6 +5623,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits",.def_value_str = "16"
},
+ {
+ .name =BLOCK_OPT_COMPRESSION_TYPE,
+ .type =QEMU_OPT_STRING,
+ .help = "Compression method used for image cluster compression",
+ .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};[...]
Re: [PATCH v8 1/4] qcow2: introduce compression type feature
Quoted 61 lines · Denis Plotnikov
Quoted 50 lines · Vladimir Sementsov-Ogievskiy
Quoted 33 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter outcompression_typefor many tests
* fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080
header_size+=8: 1 byte compression type
7 bytes padding
feature_table+= 48: incompatible feature compression typebacking_file_offset+= 56 (8 + 48 ->header_change+feature_table_change)
* add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>---
[...]
Quoted 13 lines
block/qcow2.c +2
-0@@ -4859,6 +4949,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; } else { /* if this assertion fails, this probably means a new version was@@ -5248,6 +5339,22 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, "images"); return -EINVAL; } + } else if (!strcmp(desc->name, BLOCK_OPT_COMPRESSION_TYPE)) {Quoted 8 lines
block/qcow2.c static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 5348
Quoted 6 lines
block/qcow2.c +6
-0+ int compression_type = +qapi_enum_parse (&Qcow2CompressionType_lookup, + qemu_opt_get(opts, BLOCK_OPT_COMPRESSION_TYPE), + -1, errp); + + if (compression_type == -EINVAL) {You should compare to -1, as
qapi_enum_parsereturns given default on error.
Quoted 6 lines
ok
Quoted 4 lines
Review static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts, · line 5349
Quoted 2 lines
(quoted diff) +1
-0+ error_setg(errp, "Unknown compression type");and errp is already set (ofcourse, if
qemu_opt_getreturned non NULL, but I hope it is guaranteed by if (!strcmp(desc->name,BLOCK_OPT_COMPRESSION_TYPE)) conditionI wouldn't propagate the error from
qapi_enum_parsebecause it looks like "invalid parameter value: foo". I think it's better to print "Unknown compression type: foo"
No objections. Then you should pass NULL to qapi_enum_parse instead of errp. (As you can't set errp twice, second try will crash).
Quoted 31 lines
Quoted 30 lines
Quoted 25 lines
block/qcow2.c +8
-0+ return -ENOTSUP; + } + + if (compression_type != s->compression_type) { + error_setg(errp, "Changing the compression type " + "is not supported"); + return -ENOTSUP; + } } else { /* if this point is reached, this probably means a new option was* added without having it covered here */ @@ -5516,6 +5623,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits",.def_value_str = "16"
},
+ {
+ .name =BLOCK_OPT_COMPRESSION_TYPE,
+ .type =QEMU_OPT_STRING,
+ .help = "Compression method used for image cluster compression", + .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};[...]
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 1/4] qcow2: introduce compression type feature
Quoted 65 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines · Denis Plotnikov
Quoted 61 lines · Vladimir Sementsov-Ogievskiy
Quoted 33 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to qcow2 allowing the use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the compression type is reported on image creation and there are some changes in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter outcompression_typefor many tests
* fix header size, feature table size and backing file offset affected tests: 031, 036, 061, 080
header_size+=8: 1 byte compression type
7 bytes padding
feature_table+= 48: incompatible feature compression typebacking_file_offset+= 56 (8 + 48 ->header_change+feature_table_change)
* add "compression type" for test output matching when it isn't filtered affected tests: 049, 060, 061, 065, 144, 182, 242, 255Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>---
[...]
Quoted 12 lines
block/qcow2.c +1
-0@@ -4859,6 +4949,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; } else {/* if this assertion fails, this probably means a new version was
block/qcow2.c +0
-0@@ -5248,6 +5339,22 @@ static intqcow2_amend_options (BlockDriverState *bs, QemuOpts *opts, "images");block/qcow2.c static int · line 5350
Quoted 10 lines
block/qcow2.c +7
-0return -EINVAL; } + } else if (!strcmp(desc->name, BLOCK_OPT_COMPRESSION_TYPE)) { + int compression_type = +qapi_enum_parse (&Qcow2CompressionType_lookup, + qemu_opt_get(opts,BLOCK_OPT_COMPRESSION_TYPE ), + -1, errp); + + if (compression_type == -EINVAL) {You should compare to -1, as
qapi_enum_parsereturns given default on error.
Quoted 8 lines
Quoted 6 lines
ok
Quoted 4 lines
Review static int · line 5351
Quoted 2 lines
(quoted diff) +1
-0+ error_setg(errp, "Unknown compression type");and errp is already set (ofcourse, if
qemu_opt_getreturned non NULL, but I hope it is guaranteed by if (!strcmp(desc->name,BLOCK_OPT_COMPRESSION_TYPE)) conditionI wouldn't propagate the error from
qapi_enum_parsebecause it looks like "invalid parameter value: foo". I think it's better to print "Unknown compression type: foo"No objections. Then you should pass NULL to
qapi_enum_parseinstead of errp. (As you can't set errp twice, second try will crash).
ok
Quoted 35 lines
Quoted 33 lines
Quoted 32 lines
Quoted 27 lines
block/qcow2.c +8
-0+ return -ENOTSUP; + } + + if (compression_type != s->compression_type) { + error_setg(errp, "Changing the compression type " + "is not supported"); + return -ENOTSUP; + } } else {/* if this point is reached, this probably means a new option was
* added without having it covered here */block/qcow2.c +0
-0@@ -5516,6 +5623,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits",.def_value_str = "16"
},
+ {
+ .name =BLOCK_OPT_COMPRESSION_TYPE,
+ .type =QEMU_OPT_STRING,
+ .help = "Compression method used for image cluster compression",
+ .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};[...]
PATCH v8 2/4 qcow2: rework the cluster compression routine
In reply to Denis PlotnikovThe patch enables processing the image compression type defined for the image and chooses an appropriate method for image clusters (de)compression.
--- block/qcow2-threads.c | 71 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 60 insertions(+), 11 deletions(-)
block/qcow2-threads.c +60 -11
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData { } Qcow2CompressData;/* -* qcow2_compress() +* qcow2_zlib_compress() +* +* Compress @src_size bytes of data using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {* -ENOMEM destination buffer is not enough to store compressed data * -EIO on any other error */ -static ssize_t qcow2_compress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) {ssize_t ret;z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size, }/* -* qcow2_decompress() +* qcow2_zlib_decompress() * * Decompress some data (not more than @src_size bytes) to produce exactly -* @dest_size bytes. +* @dest_size bytes using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,* Returns: 0 on success * -EIO on fail */ -static ssize_t qcow2_decompress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) {int ret;z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,return arg.ret; } +/* +* qcow2_co_compress() +* +* Compress @src_size bytes of data using the compression +* method defined by the image compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success +* a negative error code on failure +*/ ssize_t coroutine_fnqcow2_co_compress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_compress); +BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_compress; +break ; + +default : +abort (); + } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); } +/* +* qcow2_co_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using the compression method defined by the image +* compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* a negative error code on failure +*/ ssize_t coroutine_fnqcow2_co_decompress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_decompress); +BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_decompress; +break ; + +default : +abort (); + } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); }
Signature · 1 line
2.17.0
PATCH v8 3/4 qcow2: add zstd cluster compression
In reply to Denis Plotnikovzstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
QAPI part:
--- docs/interop/qcow2.txt | 1 + configure | 2 +- qapi/block-core.json | 3 +- block/qcow2-threads.c | 129 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++ 5 files changed, 140 insertions(+), 2 deletions(-)
docs/interop/qcow2.txt +1 -0
@@ -208,6 +208,7 @@ version 2. Available compression type values: 0: zlib <https://www.zlib.net/> + 1: zstd <http://github.com/facebook/zstd> === Header padding ===
configure +1 -1
@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if available: lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) zstd support for zstd compression library - (for migration compression) + (for migration compression and qcow2 cluster compression) seccomp seccomp support coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backend
qapi/block-core.json +2 -1
@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:
block/qcow2-threads.c +129 -0
153 lines not on this page · raw message
block/qcow2.c +7 -0
@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) {switch (s->compression_type) {case QCOW2_COMPRESSION_TYPE_ZLIB : +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +#endif break ;default :
@@ -3454,6 +3457,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) }switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +break ; +#endif default :error_setg (errp,"Unknown compression type" ); goto out;
Signature · 1 line
2.17.0
Re: [PATCH v8 3/4] qcow2: add zstd cluster compression
Quoted 29 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.imgcompression decompression
zlib zstd zlib zstd
------------------------------------------------------------real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0
Quoted 8 lines
Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Alberto Garcia <[email protected]>QAPI part:
You forget to drop signs, patch is changed significantly, including algorithm.
Quoted 89 lines
--- docs/interop/qcow2.txt | 1 + configure | 2 +- qapi/block-core.json | 3 +- block/qcow2-threads.c | 129 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++ 5 files changed, 140 insertions(+), 2 deletions(-)docs/interop/qcow2.txt +1
-0@@ -208,6 +208,7 @@ version 2. Available compression type values: 0: zlib <https://www.zlib.net/> + 1: zstd <http://github.com/facebook/zstd> === Header padding ===configure +1
-1@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if available: lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) zstd support for zstd compression library - (for migration compression) + (for migration compression and qcow2 cluster compression) seccomp seccomp support coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendqapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:block/qcow2-threads.c +23
-0@@ -28,6 +28,11 @@#define ZLIB_CONST #include <zlib.h> +#ifdef CONFIG_ZSTD +#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "qcow2.h" #include "block/thread-pool.h" #include "crypto.h" @@ -166,6 +171,120 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,return ret; } +#ifdef CONFIG_ZSTD + +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success +* -ENOMEM destination buffer is not enough to store compressed data +* -EIO on any other error +*/ +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +size_t ret;
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 201
Quoted 10 lines
block/qcow2-threads.c +10
-0+ZSTD_outBuffer output = { dest, dest_size,0 }; +ZSTD_inBuffer input = { src, src_size,0 }; +ZSTD_CCtx *cctx =ZSTD_createCCtx (); + +if (!cctx) { +return -EIO ; + } + + ret =ZSTD_CCtx_setParameter (cctx, ZSTD_c_compressionLevel, +ZSTD_CLEVEL_DEFAULT );
Hmm, looks a bit strange.. Isn't it already default by default?)
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 206
Quoted 5 lines
block/qcow2-threads.c +5
-0+if (ZSTD_isError (ret)) { + ret = -EIO ; + goto out; + } +
Hmm, strange that we need a loop, but zstd spec directly requires it, possibly we need to make a comment from it:
"ZSTD spec: You must continue calling ZSTD_compressStream2() with ZSTD_e_flush until it returns 0, at which point you can change the operation."
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 212
Quoted 6 lines
block/qcow2-threads.c +6
-0+ { +/* +* Instruct zstd to compress the whole buffer and write +* the frame epilogue. This allows as to use zstd streaming +* semantics and don't store the compressed size for the +* zstd decompression.
May be not "to don't store", but to "decompress without knowing exact compressed data length" or something like this. I mean, make the comment for people who read the final code, not for those who review changes from version to version.
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 214
Quoted 2 lines
block/qcow2-threads.c +2
-0+*/ + ret =ZSTD_compressStream2 (cctx, &output , &input, ZSTD_e_end);
extra whitespace
Quoted 48 lines
block/qcow2-threads.c +48
-0+if (ZSTD_isError (ret)) { + ret = -EIO ; + goto out; + } +/* Dest buffer isn't big enough to store compressed content */ +if (output.pos + ret > output.size) { + ret = -ENOMEM ; + goto out; + } + }while (ret); + +/* if no error, the input data must be fully consumed */ +assert (input.pos == input.size); +/* make sure we can safely return compressed buffer size with ssize_t */ +assert (output.pos <=SSIZE_MAX ); + ret = output.pos; + +out: +ZSTD_freeCCtx (cctx); +return ret; +} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* -EIO on any error +*/ +static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +size_t ret =0 ; +ZSTD_outBuffer output = { dest, dest_size,0 }; +ZSTD_inBuffer input = { src, src_size,0 }; +ZSTD_DCtx *dctx =ZSTD_createDCtx (); + +if (!dctx) { +return -EIO ; + } + + { + ret =ZSTD_decompressStream (dctx, &output, &input);
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 272
Quoted 10 lines
block/qcow2-threads.c +10
-0+if (ZSTD_isError (ret)) { + ret = -EIO ; + goto out; + } +/* +* Dest buffer size is the image cluster size. +* It should be big enough to store uncompressed content. +* There shouldn't be any cases when the decompressed content +* size is greater then the cluster size. +*/
But consider corrupted image: it may contain any data. And we should not crash because of it. So, we should return error here.
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 278
Quoted 6 lines
block/qcow2-threads.c +6
-0+assert (output.pos + ret <= output.size); + }while (ret); + +/* +* If decompression went fine we must have the compressed +* cluser fully consumed and flushed
cluster
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 280
Quoted 2 lines
block/qcow2-threads.c +2
-0+*/ +assert (output.pos == output.size);
again, it should be EIO, not crash.
Quoted 60 lines
block/qcow2-threads.c +17
-0+out: +ZSTD_freeDCtx (dctx); + return ret; + +} +#endif +static int qcow2_compress_pool_func (void *opaque) {Qcow2CompressData *data = opaque;@@ -217,6 +336,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_compress; +break ; +#endif default :abort (); }@@ -249,6 +373,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_decompress; +break ; +#endif default :abort (); }block/qcow2.c +7
-0@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) {switch (s->compression_type) {case QCOW2_COMPRESSION_TYPE_ZLIB : +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +#endif break ;default :@@ -3454,6 +3457,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) }switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +break ; +#endif default :error_setg (errp,"Unknown compression type" ); goto out;
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 3/4] qcow2: add zstd cluster compression
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
Quoted 32 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.imgcompression decompression zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Quoted 7 lines
Quoted 5 lines
Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Alberto Garcia <[email protected]>QAPI part:
You forget to drop signs, patch is changed significantly, including algorithm.
ok, my fault
Quoted 93 lines
Quoted 93 lines
---
docs/interop/qcow2.txt| 1 +
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 129 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++5 files changed, 140 insertions(+), 2 deletions(-)docs/interop/qcow2.txt +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===configure +0
-0@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default isenabled if available:
lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) zstd support for zstd compression library - (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendqapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:block/qcow2-threads.c +0
-0@@ -28,6 +28,11 @@ #define ZLIB_CONST #include <zlib.h>+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +4
-0+#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "
qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"block/qcow2-threads.c +0
-0@@ -166,6 +171,120 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, return ret; }+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +11
-0+ +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success +* -ENOMEM destination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */block/qcow2-threads.c +8
-0+static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + ZSTD_outBuffer output = { dest, dest_size, 0 }; + ZSTD_inBuffer input = { src, src_size, 0 }; + ZSTD_CCtx *cctx = ZSTD_createCCtx(); +
Quoted 8 lines
Quoted 6 lines
+ if (!cctx) {
+ return-EIO;
+ }
+
+ ret = ZSTD_CCtx_setParameter(cctx,ZSTD_c_compressionLevel, +ZSTD_CLEVEL_DEFAULT);Hmm, looks a bit strange.. Isn't it already default by default?)
Should I remove it? Doesn't worth to express that explicitly? This line removes the question what compression level is used. And gives a hint where to change it ,if we decide to implement compression ratio changing.
Quoted 10 lines
Quoted 6 lines
+ if (
ZSTD_isError(ret)) {
+ ret =-EIO;
+ goto out;
+ }
+Hmm, strange that we need a loop, but zstd spec directly requires it, possibly we need to make a comment from it:
"ZSTD spec: You must continue calling
ZSTD_compressStream2()withZSTD_e_flushuntil it returns 0, at which point you can change the operation."
ok
Quoted 9 lines
Quoted 7 lines
+ {
+ /*
+ * Instruct zstd to compress the whole buffer and write + * the frame epilogue. This allows as to use zstd streaming + * semantics and don't store the compressed size for the + * zstd decompression.May be not "to don't store", but to "decompress without knowing exact compressed data length" or something like this. I mean, make the comment for people who read the final code, not for those who review changes from version to version.
I'd better write
* zstd simple interface requires exact compressed size.
* zstd stream interface holds all the data in the compressed frame.
* Instruct zstd to compress the whole buffer and write
* the frame epilogue. This allows as to use zstd streaming
* semantics and don't store the compressed size for the
* zstd decompression.
Quoted 60 lines
Quoted 3 lines
+ */
+ ret =ZSTD_compressStream2(cctx, &output , &input, ZSTD_e_end);extra whitespace
Quoted 54 lines
+ if (
ZSTD_isError(ret)) {
+ ret =-EIO;
+ goto out;
+ }
+ /* Dest buffer isn't big enough to store compressed content */ + if (output.pos + ret > output.size) {
+ ret =-ENOMEM;
+ goto out;
+ }
+ } while (ret);
+
+ /* if no error, the input data must be fully consumed */ + assert(input.pos == input.size);
+ /* make sure we can safely return compressed buffer size withssize_t*/
+assert(output.pos <= SSIZE_MAX);
+ ret = output.pos;block/qcow2-threads.c +10
-0+ +out: + ZSTD_freeCCtx(cctx); + return ret; +} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *-EIOon any error
+ */block/qcow2-threads.c +18
-0+static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret = 0; + ZSTD_outBuffer output = { dest, dest_size, 0 }; + ZSTD_inBuffer input = { src, src_size, 0 }; + ZSTD_DCtx *dctx = ZSTD_createDCtx(); + + if (!dctx) { + return -EIO; + } + + { + ret = ZSTD_decompressStream(dctx, &output, &input); + if (ZSTD_isError(ret)) { + ret = -EIO; + goto out; + }
Quoted 8 lines
Quoted 6 lines
+ /*
+ * Dest buffer size is the image cluster size.
+ * It should be big enough to store uncompressed content. + * There shouldn't be any cases when the decompressed content + * size is greater then the cluster size.
+ */But consider corrupted image: it may contain any data. And we should not crash because of it. So, we should return error here.
If the image is corrupted we can't continue anyway. If we return -EIO on this condition, we need to do some work investigating what has happened. Assert explicitly points out what happened. I though we should return -EIO if we know what to do with the error. But if the invariant doesn't hold, do we really know what to do rather then stop the execution and report an error?
Quoted 77 lines
Quoted 7 lines
+ assert(output.pos + ret <= output.size);
+ } while (ret);
+
+ /*
+ * If decompression went fine we must have the compressed + * cluser fully consumed and flushedcluster
Quoted 2 lines
+ */
+ assert(output.pos == output.size);again, it should be EIO, not crash.
Quoted 60 lines
block/qcow2-threads.c +7
-0+out: + ZSTD_freeDCtx(dctx); + return ret; + +} +#endif +static int
qcow2_compress_pool_func(void *opaque){
Qcow2CompressData *data = opaque;block/qcow2-threads.c +0
-0@@ -217,6 +336,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_compress;
+ break;
+#endif
default:
abort();
}block/qcow2-threads.c +0
-0@@ -249,6 +373,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_decompress;
+ break;
+#endif
default:
abort();
}block/qcow2.c +0
-0@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) { switch (s->compression_type) {case
QCOW2_COMPRESSION_TYPE_ZLIB:block/qcow2.c +3
-0+#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break;
default:block/qcow2.c +4
-0@@ -3454,6 +3457,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; +#endif default:
error_setg(errp, "Unknown compression type"); goto out;
Re: [PATCH v8 3/4] qcow2: add zstd cluster compression
Quoted 140 lines · Denis Plotnikov
Quoted 33 lines · Vladimir Sementsov-Ogievskiy
Quoted 32 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.imgcompression decompression zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Quoted 7 lines
Quoted 5 lines
Signed-off-by Denis Plotnikov <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Reviewed-by Alberto Garcia <[email protected]>QAPI part:
You forget to drop signs, patch is changed significantly, including algorithm.
ok, my fault
Quoted 88 lines
Quoted 88 lines
---
docs/interop/qcow2.txt| 1 +
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 129 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++5 files changed, 140 insertions(+), 2 deletions(-)docs/interop/qcow2.txt +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===configure +0
-0@@ -1835,7 +1835,7 @@ disabled with --disable-FEATURE, default is enabled if available:lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images) zstd support for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression) seccomp seccomp support
coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendqapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:block/qcow2-threads.c +0
-0@@ -28,6 +28,11 @@ #define ZLIB_CONST #include <zlib.h>+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +4
-0+#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "
qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"block/qcow2-threads.c +0
-0@@ -166,6 +171,120 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, return ret; }+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +21
-0+ +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success +* -ENOMEM destination buffer is not enough to store compressed data +* -EIO on any other error +*/ +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + ZSTD_outBuffer output = { dest, dest_size, 0 }; + ZSTD_inBuffer input = { src, src_size, 0 }; + ZSTD_CCtx *cctx = ZSTD_createCCtx(); +Quoted 8 lines
Quoted 6 lines
+ if (!cctx) {
+ return-EIO;
+ }
+
+ ret = ZSTD_CCtx_setParameter(cctx,ZSTD_c_compressionLevel, +ZSTD_CLEVEL_DEFAULT);Hmm, looks a bit strange.. Isn't it already default by default?)
Quoted 1 line
Should I remove it? Doesn't worth to express that explicitly? This line removes the question what compression level is used.
No it doesn't, as we don't know the constant, neither we have guarantee that it is the same in different version of library..
I'd drop it if it changes nothing.
Quoted 95 lines
And gives a hint where to change it ,if we decide to implement compression ratio changing.
Quoted 10 lines
Quoted 6 lines
+ if (
ZSTD_isError(ret)) {
+ ret =-EIO;
+ goto out;
+ }
+Hmm, strange that we need a loop, but zstd spec directly requires it, possibly we need to make a comment from it:
"ZSTD spec: You must continue calling
ZSTD_compressStream2()withZSTD_e_flushuntil it returns 0, at which point you can change the operation."ok
Quoted 9 lines
Quoted 7 lines
+ {
+ /*
+ * Instruct zstd to compress the whole buffer and write + * the frame epilogue. This allows as to use zstd streaming + * semantics and don't store the compressed size for the + * zstd decompression.May be not "to don't store", but to "decompress without knowing exact compressed data length" or something like this. I mean, make the comment for people who read the final code, not for those who review changes from version to version.
I'd better write
* zstd simple interface requires exact compressed size.
* zstd stream interface holds all the data in the compressed frame.
* Instruct zstd to compress the whole buffer and write
* the frame epilogue. This allows as to use zstd streaming
* semantics and don't store the compressed size for the
* zstd decompression.Quoted 58 lines
Quoted 3 lines
+ */
+ ret =ZSTD_compressStream2(cctx, &output , &input, ZSTD_e_end);extra whitespace
Quoted 52 lines
+ if (
ZSTD_isError(ret)) {
+ ret =-EIO;
+ goto out;
+ }
+ /* Dest buffer isn't big enough to store compressed content */ + if (output.pos + ret > output.size) {
+ ret =-ENOMEM;
+ goto out;
+ }
+ } while (ret);block/qcow2-threads.c +42
-0+ + /* if no error, the input data must be fully consumed */ + assert(input.pos == input.size); + /* make sure we can safely return compressed buffer size with ssize_t */ + assert(output.pos <= SSIZE_MAX); + ret = output.pos; + +out: + ZSTD_freeCCtx(cctx); + return ret; +} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* -EIO on any error +*/ +static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret = 0; + ZSTD_outBuffer output = { dest, dest_size, 0 }; + ZSTD_inBuffer input = { src, src_size, 0 }; + ZSTD_DCtx *dctx = ZSTD_createDCtx(); + + if (!dctx) { + return -EIO; + } + + { + ret = ZSTD_decompressStream(dctx, &output, &input); + if (ZSTD_isError(ret)) { + ret = -EIO; + goto out; + }Quoted 8 lines
Quoted 6 lines
+ /*
+ * Dest buffer size is the image cluster size.
+ * It should be big enough to store uncompressed content. + * There shouldn't be any cases when the decompressed content + * size is greater then the cluster size.
+ */But consider corrupted image: it may contain any data. And we should not crash because of it. So, we should return error here.
Quoted 1 line
If the image is corrupted we can't continue anyway. If we return
-EIOon this condition, we need to do some work investigating what has happened. Assert explicitly points out what happened. I though we should return-EIOif we know what to do with the error. But if the invariant doesn't hold, do we really know what to do rather then stop the execution and report an error?
If only one compressed cluster is corrupted because of one changed byte, it's not a reason to crash the whole program. We still can read another clusters. You already return EIO on ZSTD failure, which includes some kind of corrupted image of course.
What to do with error - nothing. This cluster is corrupted, unreadable. Why not? Real devices may have corrupted clusters too.
Quoted 74 lines
Quoted 73 lines
Quoted 7 lines
+ assert(output.pos + ret <= output.size);
+ } while (ret);
+
+ /*
+ * If decompression went fine we must have the compressed + * cluser fully consumed and flushedcluster
Quoted 2 lines
+ */
+ assert(output.pos == output.size);again, it should be EIO, not crash.
Quoted 56 lines
block/qcow2-threads.c +7
-0+out: + ZSTD_freeDCtx(dctx); + return ret; + +} +#endif +static int
qcow2_compress_pool_func(void *opaque){
Qcow2CompressData *data = opaque;block/qcow2-threads.c +0
-0@@ -217,6 +336,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_compress;
+ break;
+#endif
default:
abort();
}block/qcow2-threads.c +0
-0@@ -249,6 +373,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_decompress;
+ break;
+#endif
default:
abort();
}block/qcow2.c +0
-0@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) { switch (s->compression_type) {case
QCOW2_COMPRESSION_TYPE_ZLIB:block/qcow2.c +3
-0+#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break;
default:block/qcow2.c +4
-0@@ -3454,6 +3457,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; +#endif default:
error_setg(errp, "Unknown compression type"); goto out;
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v8 3/4] qcow2: add zstd cluster compression
In reply to Denis PlotnikovQuoted 5 lines · Denis Plotnikov
Quoted 2 lines
But consider corrupted image: it may contain any data. And we should not crash because of it. So, we should return error here.
If the image is corrupted we can't continue anyway. If we return
-EIOon this condition, we need to do some work investigating what has happened.
Cannot you just mark the image as corrupted and return -EIO ? I also don't think that we should crash QEMU because of malformed input data.
Berto
Re: [PATCH v8 3/4] qcow2: add zstd cluster compression
Quoted 10 lines · Alberto Garcia
Quoted 5 lines · Denis Plotnikov
Quoted 2 lines
But consider corrupted image: it may contain any data. And we should not crash because of it. So, we should return error here.
If the image is corrupted we can't continue anyway. If we return
-EIOon this condition, we need to do some work investigating what has happened.Cannot you just mark the image as corrupted and return
-EIO? I also don't think that we should crash QEMU because of malformed input data.Berto
ok, will return -EIO
PATCH v8 4/4 iotests: 287: add qcow2 compression type test
In reply to Denis PlotnikovThe test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.
--- tests/qemu-iotests/287 | 128 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/287.out | 43 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 172 insertions(+) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +128 -0
@@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# +# Test case for an image using zstd compression +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# standard environment +. ./common.rc +. ./common.filter + +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +# for all the cases +CLUSTER_SIZE=65536 + +# Check if we can run this test. + +IMGOPTS='compression_type=zstd' _make_test_img 64M | grep "Invalid parameter 'zstd'" 2>&1 1>/dev/null + +ZSTD_SUPPORTED=$? + +if (($ZSTD_SUPPORTED==0)); then + _notrun "ZSTD is disabled" +fi + +# Test: when compression is zlib the incompatible bit is unset +echo +echo "=== Testing compression type incompatible bit setting for zlib ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: when compression differs from zlib the incompatible bit is set +echo +echo "=== Testing compression type incompatible bit setting for zstd ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: an image can't be openned if compression type is zlib and +# incompatible feature compression type is set +echo +echo "=== Testing zlib with incompatible bit set ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3 +# to make sure the bit was actually set +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi + +# Test: an image can't be openned if compression type is NOT zlib and +# incompatible feature compression type is UNSET +echo +echo "=== Testing zstd with incompatible bit unset ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0 +# to make sure the bit was actually unset +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi +# Test: check compression type values +echo +echo "=== Testing compression type values ===" +echo +# zlib=0 +IMGOPTS='compression_type=zlib' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# zstd=1 +IMGOPTS='compression_type=zstd' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$QEMU_IO -c "write -c -P 0xAC 65536 64k " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -P 0xAC 65536 65536 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io + +# success, all done +echo "*** done" +rm -f $seq.full +status=0
tests/qemu-iotests/287.out new file +43 -0
43 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -293,3 +293,4 @@ 283 auto quick 284 rw 286 rw quick +287 auto quick
Signature · 1 line
2.17.0