qcow2: Implement zstd cluster compression method
PATCH 7/8 iotests: 080: update header size value because of adding compression type
In reply to Denis Plotnikov--- tests/qemu-iotests/080 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/080 +1 -1
@@ -45,7 +45,7 @@ _supported_os Linux # - This is generally a test for compat=1.1 images _unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10' -header_size=104 +header_size=112 offset_backing_file_offset=8 offset_backing_file_size=16
Signature · 1 line
2.17.0
PATCH 8/8 iotests: 287: add qcow2 compression type test
In reply to Denis PlotnikovThe test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.
--- tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/287.out | 41 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 165 insertions(+) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +123 -0
@@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Test case for an image using zstd compression +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# standard environment +. ./common.rc +. ./common.filter + +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"` + +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null +RES=$? +if (($RES)); then + _notrun "ZSTD is disabled in the current configuration" +fi + +# Test: when compression is zlib the incompatible is unset +echo +echo "=== Testing compression type incompatible bit setting for zlib ===" +echo + +_make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: when compression differs from zlib the incompatible bit is set +echo +echo "=== Testing compression type incompatible bit setting for zstd ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: an image can't be openned if compression type is zlib and +# incompatible feature compression type is set +echo +echo "=== Testing zlib with incompatible bit set ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3 +# to make sure the bit was actually set +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi + +# Test: an image can't be openned if compression type is NOT zlib and +# incompatible feature compression type is UNSET +echo +echo "=== Testing zstd with incompatible bit unset ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0 +# to make sure the bit was actually unset +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi +# Test: check compression type values +echo +echo "=== Testing compression type values ===" +echo +# zlib=0 +IMGOPTS='compression_type=zlib' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# zstd=1 +IMGOPTS='compression_type=zstd' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo + +CLUSTER_SIZE=65536 +IMGOPTS='compression_type=zstd' _make_test_img 64M +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io + +# success, all done +echo "*** done" +rm -f $seq.full +status=0
tests/qemu-iotests/287.out new file +41 -0
41 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -293,3 +293,4 @@ 283 auto quick 284 rw 286 rw quick +287 auto quick
Signature · 1 line
2.17.0
PATCH 1/8 qcow2: introduce compression type feature
In reply to Denis PlotnikovThe patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
--- block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++--- include/block/block_int.h | 1 + qapi/block-core.json | 22 +++++++- 4 files changed, 150 insertions(+), 9 deletions(-)
block/qcow2.c +105 -0
@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,return ret; } +static int validate_compression_type (BDRVQcow2State *s,Error **errp) +{ +/* +* Sanity check +* according to qcow2 spec, the compression type is 1-byte field +* but in BDRVQcow2State the compression_type is enum sizeof(int) +* so, the max compression_type value is 255. +*/ +if (s->compression_type >0xff ) { +error_setg (errp,"qcow2: compression type value is too big" ); +return -EINVAL ; + } + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : +break ; + +default : +error_setg (errp,"qcow2: unknown compression type: %u" , + s->compression_type); +return -ENOTSUP ; + } + +/* +* if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB +* the incompatible feature flag must be set +*/ +if (s->compression_type ==QCOW2_COMPRESSION_TYPE_ZLIB ) { +if (s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE ) { +error_setg (errp,"qcow2: Compression type incompatible feature " +"bit must not be set" ); +return -EINVAL ; + } + }else { +if (!(s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE )) { +error_setg (errp,"qcow2: Compression type incompatible feature " +"bit must be set" ); +return -EINVAL ; + } + } + +return 0 ; +} +/* Called with s->lock held. */ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,int flags,Error **errp)
@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, s->compatible_features = header.compatible_features; s->autoclear_features = header.autoclear_features; +/* +* Handle compression type +* Older qcow2 images don't contain the compression type header. +* Distinguish them by the header length and use +* the only valid (default) compression type in that case +*/ +if (header.header_length > offsetof(QCowHeader , compression_type)) { +/* +* don't deal with endians since compression_type is 1 byte long +*/ + s->compression_type = header.compression_type; + }else { + s->compression_type =QCOW2_COMPRESSION_TYPE_ZLIB ; + } + + ret =validate_compression_type (s, errp); +if (ret) { + goto fail; + } +if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK ) {void *feature_table =NULL ; qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs) total_size = bs->total_sectors *BDRV_SECTOR_SIZE ; refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits -3 ); + ret =validate_compression_type (s,NULL ); + +if (ret) { + goto fail; + } + *header = (QCowHeader ) {/* Version 2 fields */ .magic =cpu_to_be32 (QCOW_MAGIC ),
@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features = cpu_to_be64(s->autoclear_features), .refcount_order = cpu_to_be32(s->refcount_order), .header_length = cpu_to_be32(header_length), + .compression_type = (uint8_t) s->compression_type, };/* For older versions, write a shorter header */
@@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name = "compression type", + }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), .refcount_order = cpu_to_be32(refcount_order), + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB, .header_length = cpu_to_be32(sizeof(*header)), };
@@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)cpu_to_be64 (QCOW2_AUTOCLEAR_DATA_FILE_RAW ); } +if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type !=QCOW2_COMPRESSION_TYPE_ZLIB ) { + +if (qcow2_opts->compression_type >0xff ) { +error_setg_errno (errp, -EINVAL ,"Too big compression type value" ); + goto out; + } + +switch (qcow2_opts->compression_type) { +default : +error_setg_errno (errp, -EINVAL ,"Unknown compression type" ); + goto out; + } + + header->compression_type = (uint8_t ) qcow2_opts->compression_type; + + header->incompatible_features |= +cpu_to_be64 (QCOW2_INCOMPAT_COMPRESSION_TYPE ); + } + ret =blk_pwrite (blk,0 , header, cluster_size,0 );g_free (header);if (ret <0 ) {
@@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, { BLOCK_OPT_COMPAT_LEVEL, "version" }, { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" }, + { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" }, { NULL, NULL }, };
@@ -4859,6 +4957,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; }else {/* if this assertion fails, this probably means a new version was
@@ -5516,6 +5615,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits", .def_value_str = "16" }, + { + .name = BLOCK_OPT_COMPRESSION_TYPE, + .type = QEMU_OPT_STRING, + .help = "Compression method used for image clusters compression", + .def_value_str = "zlib" + }, {/* end of list */ } } };
block/qcow2.h +24 -7
@@ -146,6 +146,12 @@ typedef struct QCowHeader {uint32_t refcount_order;uint32_t header_length; + +/* Additional fields */ +uint8_t compression_type; + +/* header must be a multiple of 8 */ +uint8_t padding[7 ]; }QEMU_PACKED QCowHeader; typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -213,16 +219,20 @@ enum {/* Incompatible feature bits */ enum { -QCOW2_INCOMPAT_DIRTY_BITNR =0 , -QCOW2_INCOMPAT_CORRUPT_BITNR =1 , -QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , -QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , -QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , -QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_DIRTY_BITNR =0 , +QCOW2_INCOMPAT_CORRUPT_BITNR =1 , +QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , +QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3 , +QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , +QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , +QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_COMPRESSION_TYPE = +1 <<QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR ,QCOW2_INCOMPAT_MASK =QCOW2_INCOMPAT_DIRTY |QCOW2_INCOMPAT_CORRUPT - |QCOW2_INCOMPAT_DATA_FILE , + |QCOW2_INCOMPAT_DATA_FILE + |QCOW2_INCOMPAT_COMPRESSION_TYPE , };/* Compatible feature bits */
@@ -369,6 +379,13 @@ typedef struct BDRVQcow2State {bool metadata_preallocation_checked;bool metadata_preallocation; +/* +* Compression type used for the image. Default: 0 - ZLIB +* The image compression type is set on image creation. +* The only way to change the compression type is to convert the image +* with the desired compression type set +*/ +Qcow2CompressionType compression_type; } BDRVQcow2State; typedef struct Qcow2COWRegion {
include/block/block_int.h +1 -0
@@ -57,6 +57,7 @@#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"#define BLOCK_OPT_DATA_FILE "data_file"#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"#define BLOCK_PROBE_BUF_SIZE 512
qapi/block-core.json +20 -2
@@ -78,6 +78,8 @@ # # @bitmaps: A list of qcow2 bitmap details (since 4.0) # +# @compression-type: the image cluster compression method (since 5.0) +# # Since: 1.7 ## { 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@ '*corrupt': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', - '*bitmaps': ['Qcow2BitmapInfo'] + '*bitmaps': ['Qcow2BitmapInfo'], + 'compression-type': 'Qcow2CompressionType' } } ##
@@ -4392,6 +4395,18 @@ 'data': [ 'v2', 'v3' ] } +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib: zlib compression, see <http://zlib.net/> +# +# Since: 5.0 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib' ] } + ## # @BlockdevCreateOptionsQcow2: #
@@ -4415,6 +4430,8 @@ # allowed values: off, falloc, full, metadata) # @lazy-refcounts: True if refcounts may be updated lazily (default: off) # @refcount-bits: Width of reference counts in bits (default: 16) +# @compression-type: The image cluster compression method +# (default: zlib, since 5.0) # # Since: 2.12 ##
@@ -4430,7 +4447,8 @@ '*cluster-size': 'size', '*preallocation': 'PreallocMode', '*lazy-refcounts': 'bool', - '*refcount-bits': 'int' } } + '*refcount-bits': 'int', + '*compression-type': 'Qcow2CompressionType' } } ## # @BlockdevCreateOptionsQed:
Signature · 1 line
2.17.0
PATCH 3/8 qcow2: add zstd cluster compression
In reply to Denis Plotnikovzstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
--- block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++ configure | 29 ++++++++++ docs/interop/qcow2.txt | 18 ++++++ qapi/block-core.json | 3 +- 5 files changed, 178 insertions(+), 1 deletion(-)
block/qcow2-threads.c +122 -0
@@ -28,6 +28,11 @@#define ZLIB_CONST #include <zlib.h> +#ifdef CONFIG_ZSTD +#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "qcow2.h" #include "block/thread-pool.h" #include "crypto.h"
@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,return ret; } +#ifdef CONFIG_ZSTD + +#define ZSTD_LEN_BUF 4 + +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success +* -ENOMEM destination buffer is not enough to store compressed data +* -EIO on any other error +*/ + +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +size_t ret; + +/* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer +* to store compressed chunk size +*/ +char *d_buf = ((char *) dest) +ZSTD_LEN_BUF ; + +/* +* sanity check that we can store the compressed data length, +* and there is some space left for the compressor buffer +*/ +if (dest_size <=ZSTD_LEN_BUF ) { +return -ENOMEM ; + } + + dest_size -=ZSTD_LEN_BUF ; + + ret =ZSTD_compress (d_buf, dest_size, src, src_size,5 ); + +if (ZSTD_isError (ret)) { +if (ZSTD_getErrorCode (ret) == ZSTD_error_dstSize_tooSmall) { +return -ENOMEM ; + }else { +return -EIO ; + } + } + +/* paraniod sanity check that we can store the commpressed size */ +if (ret >UINT_MAX ) { +return -ENOMEM ; + } + +/* store the compressed chunk size in the very beginning of the buffer */ +stl_be_p (dest, ret); + +return ret +ZSTD_LEN_BUF ; +} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* -EIO on any error +*/ + +static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +/* +* zstd decompress wants to know the exact length of the data. +* For that purpose, on compression, the length is stored in +* the very beginning of the compressed buffer +*/ +size_t s_size; +const char *s_buf = ((const char *) src) +ZSTD_LEN_BUF ; + +/* +* sanity check that we can read 4 byte the content length and +* and there is some content to decompress +*/ +if (src_size <=ZSTD_LEN_BUF ) { +return -EIO ; + } + + s_size =ldl_be_p (src); + +/* sanity check that the buffer is big enough to read the content from */ +if (src_size -ZSTD_LEN_BUF < s_size) { +return -EIO ; + } + +if (ZSTD_isError ( +ZSTD_decompress (dest, dest_size, s_buf, s_size))) { +return -EIO ; + } + +return 0 ; +} +#endif +static int qcow2_compress_pool_func (void *opaque) {Qcow2CompressData *data = opaque;
@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_compress; +break ; +#endif default :return -ENOTSUP ; }
@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_decompress; +break ; +#endif default :return -ENOTSUP ; }
block/qcow2.c +7 -0
@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)switch (s->compression_type) {case QCOW2_COMPRESSION_TYPE_ZLIB : +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +#endif break ;default :
@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) }switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +break ; +#endif default :error_setg_errno (errp, -EINVAL ,"Unknown compression type" ); goto out;
configure +29 -0
@@ -444,6 +444,7 @@ opengl_dmabuf="no" cpuid_h="no" avx2_opt="" zlib="yes" +zstd="" capstone="" lzo="" snappy=""
@@ -1371,6 +1372,10 @@ for opt do ;; --disable-lzfse) lzfse="no" ;; + --enable-zstd) zstd="yes" + ;; + --disable-zstd) zstd="no" + ;; --enable-guest-agent) guest_agent="yes" ;; --disable-guest-agent) guest_agent="no"
@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available: (for reading bzip2-compressed dmg images) lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backend
@@ -2453,6 +2459,25 @@ EOF fi fi +######################################### +# zstd check + +if test "$zstd" != "no" ; then + cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOF + if compile_prog "" "-lzstd" ; then + LIBS="$LIBS -lzstd" + zstd="yes" + else + if test "$zstd" = "yes"; then + feature_not_found "zstd" "Install libzstd-devel" + fi + zstd="no" + fi +fi + ########################################## # libseccomp check
@@ -6668,6 +6693,7 @@ echo "lzo support $lzo" echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "lzfse support $lzfse" +echo "zstd support $zstd" echo "NUMA host support $numa" echo "libxml2 $libxml2" echo "tcmalloc support $tcmalloc"
@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then "\$ld_exported_symbols_list should have been set to 'yes'." fi fi +if test "$zstd" = "yes" ; then + echo "CONFIG_ZSTD=y" >> $config_host_mak +fi if test "$tcg_interpreter" = "yes"; then QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
docs/interop/qcow2.txt +18 -0
@@ -208,6 +208,7 @@ version 2. Available compression type values: 0: zlib <https://www.zlib.net/> + 1: zstd <http://github.com/facebook/zstd> === Header padding ===
@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): Another compressed cluster may map to the tail of the final sector used by this compressed cluster. + The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout). + If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file. +=== Compressed Cluster Layout === + +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1) + + 0: (default) zlib <http://zlib.net/>: + Byte 0 - x: the compressed data content + all the space provided used for compressed data + 1: zstd <http://github.com/facebook/zstd>: + Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data content == Snapshots ==
qapi/block-core.json +2 -1
@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:
Signature · 1 line
2.17.0
cover qcow2: Implement zstd cluster compression method
zstd comression method is faster than the only available zlib. The series adds zstd to the methods available for clusters compression.
The implementation is done with respect to the recently added compression type additional header to the qcow2 specification.
Denis Plotnikov (8):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: filter out compression_type
iotests: fix header size, feature table size and backing file offset
iotests: add "compression type" for test output matching
iotests: 080: update header size value because of adding compression
type
iotests: 287: add qcow2 compression type test
block/qcow2-threads.c | 199 ++++++++++++++++++++++++++++--- block/qcow2.c | 112 +++++++++++++++++ block/qcow2.h | 31 +++-- configure | 29 +++++ docs/interop/qcow2.txt | 18 +++ include/block/block_int.h | 1 + qapi/block-core.json | 23 +++- tests/qemu-iotests/031.out | 14 +-- tests/qemu-iotests/036.out | 4 +- tests/qemu-iotests/049.out | 102 ++++++++-------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 34 +++--- tests/qemu-iotests/065 | 20 ++-- tests/qemu-iotests/080 | 2 +- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 + tests/qemu-iotests/255.out | 8 +- tests/qemu-iotests/287 | 123 +++++++++++++++++++ tests/qemu-iotests/287.out | 41 +++++++ tests/qemu-iotests/common.filter | 3 +- tests/qemu-iotests/group | 1 + 22 files changed, 663 insertions(+), 114 deletions(-) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
Signature · 1 line
2.17.0
PATCH 4/8 iotests: filter out compression_type
After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started reporting the compression type for the image which breaks the iotests output matching.
To fix it, add compression_type=zlib to the list of filtered image parameters.
--- tests/qemu-iotests/common.filter | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tests/qemu-iotests/common.filter +2 -1
@@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=zlib##g" } _filter_img_info()
Signature · 1 line
2.17.0
PATCH 2/8 qcow2: rework the cluster compression routine
In reply to Denis PlotnikovThe patch enables processing the image compression type defined for the image and chooses an appropriate method for image clusters (de)compression.
--- block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 14 deletions(-)
block/qcow2-threads.c +63 -14
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData { } Qcow2CompressData;/* -* qcow2_compress() +* qcow2_zlib_compress() +* +* Compress @src_size bytes of data using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {* -ENOMEM destination buffer is not enough to store compressed data * -EIO on any other error */ -static ssize_t qcow2_compress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) {ssize_t ret;z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size, }/* -* qcow2_decompress() +* qcow2_zlib_decompress() * * Decompress some data (not more than @src_size bytes) to produce exactly -* @dest_size bytes. +* @dest_size bytes using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes * * Returns: 0 on success -* -1 on fail +* -EIO on failure */ -static ssize_t qcow2_decompress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) {int ret =0 ;z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size, ret =inflateInit2 (&strm,-12 );if (ret !=Z_OK ) { -return -1 ; +return -EIO ; } ret =inflate (&strm,Z_FINISH );
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,* @src buffer may be processed partly (because in qcow2 we know size of * compressed data with precision of one sector) */ - ret =-1 ; + ret = -EIO ; }inflateEnd (&strm);
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,return arg.ret; } +/* +* qcow2_co_compress() +* +* Compress @src_size bytes of data using the compression +* method defined by the image compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* a negative error code on failure +*/ ssize_t coroutine_fnqcow2_co_compress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_compress); +BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_compress; +break ; + +default : +return -ENOTSUP ; + } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); } +/* +* qcow2_co_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using the compression method defined by the image +* compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* a negative error code on failure +*/ ssize_t coroutine_fnqcow2_co_decompress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_decompress); +BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_decompress; +break ; + +default : +return -ENOTSUP ; + } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); }
Signature · 1 line
2.17.0
PATCH 5/8 iotests: fix header size, feature table size and backing file offset
In reply to Denis PlotnikovAffected tests: 031, 036, 061
Because of adding the compression type feature, some size values in the qcow2 v3 header are changed:
header_size +=8: 1 byte compression type
7 bytes paddingfeature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
Change the values for the test output comparison accordingly.
--- tests/qemu-iotests/031.out | 14 +++++++------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 28 ++++++++++++++-------------- 3 files changed, 23 insertions(+), 23 deletions(-)
tests/qemu-iotests/031.out +7 -7
@@ -113,11 +113,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
@@ -146,11 +146,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x178 +backing_file_offset 0x1b0 backing_file_size 0x17 cluster_bits 16 size 67108864
@@ -179,7 +179,7 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0xe2792aca
@@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:
tests/qemu-iotests/036.out +2 -2
@@ -26,7 +26,7 @@ compatible_features [] autoclear_features [63] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary>
@@ -38,7 +38,7 @@ compatible_features [] autoclear_features [] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> *** done
tests/qemu-iotests/061.out +14 -14
@@ -22,11 +22,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -80,11 +80,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -136,11 +136,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features [] compatible_features [42] autoclear_features [42] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb
@@ -260,11 +260,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 131072/131072 bytes at offset 0
Signature · 1 line
2.17.0
PATCH 6/8 iotests: add "compression type" for test output matching
In reply to Denis PlotnikovAffected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output matching.
--- tests/qemu-iotests/049.out | 102 ++++++++++++++++++------------------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 6 +++ tests/qemu-iotests/065 | 20 +++++--- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-)
tests/qemu-iotests/049.out +51 -51
@@ -4,90 +4,90 @@ QA output created by 049 == 1. Traditional size parameter == qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 2. Specifying size via -o == qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 3. Invalid sizes ==
@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once == Check correct interpretation of suffixes for cluster size == qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check compat level option == qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check preallocation option == qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check encryption option == qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check lazy_refcounts option (only with v3) == qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib *** done
tests/qemu-iotests/060.out +1 -0
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: true
tests/qemu-iotests/061.out +6 -0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data
tests/qemu-iotests/065 +12 -8
@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10', 'refcount-bits': 16 } - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ] + json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' } + human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ] class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' json_compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' json_compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP):
@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } TestImageInfoSpecific = None TestQemuImgInfo = None
tests/qemu-iotests/144.out +2 -2
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912 { 'execute': 'qmp_capabilities' } {"return": {}} { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} === Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/ === Performing Live Snapshot 2 === { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} *** done
tests/qemu-iotests/182.out +1 -1
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]? {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}} {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } } -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}
tests/qemu-iotests/242.out +5 -0
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:
tests/qemu-iotests/255.out +4 -4
@@ -3,9 +3,9 @@ Finishing a commit job with background reads === Create backing chain and start VM === -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib === Start background read requests ===
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled === Create images and start VM === -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib wrote 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Signature · 1 line
2.17.0
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Denis Plotnikovblock/qcow2.c
Quoted 27 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Signed-off-by Denis Plotnikov <[email protected]>--- block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++--- include/block/block_int.h | 1 + qapi/block-core.json | 22 +++++++- 4 files changed, 150 insertions(+), 9 deletions(-)block/qcow2.c +0
-0
Please, add to .git/config:
[diff]
orderFile = /path/to/qemu/scripts/git.orderfile
This will force git format-patch to sort files in more comfortable order (header changes first, etc).
Quoted 16 lines
block/qcow2.c +12
-0@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,return ret; } +static int validate_compression_type (BDRVQcow2State *s,Error **errp) +{ +/* +* Sanity check +* according to qcow2 spec, the compression type is 1-byte field +* but in BDRVQcow2State the compression_type is enum sizeof(int) +* so, the max compression_type value is 255. +*/ +if (s->compression_type >0xff ) { +error_setg (errp,"qcow2: compression type value is too big" ); +return -EINVAL ; + }
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1266
Quoted 10 lines
block/qcow2.c +10
-0+ +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : +break ; + +default : +error_setg (errp,"qcow2: unknown compression type: %u" , + s->compression_type); +return -ENOTSUP ; + }
honestly, I think that just
if (s->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
error out
}
is enough, and don't see why to check > 0xff in separate..
But it works as is.
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1278
Quoted 12 lines
block/qcow2.c +12
-0+ +/* +* if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB +* the incompatible feature flag must be set +*/ +if (s->compression_type ==QCOW2_COMPRESSION_TYPE_ZLIB ) { +if (s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE ) { +error_setg (errp,"qcow2: Compression type incompatible feature " +"bit must not be set" ); +return -EINVAL ; + } + }else {
This is unreachable now.. But it's OK as preparation for further patches I think.
Quoted 17 lines
block/qcow2.c +10
-0+if (!(s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE )) { +error_setg (errp,"qcow2: Compression type incompatible feature " +"bit must be set" ); +return -EINVAL ; + } + } + +return 0 ; +} +/* Called with s->lock held. */ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,int flags,Error **errp)@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, s->compatible_features = header.compatible_features; s->autoclear_features = header.autoclear_features;
block/qcow2.c static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, · line 1413
Quoted 10 lines
block/qcow2.c +10
-0+/* +* Handle compression type +* Older qcow2 images don't contain the compression type header. +* Distinguish them by the header length and use +* the only valid (default) compression type in that case +*/ +if (header.header_length > offsetof(QCowHeader , compression_type)) { +/* +* don't deal with endians since compression_type is 1 byte long +*/
this comment would be more appropriate above, where be-to-cpu transformation is done,
as actually previous "s->autoclear_features = header.autoclear_features" doesn't
deal with endians too.
Quoted 47 lines
block/qcow2.c +22
-0+ s->compression_type = header.compression_type; + }else { + s->compression_type =QCOW2_COMPRESSION_TYPE_ZLIB ; + } + + ret =validate_compression_type (s, errp); +if (ret) { + goto fail; + } +if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK ) {void *feature_table =NULL ; qcow2_read_extensions(bs, header.header_length, ext_end,@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs) total_size = bs->total_sectors *BDRV_SECTOR_SIZE ; refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits -3 ); + ret =validate_compression_type (s,NULL ); + +if (ret) { + goto fail; + } + *header = (QCowHeader ) {/* Version 2 fields */ .magic =cpu_to_be32 (QCOW_MAGIC ),@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features = cpu_to_be64(s->autoclear_features), .refcount_order = cpu_to_be32(s->refcount_order), .header_length = cpu_to_be32(header_length), + .compression_type = (uint8_t) s->compression_type, };/* For older versions, write a shorter header */ @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE, + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name = "compression type", + }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3480
Quoted 4 lines
block/qcow2.c +1
-0@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), .refcount_order = cpu_to_be32(refcount_order), + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
Hmm. Following qcow2_co_create existing code, you'd better
1. do check compression_type correctnes before header initialization, may be add local variable compression_type, don't forget to check version < 3, and here you'll have only .compression_type = compression_type
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3499
Quoted 7 lines
block/qcow2.c +0
-0.header_length = cpu_to_be32(sizeof(*header)), };@@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)cpu_to_be64 (QCOW2_AUTOCLEAR_DATA_FILE_RAW ); }
2. Here you'll need only set incompatible_features if needed.
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3504
Quoted 5 lines
block/qcow2.c +5
-0+if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type !=QCOW2_COMPRESSION_TYPE_ZLIB ) { + +if (qcow2_opts->compression_type >0xff ) { +error_setg_errno (errp, -EINVAL ,"Too big compression type value" );
I still doubt that we need seperate check for this thing. Moreover, here we can get only element of enum, so we can do assertion, not if.
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3505
Quoted 1 line
block/qcow2.c +1
-0+ goto out;
going out here, you leak header
Quoted 179 lines
block/qcow2.c +22
-0+ } + +switch (qcow2_opts->compression_type) { +default : +error_setg_errno (errp, -EINVAL ,"Unknown compression type" ); + goto out; + } + + header->compression_type = (uint8_t ) qcow2_opts->compression_type; + + header->incompatible_features |= +cpu_to_be64 (QCOW2_INCOMPAT_COMPRESSION_TYPE ); + } + ret =blk_pwrite (blk,0 , header, cluster_size,0 );g_free (header);if (ret <0 ) {@@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT }, { BLOCK_OPT_COMPAT_LEVEL, "version" }, { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" }, + { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" }, { NULL, NULL }, };@@ -4859,6 +4957,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs, .data_file = g_strdup(s->image_data_file), .has_data_file_raw = has_data_file(bs), .data_file_raw = data_file_is_raw(bs), + .compression_type = s->compression_type, }; }else {/* if this assertion fails, this probably means a new version was @@ -5516,6 +5615,12 @@ static QemuOptsList qcow2_create_opts = { .help = "Width of a reference count entry in bits", .def_value_str = "16" }, + { + .name = BLOCK_OPT_COMPRESSION_TYPE, + .type = QEMU_OPT_STRING, + .help = "Compression method used for image clusters compression", + .def_value_str = "zlib" + }, {/* end of list */ } } };block/qcow2.h +24
-7@@ -146,6 +146,12 @@ typedef struct QCowHeader {uint32_t refcount_order;uint32_t header_length; + +/* Additional fields */ +uint8_t compression_type; + +/* header must be a multiple of 8 */ +uint8_t padding[7 ]; }QEMU_PACKED QCowHeader; typedef struct QEMU_PACKED QCowSnapshotHeader {@@ -213,16 +219,20 @@ enum {/* Incompatible feature bits */ enum { -QCOW2_INCOMPAT_DIRTY_BITNR =0 , -QCOW2_INCOMPAT_CORRUPT_BITNR =1 , -QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , -QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , -QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , -QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_DIRTY_BITNR =0 , +QCOW2_INCOMPAT_CORRUPT_BITNR =1 , +QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , +QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3 , +QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , +QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , +QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_COMPRESSION_TYPE = +1 <<QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR ,QCOW2_INCOMPAT_MASK =QCOW2_INCOMPAT_DIRTY |QCOW2_INCOMPAT_CORRUPT - |QCOW2_INCOMPAT_DATA_FILE , + |QCOW2_INCOMPAT_DATA_FILE + |QCOW2_INCOMPAT_COMPRESSION_TYPE , };/* Compatible feature bits */ @@ -369,6 +379,13 @@ typedef struct BDRVQcow2State {bool metadata_preallocation_checked;bool metadata_preallocation; +/* +* Compression type used for the image. Default: 0 - ZLIB +* The image compression type is set on image creation. +* The only way to change the compression type is to convert the image +* with the desired compression type set +*/ +Qcow2CompressionType compression_type; } BDRVQcow2State; typedef struct Qcow2COWRegion {include/block/block_int.h +1
-0@@ -57,6 +57,7 @@#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"#define BLOCK_OPT_DATA_FILE "data_file"#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw" +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"#define BLOCK_PROBE_BUF_SIZE 512qapi/block-core.json +20
-2@@ -78,6 +78,8 @@ # # @bitmaps: A list of qcow2 bitmap details (since 4.0) # +# @compression-type: the image cluster compression method (since 5.0) +# # Since: 1.7 ## { 'struct': 'ImageInfoSpecificQCow2',@@ -89,7 +91,8 @@ '*corrupt': 'bool', 'refcount-bits': 'int', '*encrypt': 'ImageInfoSpecificQCow2Encryption', - '*bitmaps': ['Qcow2BitmapInfo'] + '*bitmaps': ['Qcow2BitmapInfo'], + 'compression-type': 'Qcow2CompressionType' } } ##@@ -4392,6 +4395,18 @@ 'data': [ 'v2', 'v3' ] } +## +# @Qcow2CompressionType: +# +# Compression type used in qcow2 image file +# +# @zlib: zlib compression, see <http://zlib.net/> +# +# Since: 5.0 +## +{ 'enum': 'Qcow2CompressionType', + 'data': [ 'zlib' ] } + ## # @BlockdevCreateOptionsQcow2: #@@ -4415,6 +4430,8 @@ # allowed values: off, falloc, full, metadata) # @lazy-refcounts: True if refcounts may be updated lazily (default: off) # @refcount-bits: Width of reference counts in bits (default: 16) +# @compression-type: The image cluster compression method +# (default: zlib, since 5.0) # # Since: 2.12 ##@@ -4430,7 +4447,8 @@ '*cluster-size': 'size', '*preallocation': 'PreallocMode', '*lazy-refcounts': 'bool', - '*refcount-bits': 'int' } } + '*refcount-bits': 'int', + '*compression-type': 'Qcow2CompressionType' } } ## # @BlockdevCreateOptionsQed:
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 2/8] qcow2: rework the cluster compression routine
In reply to Denis PlotnikovQuoted 83 lines · Denis Plotnikov
The patch enables processing the image compression type defined for the image and chooses an appropriate method for image clusters (de)compression.
Signed-off-by Denis Plotnikov <[email protected]>--- block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 63 insertions(+), 14 deletions(-)block/qcow2-threads.c +13
-10@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData { } Qcow2CompressData;/* -* qcow2_compress() +* qcow2_zlib_compress() +* +* Compress @src_size bytes of data using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes @@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {* -ENOMEM destination buffer is not enough to store compressed data * -EIO on any other error */ -static ssize_t qcow2_compress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) {ssize_t ret;z_stream strm;@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size, }/* -* qcow2_decompress() +* qcow2_zlib_decompress() * * Decompress some data (not more than @src_size bytes) to produce exactly -* @dest_size bytes. +* @dest_size bytes using zlib compression method * * @dest - destination buffer, @dest_size bytes * @src - source buffer, @src_size bytes * * Returns: 0 on success -* -1 on fail +* -EIO on failure */ -static ssize_t qcow2_decompress (void *dest,size_t dest_size, -const void *src,size_t src_size) +static ssize_t qcow2_zlib_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) {int ret =0 ;z_stream strm;@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size, ret =inflateInit2 (&strm,-12 );if (ret !=Z_OK ) { -return -1 ; +return -EIO ; } ret =inflate (&strm,Z_FINISH );@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,* @src buffer may be processed partly (because in qcow2 we know size of * compressed data with precision of one sector) */ - ret =-1 ; + ret = -EIO ; }inflateEnd (&strm);@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,return arg.ret; } +/*
block/qcow2-threads.c qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, · line 204
Quoted 10 lines
block/qcow2-threads.c +10
-0+* qcow2_co_compress() +* +* Compress @src_size bytes of data using the compression +* method defined by the image compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* a negative error code on failure
Hmm, it's default semantics and it used without any comment for most of C functions, so I don't think we need the comment. As well as dest/src argument names are obvious enough. Still, I'm not against too.
Quoted 7 lines
block/qcow2-threads.c +1
-2+*/ ssize_t coroutine_fnqcow2_co_compress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_compress);
block/qcow2-threads.c qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size, · line 219
Quoted 10 lines
block/qcow2-threads.c +10
-0+BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_compress; +break ; + +default : +return -ENOTSUP ;
it can't be anything other. Maybe, better abort() ?
Quoted 41 lines
block/qcow2-threads.c +29
-2+ } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); } +/* +* qcow2_co_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using the compression method defined by the image +* compression type +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* a negative error code on failure +*/ ssize_t coroutine_fnqcow2_co_decompress (BlockDriverState *bs,void *dest,size_t dest_size,const void *src,size_t src_size) { -return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, - qcow2_decompress); +BDRVQcow2State *s = bs->opaque; +Qcow2CompressFunc fn; + +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : + fn = qcow2_zlib_decompress; +break ; + +default : +return -ENOTSUP ; + } + +return qcow2_co_do_compress (bs, dest, dest_size, src, src_size, fn); }
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Denis PlotnikovQuoted 64 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.imgcompression decompression
zlib zstd zlib zstd
------------------------------------------------------------real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Signed-off-by Denis Plotnikov <[email protected]>--- block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++ configure | 29 ++++++++++ docs/interop/qcow2.txt | 18 ++++++ qapi/block-core.json | 3 +- 5 files changed, 178 insertions(+), 1 deletion(-)block/qcow2-threads.c +8
-0@@ -28,6 +28,11 @@#define ZLIB_CONST #include <zlib.h> +#ifdef CONFIG_ZSTD +#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "qcow2.h" #include "block/thread-pool.h" #include "crypto.h" @@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,return ret; } +#ifdef CONFIG_ZSTD + +#define ZSTD_LEN_BUF 4
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 184
Quoted 10 lines
block/qcow2-threads.c +10
-0+ +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success
This doesn't match qcow2_co_compress definition. You should return 0 on success.
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 11 lines
block/qcow2-threads.c +11
-0+* -ENOMEM destination buffer is not enough to store compressed data +* -EIO on any other error +*/ + +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +size_t ret; + +/* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
Quoted 18 lines
block/qcow2-threads.c +18
-0+* to store compressed chunk size +*/ +char *d_buf = ((char *) dest) +ZSTD_LEN_BUF ; + +/* +* sanity check that we can store the compressed data length, +* and there is some space left for the compressor buffer +*/ +if (dest_size <=ZSTD_LEN_BUF ) { +return -ENOMEM ; + } + + dest_size -=ZSTD_LEN_BUF ; + + ret =ZSTD_compress (d_buf, dest_size, src, src_size,5 ); + +if (ZSTD_isError (ret)) { +if (ZSTD_getErrorCode (ret) == ZSTD_error_dstSize_tooSmall) {
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 223
Quoted 10 lines
block/qcow2-threads.c +10
-0+return -ENOMEM ; + }else { +return -EIO ; + } + } + +/* paraniod sanity check that we can store the commpressed size */ +if (ret >UINT_MAX ) { +return -ENOMEM ; + }
I'd use UINT32_MAX, possibly even more paranoid)
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 228
Quoted 5 lines
block/qcow2-threads.c +5
-0+ +/* store the compressed chunk size in the very beginning of the buffer */ +stl_be_p (dest, ret); + +return ret +ZSTD_LEN_BUF ;
return 0;
Quoted 5 lines
block/qcow2-threads.c +5
-0+} + +/* +* qcow2_zstd_decompress() +*
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 243
Quoted 10 lines
block/qcow2-threads.c +10
-0+* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: 0 on success +* -EIO on any error +*/ +
extra empty line.
Quoted 18 lines
block/qcow2-threads.c +18
-0+static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +/* +* zstd decompress wants to know the exact length of the data. +* For that purpose, on compression, the length is stored in +* the very beginning of the compressed buffer +*/ +size_t s_size; +const char *s_buf = ((const char *) src) +ZSTD_LEN_BUF ; + +/* +* sanity check that we can read 4 byte the content length and +* and there is some content to decompress +*/ +if (src_size <=ZSTD_LEN_BUF ) { +return -EIO ; + }
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 271
Quoted 10 lines
block/qcow2-threads.c +10
-0+ + s_size =ldl_be_p (src); + +/* sanity check that the buffer is big enough to read the content from */ +if (src_size -ZSTD_LEN_BUF < s_size) { +return -EIO ; + } + +if (ZSTD_isError ( +ZSTD_decompress (dest, dest_size, s_buf, s_size))) {
hmm, it fit into one line actually
Quoted 98 lines
block/qcow2-threads.c +17
-0+ return -EIO; + } + +return 0 ; +} +#endif +static int qcow2_compress_pool_func (void *opaque) {Qcow2CompressData *data = opaque;@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_compress; +break ; +#endif default :return -ENOTSUP ; }@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress;break ; +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : + fn = qcow2_zstd_decompress; +break ; +#endif default :return -ENOTSUP ; }block/qcow2.c +7
-0@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)switch (s->compression_type) {case QCOW2_COMPRESSION_TYPE_ZLIB : +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +#endif break ;default :@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) }switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD +case QCOW2_COMPRESSION_TYPE_ZSTD : +break ; +#endif default :error_setg_errno (errp, -EINVAL ,"Unknown compression type" ); goto out;configure +10
-0@@ -444,6 +444,7 @@ opengl_dmabuf="no" cpuid_h="no" avx2_opt="" zlib="yes" +zstd="" capstone="" lzo="" snappy=""@@ -1371,6 +1372,10 @@ for opt do ;; --disable-lzfse) lzfse="no" ;; + --enable-zstd) zstd="yes" + ;; + --disable-zstd) zstd="no" + ;; --enable-guest-agent) guest_agent="yes" ;; --disable-guest-agent) guest_agent="no"@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available: (for reading bzip2-compressed dmg images) lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backend@@ -2453,6 +2459,25 @@ EOF fi fi +######################################### +# zstd check + +if test "$zstd" != "no" ; then
configure EOF · line 2475
Quoted 10 lines
configure +10
-0+ cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOF + if compile_prog "" "-lzstd" ; then + LIBS="$LIBS -lzstd" + zstd="yes" + else + if test "$zstd" = "yes"; then + feature_not_found "zstd" "Install libzstd-devel"
to correspond to style used around: s/libzstd-devel/libzstd devel/
Quoted 47 lines
configure +9
-0+ fi + zstd="no" + fi +fi + ########################################## # libseccomp check@@ -6668,6 +6693,7 @@ echo "lzo support $lzo" echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "lzfse support $lzfse" +echo "zstd support $zstd" echo "NUMA host support $numa" echo "libxml2 $libxml2" echo "tcmalloc support $tcmalloc"@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then "\$ld_exported_symbols_list should have been set to 'yes'." fi fi +if test "$zstd" = "yes" ; then + echo "CONFIG_ZSTD=y" >> $config_host_mak +fi if test "$tcg_interpreter" = "yes"; then QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"docs/interop/qcow2.txt +4
-0@@ -208,6 +208,7 @@ version 2. Available compression type values: 0: zlib <https://www.zlib.net/> + 1: zstd <http://github.com/facebook/zstd> === Header padding ===@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): Another compressed cluster may map to the tail of the final sector used by this compressed cluster. + The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout). + If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 593
Quoted 10 lines
docs/interop/qcow2.txt +7
-0no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file. +=== Compressed Cluster Layout === + +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1)
Note, that term 'data_space_lenght' is not defined in the spec..
Hmm. And it's not trivial to define it correctly, all we have is offset and "number of additional sectors", and may be not all bytes of final sector are occupied... What about something like this:
In the following layout description byte 0 corresponds to byte at host cluster offset, as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy space starting from this offset, using additional 512-byte sectors defined by "Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes in the final sector. Let x be number of last byte of the layout.
docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 600
Quoted 7 lines
docs/interop/qcow2.txt +7
-0+ + 0: (default) zlib <http://zlib.net/>: + Byte 0 - x: the compressed data content + all the space provided used for compressed data + 1: zstd <http://github.com/facebook/zstd>: + Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data content
Maybe, note that x+! == 4 + length, where length is value of first field of the layout.
Quoted 22 lines
docs/interop/qcow2.txt +0
-0== Snapshots ==qapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 4/8] iotests: filter out compression_type
In reply to Denis PlotnikovQuoted 6 lines · Denis Plotnikov
After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started reporting the compression type for the image which breaks the iotests output matching.
To fix it, add compression_type=zlib to the list of filtered image parameters.
So, the first patch breaks iotests? Than it should be merged into first patch or moved before it, to not break git bisect.
Quoted 21 lines
Signed-off-by Denis Plotnikov <[email protected]>--- tests/qemu-iotests/common.filter | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)tests/qemu-iotests/common.filter +2
-1@@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=zlib##g" } _filter_img_info()
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset
In reply to Denis PlotnikovQuoted 5 lines · Denis Plotnikov
Affected tests: 031, 036, 061
Because of adding the compression type feature, some size values in the qcow2 v3 header are changed:
Quoted 7 lines
header_size+=8: 1 byte compression type
7 bytes paddingfeature_table+= 48: incompatible feture compression type
backing_file_offset+= 56 (8 + 48 ->header_change+fature_table_change)Change the values for the test output comparison accordingly.
Again, this should be merged to the patch, which actually break iotests.
Quoted 192 lines
Signed-off-by Denis Plotnikov <[email protected]>--- tests/qemu-iotests/031.out | 14 +++++++------- tests/qemu-iotests/036.out | 4 ++-- tests/qemu-iotests/061.out | 28 ++++++++++++++-------------- 3 files changed, 23 insertions(+), 23 deletions(-)tests/qemu-iotests/031.out +7
-7@@ -113,11 +113,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:@@ -146,11 +146,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:@@ -164,7 +164,7 @@ No errors were found on the image. magic 0x514649fb version 3 -backing_file_offset 0x178 +backing_file_offset 0x1b0 backing_file_size 0x17 cluster_bits 16 size 67108864@@ -179,7 +179,7 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0xe2792aca@@ -188,7 +188,7 @@ data 'host_device' Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> Header extension:tests/qemu-iotests/036.out +2
-2@@ -26,7 +26,7 @@ compatible_features [] autoclear_features [63] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary>@@ -38,7 +38,7 @@ compatible_features [] autoclear_features [] Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> *** donetests/qemu-iotests/061.out +14
-14@@ -22,11 +22,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb@@ -80,11 +80,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb@@ -136,11 +136,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1@@ -191,11 +191,11 @@ incompatible_features [] compatible_features [42] autoclear_features [42] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> magic 0x514649fb@@ -260,11 +260,11 @@ incompatible_features [] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 65536/65536 bytes at offset 44040192@@ -294,11 +294,11 @@ incompatible_features [0] compatible_features [0] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> ERROR cluster 5 refcount=0 reference=1@@ -323,11 +323,11 @@ incompatible_features [] compatible_features [] autoclear_features [] refcount_order 4 -header_length 104 +header_length 112 Header extension: magic 0x6803f857 -length 192 +length 240 data <binary> read 131072/131072 bytes at offset 0
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
In reply to Denis PlotnikovQuoted 7 lines · Denis Plotnikov
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output matching.
And this and the following patch.
Ideally, patch should not break any iotests. This means that all iotest updates should be merged to the patch which changes their output. Probably, you can split behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch with a lot of iotests updates is better than patch which breaks iotests.
Quoted 471 lines
Signed-off-by Denis Plotnikov <[email protected]>--- tests/qemu-iotests/049.out | 102 ++++++++++++++++++------------------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 6 +++ tests/qemu-iotests/065 | 20 +++++--- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-)tests/qemu-iotests/049.out +51
-51@@ -4,90 +4,90 @@ QA output created by 049 == 1. Traditional size parameter == qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 2. Specifying size via -o == qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 3. Invalid sizes ==@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once == Check correct interpretation of suffixes for cluster size == qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check compat level option == qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check preallocation option == qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234' -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check encryption option == qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Check lazy_refcounts option (only with v3) == qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib *** donetests/qemu-iotests/060.out +1
-0@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: truetests/qemu-iotests/061.out +6
-0@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: foo@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file raw: false@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.data@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/065 +12
-8@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific): class TestQCow2(TestQemuImgInfo): '''Testing a qcow2 version 2 image''' img_options = 'compat=0.10' - json_compare = { 'compat': '0.10', 'refcount-bits': 16 } - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ] + json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' } + human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ] class TestQCow3NotLazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts disabled''' img_options = 'compat=1.1,lazy_refcounts=off' json_compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: false', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo): '''Testing a qcow2 version 3 image with lazy refcounts enabled''' img_options = 'compat=1.1,lazy_refcounts=on' json_compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } - human_compare = [ 'compat: 1.1', 'lazy refcounts: true', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true', 'refcount bits: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP):@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=off' qemu_options = 'lazy-refcounts=on' compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } class TestQCow3LazyQMP(TestQMP):@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP): img_options = 'compat=1.1,lazy_refcounts=on' qemu_options = 'lazy-refcounts=off' compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } TestImageInfoSpecific = None TestQemuImgInfo = Nonetests/qemu-iotests/144.out +2
-2@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912 { 'execute': 'qmp_capabilities' } {"return": {}} { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} === Performing block-commit on active layer ===@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/ === Performing Live Snapshot 2 === { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} *** donetests/qemu-iotests/182.out +1
-1@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]? {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}} {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } } -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}tests/qemu-iotests/242.out +5
-0@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes) cluster_size: 65536 Format specific information: compat: 1.1 + compression type: zlib lazy refcounts: false bitmaps: [0]:tests/qemu-iotests/255.out +4
-4@@ -3,9 +3,9 @@ Finishing a commit job with background reads === Create backing chain and start VM === -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib === Start background read requests ===@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled === Create images and start VM === -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib wrote 1048576/1048576 bytes at offset 0 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Denis Plotnikov
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output matching.
Quoted 6 lines
And this and the following patch.
Ideally, patch should not break any iotests. This means that all iotest updates should be merged to the patch which changes their output. Probably, you can split behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch with a lot of iotests updates is better than patch which breaks iotests.
Or we can try to reduce behavior changes in case of zlib:
- keep header small in case of zlib, not adding zero field
- don't add feature table entry, if compress type is zlib
- don't report compression type on quering, if it is zlib
- then, all iotests output will be saved. And, then, if we need, we can change these theree points one-by-one, updating iotests outputs. But I doubt that we really need it, compatible behavior seems good enough.
Quoted 398 lines
Quoted 396 lines
Signed-off-by Denis Plotnikov <[email protected]>---
tests/qemu-iotests/049.out | 102 ++++++++++++++++++------------------- tests/qemu-iotests/060.out | 1 + tests/qemu-iotests/061.out | 6 +++ tests/qemu-iotests/065 | 20 +++++--- tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-)tests/qemu-iotests/049.out +0
-0@@ -4,90 +4,90 @@ QA output created by 049== 1. Traditional size parameter ==
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib== 2. Specifying size via -o ==
qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib== 3. Invalid sizes ==
tests/qemu-iotests/049.out +0
-0@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once== Check correct interpretation of suffixes for cluster size == qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib== Check compat level option ==
qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64Mtests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib== Check preallocation option ==
qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64Mtests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib== Check encryption option ==
qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64Mtests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib== Check
lazy_refcountsoption (only with v3) ==
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64Mtests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibqemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)tests/qemu-iotests/049.out +1
-1-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib*** done
tests/qemu-iotests/060.out +0
-0@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: truetests/qemu-iotests/061.out +0
-0@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: footests/qemu-iotests/061.out +0
-0@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file raw: falsetests/qemu-iotests/061.out +0
-0@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/065 +0
-0@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
img_options= 'compat=0.10'
-json_compare= { 'compat': '0.10', 'refcount-bits': 16 }
-human_compare= [ 'compat: 0.10', 'refcount bits: 16' ]
+json_compare= { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' } +human_compare= [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ] class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''img_options= 'compat=1.1,lazy_refcounts=off'
json_compare= { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
-human_compare= [ 'compat: 1.1', 'lazy refcounts: false',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+human_compare= [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false', 'refcount bits: 16', 'corrupt: false' ]
class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''img_options= 'compat=1.1,lazy_refcounts=on'
json_compare= { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
-human_compare= [ 'compat: 1.1', 'lazy refcounts: true',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+human_compare= [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true', 'refcount bits: 16', 'corrupt: false' ]
class TestQCow3NotLazyQMP(TestQMP):tests/qemu-iotests/065 +0
-0@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
img_options= 'compat=1.1,lazy_refcounts=off'qemu_options= 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } class TestQCow3LazyQMP(TestQMP):tests/qemu-iotests/065 +0
-0@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
img_options= 'compat=1.1,lazy_refcounts=on'qemu_options= 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } TestImageInfoSpecific = None
TestQemuImgInfo = Nonetests/qemu-iotests/144.out +1
-1@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912 { 'execute': 'qmp_capabilities' } {"return": {}} { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}}=== Performing block-commit on active layer ===
tests/qemu-iotests/144.out +0
-0@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/=== Performing Live Snapshot 2 ===
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }tests/qemu-iotests/144.out +1
-1-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib{"return": {}}
*** donetests/qemu-iotests/182.out +1
-1@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]? {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}} {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } } -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}} {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}tests/qemu-iotests/242.out +0
-0@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: falsetests/qemu-iotests/242.out +0
-0@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/255.out +0
-0@@ -3,9 +3,9 @@ Finishing a commit job with background reads=== Create backing chain and start VM ===
tests/qemu-iotests/255.out +2
-2-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib=== Start background read requests ===
tests/qemu-iotests/255.out +0
-0@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled=== Create images and start VM ===
tests/qemu-iotests/255.out +2
-2-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlibwrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
In reply to Denis PlotnikovQuoted 55 lines · Denis Plotnikov
The test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.
Signed-off-by Denis Plotnikov <[email protected]>--- tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/287.out | 41 +++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 165 insertions(+)create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.outtests/qemu-iotests/287 new file +37
-0@@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Test case for an image using zstd compression +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# standard environment +. ./common.rc +. ./common.filter
tests/qemu-iotests/287 line 47
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+ +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"` + +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null +RES=$?
Hmm. This will not work for other architectures and for out of tree builds. Also, it checks config but not current binary (they may be out of sync, or even unrelated).
Probably better try to create image with zstd compression type and handle expected error.
Quoted 11 lines
tests/qemu-iotests/287 new file +11
-0+if (($RES)); then + _notrun "ZSTD is disabled in the current configuration" +fi + +# Test: when compression is zlib the incompatible is unset +echo +echo "=== Testing compression type incompatible bit setting for zlib ===" +echo + +_make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
tests/qemu-iotests/287 line 68
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+ +# Test: when compression differs from zlib the incompatible bit is set +echo +echo "=== Testing compression type incompatible bit setting for zstd ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: an image can't be openned if compression type is zlib and
opened
tests/qemu-iotests/287 line 81
Quoted 13 lines
tests/qemu-iotests/287 new file +13
-0+# incompatible feature compression type is set +echo +echo "=== Testing zlib with incompatible bit set ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3 +# to make sure the bit was actually set +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi
may be better to instead keep error output and just check it..
Quoted 27 lines
tests/qemu-iotests/287 new file +27
-0+ +# Test: an image can't be openned if compression type is NOT zlib and +# incompatible feature compression type is UNSET +echo +echo "=== Testing zstd with incompatible bit unset ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0 +# to make sure the bit was actually unset +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi +# Test: check compression type values +echo +echo "=== Testing compression type values ===" +echo +# zlib=0 +IMGOPTS='compression_type=zlib' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# zstd=1 +IMGOPTS='compression_type=zstd' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" +
tests/qemu-iotests/287 line 118
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo + +CLUSTER_SIZE=65536 +IMGOPTS='compression_type=zstd' _make_test_img 64M +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
Hmm output depends on default pattern. Better use "write -c -P 0x11 0 64k"
(or any pattern you want), to make it explicit.
Quoted 62 lines
tests/qemu-iotests/287 new file +5
-0+ +# success, all done +echo "*** done" +rm -f $seq.full +status=0tests/qemu-iotests/287.out new file +41
-0@@ -0,0 +1,41 @@ +QA output created by 287 + +=== Testing compression type incompatible bit setting for zlib === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [] + +=== Testing compression type incompatible bit setting for zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +incompatible_features [3] + +=== Testing zlib with incompatible bit set === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [3] + +=== Testing zstd with incompatible bit unset === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +incompatible_features [] + +=== Testing compression type values === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 + 0 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd + 1 + +=== Testing reading and writing with zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +00000000: cd cd cd cd cd cd cd cd cd cd .......... +read 10/10 bytes at offset 0 +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +0000fffa: cd cd cd cd cd cd 00 00 ........ +read 8/8 bytes at offset 65530 +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** donetests/qemu-iotests/group +1
-0@@ -293,3 +293,4 @@ 283 auto quick 284 rw 286 rw quick +287 auto quick
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Vladimir Sementsov-OgievskiyQuoted 31 lines · Vladimir Sementsov-Ogievskiy
block/qcow2.c
Quoted 27 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++---include/block/block_int.h| 1 +
qapi/block-core.json | 22 +++++++-4 files changed, 150 insertions(+), 9 deletions(-)block/qcow2.c +0
-0Please, add to .git/config:
Quoted 4 lines
[diff]
orderFile = /path/to/qemu/scripts/git.orderfileThis will force git format-patch to sort files in more comfortable order (header changes first, etc).
As I learned yesterday, git 2.23 and 2.24 have a bug where git format-patch fails to honor diff.orderfile (fixed in 2.25): https://bugzilla.redhat.com/show_bug.cgi?id=1807681
(and that explains why some of my recent patches have not been ordered the way I wanted, as Fedora 31 currently has a git from the broken window in time)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Denis PlotnikovQuoted 2 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression
to qcow2, allowing the use of different
Quoted 31 lines
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Signed-off-by Denis Plotnikov <[email protected]>--- block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++--- include/block/block_int.h | 1 + qapi/block-core.json | 22 +++++++- 4 files changed, 150 insertions(+), 9 deletions(-)block/qcow2.c +2
-0@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,return ret; } +static int validate_compression_type (BDRVQcow2State *s,Error **errp) +{
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1256
Quoted 10 lines
block/qcow2.c +10
-0+/* +* Sanity check +* according to qcow2 spec, the compression type is 1-byte field +* but in BDRVQcow2State the compression_type is enum sizeof(int) +* so, the max compression_type value is 255. +*/ +if (s->compression_type >0xff ) { +error_setg (errp,"qcow2: compression type value is too big" ); +return -EINVAL ; + }
Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104: compression_type
105 - 111: padding, must be 0
or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1266
Quoted 10 lines
block/qcow2.c +10
-0+ +switch (s->compression_type) { +case QCOW2_COMPRESSION_TYPE_ZLIB : +break ; + +default : +error_setg (errp,"qcow2: unknown compression type: %u" , + s->compression_type); +return -ENOTSUP ; + }
Having two checks feels redundant, compared to just letting the default catch all unrecognized values in that field.
Quoted 8 lines
block/qcow2.c +8
-0+ +/* +* if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB +* the incompatible feature flag must be set +*/ +if (s->compression_type ==QCOW2_COMPRESSION_TYPE_ZLIB ) { +if (s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE ) { + error_setg(errp, "qcow2: Compression type incompatible feature "
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1284
Quoted 10 lines
block/qcow2.c +10
-0+ "bit must not be set"); +return -EINVAL ; + } + }else { +if (!(s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE )) { +error_setg (errp,"qcow2: Compression type incompatible feature " +"bit must be set" ); +return -EINVAL ; + } + }
Matches what we documented in the spec.
Quoted 12 lines
block/qcow2.c +5
-0+ +return 0 ; +} +/* Called with s->lock held. */ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,int flags,Error **errp)@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, s->compatible_features = header.compatible_features; s->autoclear_features = header.autoclear_features; +/*
block/qcow2.c static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, · line 1414
Quoted 10 lines
block/qcow2.c +10
-0+* Handle compression type +* Older qcow2 images don't contain the compression type header. +* Distinguish them by the header length and use +* the only valid (default) compression type in that case +*/ +if (header.header_length > offsetof(QCowHeader , compression_type)) { +/* +* don't deal with endians since compression_type is 1 byte long +*/ + s->compression_type = header.compression_type;
Changes if you go with my suggestion of just making the compression_type field occupy 8 bytes in the qcow2 header. (And if you want to keep it 1 byte, I still think the spec should call out explicit padding bytes).
Quoted 26 lines
block/qcow2.c +15
-0+ }else { + s->compression_type =QCOW2_COMPRESSION_TYPE_ZLIB ; + } + + ret =validate_compression_type (s, errp); +if (ret) { + goto fail; + } +if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK ) {void *feature_table =NULL ; qcow2_read_extensions(bs, header.header_length, ext_end,@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs) total_size = bs->total_sectors *BDRV_SECTOR_SIZE ; refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits -3 ); + ret =validate_compression_type (s,NULL ); + +if (ret) { + goto fail; + } + *header = (QCowHeader ) {/* Version 2 fields */ .magic =cpu_to_be32 (QCOW_MAGIC ),
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2815
Quoted 4 lines
block/qcow2.c +1
-0@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features = cpu_to_be64(s->autoclear_features), .refcount_order = cpu_to_be32(s->refcount_order), .header_length = cpu_to_be32(header_length), + .compression_type = (uint8_t) s->compression_type,
Is the cast necessary?
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2914
Quoted 9 lines
block/qcow2.c +2
-0};/* For older versions, write a shorter header */ @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
Ordering: please group all the incompatible bits side-by-side (this should come before the lazy refcount bit).
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2916
Quoted 2 lines
block/qcow2.c +2
-0+ .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name = "compression type",
This change breaks iotests; at least 31, 36, and 61 need updates (I've got a similar patch pending which fixes the fact that we forgot the autoclear bit [1]). You'll need to squash in fixes for those at the same time.
[1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3480
Quoted 9 lines
block/qcow2.c +2
-0+ }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), .refcount_order = cpu_to_be32(refcount_order), + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
Is the cast necessary?
Quoted 10 lines
block/qcow2.c +3
-0.header_length = cpu_to_be32(sizeof(*header)), };@@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)cpu_to_be64 (QCOW2_AUTOCLEAR_DATA_FILE_RAW ); } +if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type !=QCOW2_COMPRESSION_TYPE_ZLIB ) { +
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3512
Quoted 10 lines
block/qcow2.c +10
-0+if (qcow2_opts->compression_type >0xff ) { +error_setg_errno (errp, -EINVAL ,"Too big compression type value" ); + goto out; + } + +switch (qcow2_opts->compression_type) { +default : +error_setg_errno (errp, -EINVAL ,"Unknown compression type" ); + goto out; + }
This should probably be an assert that qcow2_opts->compression_type is in range, rather than a switch statement and error_setg. Callers of qcow2_co_create should not be handing us unknown values.
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3514
Quoted 2 lines
block/qcow2.c +2
-0+ + header->compression_type = (uint8_t ) qcow2_opts->compression_type;
Why the cast?
Quoted 12 lines
block/qcow2.c +6
-0+ + header->incompatible_features |= +cpu_to_be64 (QCOW2_INCOMPAT_COMPRESSION_TYPE ); + } + ret =blk_pwrite (blk,0 , header, cluster_size,0 );g_free (header);if (ret <0 ) {@@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt +++ b/block/qcow2.h
block/qcow2.c typedef struct QCowHeader { · line 155
Quoted 10 lines
block/qcow2.c +6
-0@@ -146,6 +146,12 @@ typedef struct QCowHeader {uint32_t refcount_order;uint32_t header_length; + +/* Additional fields */ +uint8_t compression_type; + +/* header must be a multiple of 8 */ +uint8_t padding[7 ]; }QEMU_PACKED QCowHeader;
You're changing the size of this struct, which WILL break iotests (and even more than just the 3 I pointed out above for the feature name table).
/me looks ahead
Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to be squashed in here, where the change is made.
Quoted 11 lines
block/qcow2.c +0
-5typedef struct QEMU_PACKED QCowSnapshotHeader {@@ -213,16 +219,20 @@ enum {/* Incompatible feature bits */ enum { -QCOW2_INCOMPAT_DIRTY_BITNR =0 , -QCOW2_INCOMPAT_CORRUPT_BITNR =1 , -QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , -QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , -QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR ,
block/qcow2.c enum { · line 230
Quoted 10 lines
block/qcow2.c +9
-1- QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR, +QCOW2_INCOMPAT_DIRTY_BITNR =0 , +QCOW2_INCOMPAT_CORRUPT_BITNR =1 , +QCOW2_INCOMPAT_DATA_FILE_BITNR =2 , +QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR =3 , +QCOW2_INCOMPAT_DIRTY =1 <<QCOW2_INCOMPAT_DIRTY_BITNR , +QCOW2_INCOMPAT_CORRUPT =1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , +QCOW2_INCOMPAT_DATA_FILE =1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , +QCOW2_INCOMPAT_COMPRESSION_TYPE = + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
Uggh. I hate realigning = just because we added a longer name, especially when you then can't even keep things on one line because of length. If it were me, I'd leave the existing lines alone, and/or switch everything to just use 'BITNR = ' rather than trying to align =.
Bikeshedding - since the new name is so long, can you get by with the shorter QCOW2_INCOMPAT_COMPRESSION_BITNR (drop the _TYPE)?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
Quoted 49 lines · Eric Blake
Quoted 2 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression
to qcow2, allowing the use of different
Quoted 30 lines
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++---include/block/block_int.h| 1 +
qapi/block-core.json | 22 +++++++-4 files changed, 150 insertions(+), 9 deletions(-)block/qcow2.c +2
-0@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options, return ret; } +static int validate_compression_type (BDRVQcow2State *s,Error **errp) +{block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1255
Quoted 10 lines
block/qcow2.c +10
-0+ /* + * Sanity check + * according to qcow2 spec, the compression type is 1-byte field + * but in BDRVQcow2State the compression_type is enum sizeof(int) + * so, the max compression_type value is 255. + */ + if (s->compression_type > 0xff) { + error_setg(errp, "qcow2: compression type value is too big"); + return -EINVAL; + }Hmm - I think it may be worth a tweak to qcow2.txt to call out:
Quoted 8 lines
104:
compression_type
105 - 111: padding, must be 0or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
Quoted 198 lines
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1266
Quoted 11 lines
block/qcow2.c +10
-0+ + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + break; + + default: + error_setg(errp, "qcow2: unknown compression type: %u", + s->compression_type); + return -ENOTSUP; + }Having two checks feels redundant, compared to just letting the default catch all unrecognized values in that field.
Quoted 8 lines
block/qcow2.c +8
-0+ + /* + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB + * the incompatible feature flag must be set + */ + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) { + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) { + error_setg(errp, "qcow2: Compression type incompatible feature "block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1284
Quoted 10 lines
block/qcow2.c +10
-0+ "bit must not be set"); + return -EINVAL; + } + } else { + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) { + error_setg(errp, "qcow2: Compression type incompatible feature " + "bit must be set"); + return -EINVAL; + } + }Matches what we documented in the spec.
Quoted 11 lines
block/qcow2.c +5
-0+ + return 0; +} + /* Called with s->lock held. */ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, int flags, Error **errp)@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, s->compatible_features = header.compatible_features; s->autoclear_features = header.autoclear_features; + /*block/qcow2.c static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options, · line 1413
Quoted 10 lines
block/qcow2.c +10
-0+* Handle compression type +* Older qcow2 images don't contain the compression type header. +* Distinguish them by the header length and use +* the only valid (default) compression type in that case +*/ + if (header.header_length > offsetof(QCowHeader, compression_type)) { + /* + * don't deal with endians since compression_type is 1 byte long + */ + s->compression_type = header.compression_type;Changes if you go with my suggestion of just making the
compression_typefield occupy 8 bytes in the qcow2 header. (And if you want to keep it 1 byte, I still think the spec should call out explicit padding bytes).Quoted 25 lines
block/qcow2.c +15
-0+ } else { + s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB; + } + + ret = validate_compression_type(s, errp); + if (ret) { + goto fail; + } + if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) { void *feature_table = NULL; qcow2_read_extensions(bs, header.header_length, ext_end,@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs) total_size = bs->total_sectors * BDRV_SECTOR_SIZE; refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3); + ret = validate_compression_type(s, NULL); + + if (ret) { + goto fail; + } + *header = (QCowHeader) { /* Version 2 fields */ .magic = cpu_to_be32(QCOW_MAGIC),block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2815
Quoted 4 lines
block/qcow2.c +1
-0@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features = cpu_to_be64(s->autoclear_features), .refcount_order = cpu_to_be32(s->refcount_order), .header_length = cpu_to_be32(header_length), + .compression_type = (uint8_t) s->compression_type,Is the cast necessary?
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2914
Quoted 8 lines
block/qcow2.c +2
-0}; /* For older versions, write a shorter header */@@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,Ordering: please group all the incompatible bits side-by-side (this should come before the lazy refcount bit).
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2916
Quoted 2 lines
block/qcow2.c +2
-0+ .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name = "compression type",This change breaks iotests; at least 31, 36, and 61 need updates (I've got a similar patch pending which fixes the fact that we forgot the autoclear bit [1]). You'll need to squash in fixes for those at the same time.
[1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3480
Quoted 8 lines
block/qcow2.c +2
-0+ }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), .refcount_order = cpu_to_be32(refcount_order), + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,Is the cast necessary?
Quoted 8 lines
block/qcow2.c +3
-0.header_length = cpu_to_be32(sizeof(*header)), };@@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW); } + if (qcow2_opts->has_compression_type && + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) { +block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3511
Quoted 10 lines
block/qcow2.c +10
-0+ if (qcow2_opts->compression_type > 0xff) { + error_setg_errno(errp, -EINVAL, "Too big compression type value"); + goto out; + } + + switch (qcow2_opts->compression_type) { + default: + error_setg_errno(errp, -EINVAL, "Unknown compression type"); + goto out; + }This should probably be an assert that
qcow2_opts->compression_typeis in range, rather than a switch statement and error_setg. Callers ofqcow2_co_createshould not be handing us unknown values.block/qcow2.c qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) · line 3513
Quoted 2 lines
block/qcow2.c +2
-0+ + header->compression_type = (uint8_t) qcow2_opts->compression_type;Why the cast?
Quoted 12 lines
block/qcow2.c +6
-0+ + header->incompatible_features |= + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE); + } + ret = blk_pwrite(blk, 0, header, cluster_size, 0); g_free(header); if (ret < 0) {@@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt +++ b/block/qcow2.hblock/qcow2.c typedef struct QCowHeader { · line 154
Quoted 9 lines
block/qcow2.c +6
-0@@ -146,6 +146,12 @@ typedef struct QCowHeader { uint32_t refcount_order; uint32_t header_length; + + /* Additional fields */ + uint8_t compression_type; + + /* header must be a multiple of 8 */ + uint8_t padding[7]; } QEMU_PACKED QCowHeader;You're changing the size of this struct, which WILL break iotests (and even more than just the 3 I pointed out above for the feature name table).
/me looks ahead
Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to be squashed in here, where the change is made.
Quoted 9 lines
block/qcow2.c +0
-5typedef struct QEMU_PACKED QCowSnapshotHeader {@@ -213,16 +219,20 @@ enum { /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR = 0, - QCOW2_INCOMPAT_CORRUPT_BITNR = 1, - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2, - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,block/qcow2.c enum { · line 229
Quoted 10 lines
block/qcow2.c +9
-1- QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR, + QCOW2_INCOMPAT_DIRTY_BITNR = 0, + QCOW2_INCOMPAT_CORRUPT_BITNR = 1, + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2, + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3, + QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR, + QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR, + QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR, + QCOW2_INCOMPAT_COMPRESSION_TYPE = + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,Uggh. I hate realigning = just because we added a longer name, especially when you then can't even keep things on one line because of length. If it were me, I'd leave the existing lines alone, and/or switch everything to just use 'BITNR = ' rather than trying to align =.
Bikeshedding - since the new name is so long, can you get by with the shorter
QCOW2_INCOMPAT_COMPRESSION_BITNR(drop the_TYPE)?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Denis PlotnikovQuoted 7 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 7 lines
block/qcow2-threads.c +7
-0+static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, +const void *src,size_t src_size) +{ +size_t ret; + +/* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
Quoted 15 lines
block/qcow2-threads.c +15
-0+* to store compressed chunk size +*/ +char *d_buf = ((char *) dest) +ZSTD_LEN_BUF ; + +/* +* sanity check that we can store the compressed data length, +* and there is some space left for the compressor buffer +*/ +if (dest_size <=ZSTD_LEN_BUF ) { +return -ENOMEM ; + } + + dest_size -=ZSTD_LEN_BUF ; + + ret =ZSTD_compress (d_buf, dest_size, src, src_size,5 );
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 220
Quoted 10 lines
block/qcow2-threads.c +10
-0+ +if (ZSTD_isError (ret)) { +if (ZSTD_getErrorCode (ret) == ZSTD_error_dstSize_tooSmall) { +return -ENOMEM ; + }else { +return -EIO ; + } + } + +/* paraniod sanity check that we can store the commpressed size */
paranoid, compressed
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 223
Quoted 3 lines
block/qcow2-threads.c +3
-0+if (ret >UINT_MAX ) { +return -ENOMEM ; + }
This is pointless. Better is to ensure that we actually compressed data (the pigeonhole principle states that there are some inputs that MUST result in inflation, in order for most other inputs to result in compression). But that check was satisfied by checking for ZSTD_error_dstSize_tooSmall, which is what happens for one of those uncompressible inputs. Namely, zstd will never return a result larger than dest_size, and since dest_size is smaller than UINT_MAX on entry, this check is pointless. But if you want something, I'd be okay with: assert(ret <= dest_size).
Quoted 44 lines
+++
b/docs/interop/qcow2.txtdocs/interop/qcow2.txt +18
-0@@ -208,6 +208,7 @@ version 2. Available compression type values: 0: zlib <https://www.zlib.net/> + 1: zstd <http://github.com/facebook/zstd> === Header padding ===@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): Another compressed cluster may map to the tail of the final sector used by this compressed cluster. + The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout). + If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file. +=== Compressed Cluster Layout === + +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1) + + 0: (default) zlib <http://zlib.net/>: + Byte 0 - x: the compressed data content + all the space provided used for compressed data + 1: zstd <http://github.com/facebook/zstd>: + Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data content == Snapshots ==qapi/block-core.json +0
-0@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file
qapi/block-core.json line 4410
Quoted 10 lines
qapi/block-core.json +2
-1# # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
The spec and UI changes are okay.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 4/8] iotests: filter out compression_type
In reply to Denis PlotnikovQuoted 2 lines · Denis Plotnikov
After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started
settings
Quoted 9 lines
reporting the compression type for the image which breaks the iotests output matching.
To fix it, add compression_type=zlib to the list of filtered image parameters.
Signed-off-by Denis Plotnikov <[email protected]>--- tests/qemu-iotests/common.filter | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
This should be squashed in to the patch that caused the breakage (3/8, if I'm right).
tests/qemu-iotests/common.filter _filter_img_create() · line 156
Quoted 12 lines
tests/qemu-iotests/common.filter +2
-1@@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=zlib##g"
Do you really want to hard-code just zlib, or should this be more generic as compression_type=[a-zA-Z0-9]\\+ as done on other lines like key-secret?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
In reply to Vladimir Sementsov-OgievskiyQuoted 12 lines · Vladimir Sementsov-Ogievskiy
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Denis Plotnikov
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output matching.
And this and the following patch.
Quoted 8 lines
Quoted 4 lines
Ideally, patch should not break any iotests. This means that all iotest updates should be merged to the patch which changes their output. Probably, you can split behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch with a lot of iotests updates is better than patch which breaks iotests.
Or we can try to reduce behavior changes in case of zlib:
- keep header small in case of zlib, not adding zero field
No, I don't see the point in doing that. It makes maintenance harder to try to minimize the header, for no gain (older images parse the larger header just fine).
Quoted 8 lines
- don't add feature table entry, if compress type is zlib
- don't report compression type on quering, if it is zlib- then, all iotests output will be saved. And, then, if we need, we can change
these theree points one-by-one, updating iotests outputs. But I doubt that we really need it, compatible behavior seems good enough.
[by the way, replying to this series is painful, since [email protected] is not a valid email address]
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Vladimir Sementsov-OgievskiyQuoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.imgcompression decompression zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++
configure | 29 ++++++++++docs/interop/qcow2.txt| 18 ++++++
qapi/block-core.json | 3 +-5 files changed, 178 insertions(+), 1 deletion(-)block/qcow2-threads.c +0
-0@@ -28,6 +28,11 @@ #define ZLIB_CONST #include <zlib.h>+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +4
-0+#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "
qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"block/qcow2-threads.c +0
-0@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, return ret; }+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +6
-0+ +#define ZSTD_LEN_BUF 4 + +/* +* qcow2_zstd_compress() +*
Quoted 8 lines
Quoted 6 lines
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on successThis doesn't match
qcow2_co_compressdefinition. You should return 0 on success.
does it? I'd rather say it doesn't match to qcow2_co_compress description in the function comment, which we can change actually, because qcow2_co_compress is used like:
block/qcow2.c:
static coroutine_fn int qcow2_co_pwritev_compressed_task(BlockDriverState *bs, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, size_t qiov_offset) {
...
out_buf = g_malloc(s->cluster_size);
out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1, buf, s->cluster_size);
if (out_len == -ENOMEM) {
/* could not compress: write normal cluster */
ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov, qiov_offset, 0); if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len < 0) {
ret = -EINVAL;
goto fail;
}
qemu_co_mutex_lock(&s->lock);
ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len, <<<<<<<<<<<< &cluster_offset);
...
}
Quoted 38 lines
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 13 lines
+ *
-ENOMEMdestination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */block/qcow2-threads.c +8
-0+ +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + + /* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer beginning
Quoted 22 lines
+ * to store compressed chunk size
+ */
+ char*d_buf= ((char *) dest) +ZSTD_LEN_BUF;block/qcow2-threads.c +19
-0+ + /* + * sanity check that we can store the compressed data length, + * and there is some space left for the compressor buffer + */ + if (dest_size <= ZSTD_LEN_BUF) { + return -ENOMEM; + } + + dest_size -= ZSTD_LEN_BUF; + + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5); + + if (ZSTD_isError(ret)) { + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) { + return -ENOMEM; + } else { + return -EIO; + }
Quoted 8 lines
Quoted 6 lines
+ }
+
+ /* paraniod sanity check that we can store the commpressed size */ + if (ret >UINT_MAX) {
+ return-ENOMEM;
+ }I'd use
UINT32_MAX, possibly even more paranoid)
ok
Quoted 21 lines
Quoted 8 lines
+
+ /* store the compressed chunk size in the very beginning of the buffer */
+stl_be_p(dest, ret);
+
+ return ret +ZSTD_LEN_BUF;return 0;
Quoted 10 lines
block/qcow2-threads.c +6
-0+} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
Quoted 8 lines
Quoted 6 lines
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *-EIOon any error
+ */
+extra empty line.
will be removed
Quoted 25 lines
Quoted 25 lines
block/qcow2-threads.c +22
-0+static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + /* + * zstd decompress wants to know the exact length of the data. + * For that purpose, on compression, the length is stored in + * the very beginning of the compressed buffer + */ + size_t s_size; + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF; + + /* + * sanity check that we can read 4 byte the content length and + * and there is some content to decompress + */ + if (src_size <= ZSTD_LEN_BUF) { + return -EIO; + } + + s_size = ldl_be_p(src); + + /* sanity check that the buffer is big enough to read thecontent from */
Quoted 8 lines
Quoted 6 lines
+ if (
src_size-ZSTD_LEN_BUF<s_size) {
+ return-EIO;
+ }
+
+ if (ZSTD_isError(
+ZSTD_decompress(dest, dest_size, s_buf, s_size))) {hmm, it fit into one line actually
will be fixed if so
Quoted 103 lines
Quoted 103 lines
+ return
-EIO;
+ }block/qcow2-threads.c +5
-0+ + return 0; +} +#endif +static int
qcow2_compress_pool_func(void *opaque){
Qcow2CompressData *data = opaque;block/qcow2-threads.c +0
-0@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_compress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2-threads.c +0
-0@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_decompress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2.c +0
-0@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) switch (s->compression_type) {case
QCOW2_COMPRESSION_TYPE_ZLIB:block/qcow2.c +3
-0+#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break;
default:block/qcow2.c +4
-0@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; +#endif default:
error_setg_errno(errp, -EINVAL, "Unknown compression type"); goto out;configure +0
-0@@ -444,6 +444,7 @@ opengl_dmabuf="no"cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""configure +0
-0@@ -1371,6 +1372,10 @@ for opt do ;;--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"configure +0
-0@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default isenabled if available:
(for reading bzip2-compressed dmg images) lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support
coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendconfigure +0
-0@@ -2453,6 +2459,25 @@ EOFfi
fi
+#########################################configure +7
-0+# zstd check + +if test "$zstd" != "no" ; then + cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOF
Quoted 8 lines
Quoted 6 lines
+ if
compile_prog"" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ zstd="yes"
+ else
+ if test "$zstd" = "yes"; then
+feature_not_found"zstd" "Install libzstd-devel"to correspond to style used around: s/libzstd-devel/libzstd devel/
My intention was to give a package name for a user encountered this message to copy-paste it to "yum install". If this doesn't make things easier, because the package name can be changed in the future (although I doubt) I'll fix it. If the style isn't that important I'd rather leave it as is.
Quoted 60 lines
Quoted 47 lines
+ fi
+ zstd="no"
+ ficonfigure +2
-0+fi +##########################################
# libseccomp check @@ -6668,6 +6693,7 @@ echo "lzo support $lzo" echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "lzfse support $lzfse" +echo "zstd support $zstd" echo "NUMA host support $numa" echo "libxml2 $libxml2" echo "tcmalloc support $tcmalloc"configure +0
-0@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then"\$ld_exported_symbols_list should have been set to 'yes'." fi
ficonfigure +3
-0+if test "$zstd" = "yes" ; then + echo "CONFIG_ZSTD=y" >> $config_host_mak +fiif test "$tcg_interpreter" = "yes"; then QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
docs/interop/qcow2.txt +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===docs/interop/qcow2.txt +0
-0@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -(
cluster_bits- 8)):
Another compressed cluster may map to the tail of the final sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 593
Quoted 6 lines
docs/interop/qcow2.txt +6
-0+ +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1)Note, that term '
data_space_lenght' is not defined in the spec..Hmm. And it's not trivial to define it correctly, all we have is offset and "number of additional sectors", and may be not all bytes of final sector are occupied... What about something like this:
Quoted 5 lines
In the following layout description byte 0 corresponds to byte at host cluster offset, as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy space starting from this offset, using additional 512-byte sectors defined by "Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes in the final sector. Let x be number of last byte of the layout.
Sounds too complicated for me, may be
x - data chunk length available to store a compressed cluster. (for more details see "Compressed Clusters Descriptor")
Quoted 35 lines
Quoted 10 lines
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content + all the space provided used for compressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data contentMaybe, note that x+! == 4 + length, where length is value of first field of the layout.
Quoted 20 lines
== Snapshots ==
diff --gita/qapi/block-core.jsonb/qapi/block-core.jsonindex873fbef3b5..4b6e576c44100644qapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Vladimir Sementsov-OgievskiyQuoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 5 lines
Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104:
compression_type
105 - 111: padding, must be 0
Quoted 7 lines
Quoted 5 lines
or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
Okay, so requiring an 8-byte field is not necessary. But still, at least mentioning padding bytes (that may be assigned meanings later) is consistent with the rest of the document (for example, we have padding bits documented for the compatible/incompatible/autoclear feature bits), and reminds implementers to keep size rounded to a multiple of 8.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Denis PlotnikovQuoted 78 lines · Denis Plotnikov
Quoted 67 lines · Vladimir Sementsov-Ogievskiy
Quoted 66 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.imgcompression decompression zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++ block/qcow2.c | 7 +++
configure | 29 ++++++++++docs/interop/qcow2.txt| 18 ++++++
qapi/block-core.json | 3 +-5 files changed, 178 insertions(+), 1 deletion(-)block/qcow2-threads.c +0
-0@@ -28,6 +28,11 @@ #define ZLIB_CONST #include <zlib.h>+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +4
-0+#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "
qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"block/qcow2-threads.c +0
-0@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, return ret; }+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +6
-0+ +#define ZSTD_LEN_BUF 4 + +/* +* qcow2_zstd_compress() +* Quoted 8 lines
Quoted 6 lines
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on successThis doesn't match
qcow2_co_compressdefinition. You should return 0 on success.
Quoted 2 lines
does it? I'd rather say it doesn't match to
qcow2_co_compressdescription in the function comment, which we can change actually, becauseqcow2_co_compressis used like:
Oh, yes, you are right. Then we should change the comment.
Quoted 244 lines
block/qcow2.c:static
coroutine_fnint qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
uint64_toffset,uint64_tbytes, QEMUIOVector *qiov,size_tqiov_offset)
{
...
out_buf=g_malloc(s->cluster_size);
out_len=qcow2_co_compress(bs, out_buf, s->cluster_size - 1, buf, s->cluster_size);
if (out_len==-ENOMEM) {
/* could not compress: write normal cluster */
ret =qcow2_co_pwritev_part(bs, offset, bytes, qiov, qiov_offset, 0); if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len< 0) {
ret =-EINVAL;
goto fail;
}
qemu_co_mutex_lock(&s->lock);
ret = qcow2_alloc_compressed_cluster_offset(bs, offset,out_len, <<<<<<<<<<<<&cluster_offset);
...
}Quoted 37 lines
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 12 lines
+ *
-ENOMEMdestination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */block/qcow2-threads.c +8
-0+ +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + + /* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer beginning
Quoted 22 lines
+ * to store compressed chunk size
+ */
+ char*d_buf= ((char *) dest) +ZSTD_LEN_BUF;block/qcow2-threads.c +19
-0+ + /* + * sanity check that we can store the compressed data length, + * and there is some space left for the compressor buffer + */ + if (dest_size <= ZSTD_LEN_BUF) { + return -ENOMEM; + } + + dest_size -= ZSTD_LEN_BUF; + + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5); + + if (ZSTD_isError(ret)) { + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) { + return -ENOMEM; + } else { + return -EIO; + }Quoted 8 lines
Quoted 6 lines
+ }
+
+ /* paraniod sanity check that we can store the commpressed size */ + if (ret >UINT_MAX) {
+ return-ENOMEM;
+ }I'd use
UINT32_MAX, possibly even more paranoid)ok
Quoted 19 lines
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 228
Quoted 7 lines
block/qcow2-threads.c +5
-0+ + /* store the compressed chunk size in the very beginning of the buffer */ + stl_be_p(dest, ret); + + return ret + ZSTD_LEN_BUF;return 0;
Quoted 9 lines
block/qcow2-threads.c +9
-0+} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly +* @dest_size bytes using zstd compression method +* +* @dest - destination buffer, @dest_size bytes Quoted 8 lines
Quoted 6 lines
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *-EIOon any error
+ */
+extra empty line.
will be removed
Quoted 24 lines
Quoted 24 lines
block/qcow2-threads.c +22
-0+static ssize_t qcow2_zstd_decompress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + /* + * zstd decompress wants to know the exact length of the data. + * For that purpose, on compression, the length is stored in + * the very beginning of the compressed buffer + */ + size_t s_size; + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF; + + /* + * sanity check that we can read 4 byte the content length and + * and there is some content to decompress + */ + if (src_size <= ZSTD_LEN_BUF) { + return -EIO; + } + + s_size = ldl_be_p(src); + + /* sanity check that the buffer is big enough to read the content from */Quoted 8 lines
Quoted 6 lines
+ if (
src_size-ZSTD_LEN_BUF<s_size) {
+ return-EIO;
+ }
+
+ if (ZSTD_isError(
+ZSTD_decompress(dest, dest_size, s_buf, s_size))) {hmm, it fit into one line actually
will be fixed if so
Quoted 98 lines
Quoted 98 lines
+ return
-EIO;
+ }block/qcow2-threads.c +5
-0+ + return 0; +} +#endif +static int
qcow2_compress_pool_func(void *opaque){
Qcow2CompressData *data = opaque;block/qcow2-threads.c +0
-0@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_compress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2-threads.c +0
-0@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_decompress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2.c +0
-0@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) switch (s->compression_type) {case
QCOW2_COMPRESSION_TYPE_ZLIB:block/qcow2.c +3
-0+#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break;
default:block/qcow2.c +4
-0@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; +#endif default:
error_setg_errno(errp, -EINVAL, "Unknown compression type"); goto out;configure +0
-0@@ -444,6 +444,7 @@ opengl_dmabuf="no"cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""configure +0
-0@@ -1371,6 +1372,10 @@ for opt do ;;--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"configure +0
-0@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available: (for reading bzip2-compressed dmg images)lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support
coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendconfigure +0
-0@@ -2453,6 +2459,25 @@ EOFfi
fi
+#########################################configure +7
-0+# zstd check + +if test "$zstd" != "no" ; then + cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOFQuoted 8 lines
Quoted 6 lines
+ if
compile_prog"" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ zstd="yes"
+ else
+ if test "$zstd" = "yes"; then
+feature_not_found"zstd" "Install libzstd-devel"to correspond to style used around: s/libzstd-devel/libzstd devel/
Quoted 1 line
My intention was to give a package name for a user encountered this message to copy-paste it to "yum install".
What about apt users? :)
Quoted 2 lines
If this doesn't make things easier, because the package name can be changed in the future (although I doubt) I'll fix it. If the style isn't that important I'd rather leave it as is.
I don't care, keep it as is.
Quoted 58 lines
Quoted 58 lines
Quoted 45 lines
+ fi
+ zstd="no"
+ ficonfigure +2
-0+fi +##########################################
# libseccomp check @@ -6668,6 +6693,7 @@ echo "lzo support $lzo" echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "lzfse support $lzfse" +echo "zstd support $zstd" echo "NUMA host support $numa" echo "libxml2 $libxml2" echo "tcmalloc support $tcmalloc"configure +0
-0@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then"\$ld_exported_symbols_list should have been set to 'yes'." fi
ficonfigure +3
-0+if test "$zstd" = "yes" ; then + echo "CONFIG_ZSTD=y" >> $config_host_mak +fiif test "$tcg_interpreter" = "yes"; then QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
docs/interop/qcow2.txt +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===docs/interop/qcow2.txt +0
-0@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):Another compressed cluster may map to the tail of the final sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout). +
If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 593
Quoted 6 lines
docs/interop/qcow2.txt +6
-0+ +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1)Note, that term '
data_space_lenght' is not defined in the spec..Hmm. And it's not trivial to define it correctly, all we have is offset and "number of additional sectors", and may be not all bytes of final sector are occupied... What about something like this:
Quoted 7 lines
Quoted 5 lines
In the following layout description byte 0 corresponds to byte at host cluster offset, as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy space starting from this offset, using additional 512-byte sectors defined by "Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes in the final sector. Let x be number of last byte of the layout.
Sounds too complicated for me, may be
x - data chunk length available to store a compressed cluster. (for more details see "Compressed Clusters Descriptor")
Ok
Quoted 34 lines
Quoted 33 lines
Quoted 9 lines
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content
+ all the space provided used for compressed data + 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data contentMaybe, note that x+! == 4 + length, where length is value of first field of the layout.
Quoted 19 lines
== Snapshots ==
diff --gita/qapi/block-core.jsonb/qapi/block-core.jsonindex873fbef3b5..4b6e576c44100644qapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2:
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Eric BlakeQuoted 9 lines · Eric Blake
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines
Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104:
compression_type
105 - 111: padding, must be 0or else call out:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
Okay, so requiring an 8-byte field is not necessary. But still, at least mentioning padding bytes (that may be assigned meanings later) is consistent with the rest of the document (for example, we have padding bits documented for the compatible/incompatible/autoclear feature bits), and reminds implementers to keep size rounded to a multiple of 8.
Yes, we can add something about it.. But I'm not sure we need, and I can't imaging correct short wording.
We have section about padding:
"
=== Header padding ===
@header_length must be a multiple of 8, which means that if the end of the last additional field is not aligned, some padding is needed. This padding must be zeroed, so that if some existing (or future) additional field will fall into the padding, it will be interpreted accordingly to point [3.] of the previous paragraph, i.e. in the same manner as when this field is not present. "
So, if we want to add something about 104-111, it should be added to this section, not to previous "=== Additional fields (version 3 and higher) ===".
And, if we want, to add something, we should consider both cases when compression type field exists and when not... What to write?
"105 - 111: These bytes are padding, if header length > 104. May be turned into new additional fields in future."
Sounds a bit strange... Keeping in mind that different versions of qemu may consider the same bytes as additional field or as padding, and it is correct.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines
Quoted 1 line
But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
Okay, so requiring an 8-byte field is not necessary. But still, at least mentioning padding bytes (that may be assigned meanings later) is consistent with the rest of the document (for example, we have padding bits documented for the compatible/incompatible/autoclear feature bits), and reminds implementers to keep size rounded to a multiple of 8.
Yes, we can add something about it.. But I'm not sure we need, and I can't imaging correct short wording.
We have section about padding:
"
=== Header padding ===@header_length must be a multiple of 8, which means that if the end of the last
additional field is not aligned, some padding is needed. This padding must be zeroed, so that if some existing (or future) additional field will fall into the padding, it will be interpreted accordingly to point [3.] of the previous paragraph, i.e. in the same manner as when this field is not present. "
Quoted 7 lines
So, if we want to add something about 104-111, it should be added to this section, not to previous "=== Additional fields (version 3 and higher) ===".
And, if we want, to add something, we should consider both cases when compression type field exists and when not... What to write?
"105 - 111: These bytes are padding, if header length > 104. May be turned into new additional fields in future."
Sounds a bit strange... Keeping in mind that different versions of qemu may consider the same bytes as additional field or as padding, and it is correct.
Looking at the header extension, it can probably be as simple as:
105 - m: Zero padding to round up the header size to the next
multiple of 8.
I guess I'll propose a patch to make my idea concrete.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
In reply to Vladimir Sementsov-OgievskiyQuoted 19 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Denis Plotnikov
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output matching.
Quoted 6 lines
And this and the following patch.
Ideally, patch should not break any iotests. This means that all iotest updates should be merged to the patch which changes their output. Probably, you can split behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch with a lot of iotests updates is better than patch which breaks iotests.
Or we can try to reduce behavior changes in case of zlib:
Quoted 8 lines
- keep header small in case of zlib, not adding zero field
- don't add feature table entry, if compress type is zlib
- don't report compression type on quering, if it is zlib- then, all iotests output will be saved. And, then, if we need, we can change
these theree points one-by-one, updating iotests outputs. But I doubt that we really need it, compatible behavior seems good enough.
I think I would put some efforts in arranging the iotest patches
so they don't break any tests after applying
with the cost of creating a gigantic patch with the compression type implementation and iotest fixes
Denis
Quoted 526 lines
Quoted 524 lines
Quoted 522 lines
Signed-off-by Denis Plotnikov <[email protected]>---
tests/qemu-iotests/049.out | 102
++++++++++++++++++-------------------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 6 +++
tests/qemu-iotests/065 | 20 +++++---tests/qemu-iotests/144.out | 4 +- tests/qemu-iotests/182.out | 2 +- tests/qemu-iotests/242.out | 5 ++ tests/qemu-iotests/255.out | 8 +-- 8 files changed, 82 insertions(+), 66 deletions(-)tests/qemu-iotests/049.out +0
-0@@ -4,90 +4,90 @@ QA output created by 049== 1. Traditional size parameter ==
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 2. Specifying size via -o ==
qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2 -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == 3. Invalid sizes ==tests/qemu-iotests/049.out +0
-0@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image sizemust be specified only once
== Check correct interpretation of suffixes for cluster size ==
qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1048576 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=524288 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check compat level option ==
qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42
cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42
cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check preallocation option ==
qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check encryption option ==
qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib == Checklazy_refcountsoption (only with v3) ==
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater) -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
*** donetests/qemu-iotests/060.out +0
-0@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: truetests/qemu-iotests/061.out +0
-0@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: footests/qemu-iotests/061.out +0
-0@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file raw: falsetests/qemu-iotests/061.out +0
-0@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/061.out +0
-0@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.datatests/qemu-iotests/065 +0
-0@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
img_options= 'compat=0.10'
-json_compare= { 'compat': '0.10', 'refcount-bits': 16 } -human_compare= [ 'compat: 0.10', 'refcount bits: 16' ] +json_compare= { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
+human_compare= [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''img_options= 'compat=1.1,lazy_refcounts=off'json_compare= { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } -human_compare= [ 'compat: 1.1', 'lazy refcounts: false', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' }
+human_compare= [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
'refcount bits: 16', 'corrupt: false' ] class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''img_options= 'compat=1.1,lazy_refcounts=on'json_compare= { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } -human_compare= [ 'compat: 1.1', 'lazy refcounts: true', + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' }
+human_compare= [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
'refcount bits: 16', 'corrupt: false' ] class TestQCow3NotLazyQMP(TestQMP):tests/qemu-iotests/065 +0
-0@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
img_options= 'compat=1.1,lazy_refcounts=off'qemu_options= 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } class TestQCow3LazyQMP(TestQMP):tests/qemu-iotests/065 +0
-0@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
img_options= 'compat=1.1,lazy_refcounts=on'qemu_options= 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True, - 'refcount-bits': 16, 'corrupt': False } + 'refcount-bits': 16, 'corrupt': False, + 'compression-type': 'zlib' } TestImageInfoSpecific = None
TestQemuImgInfo = Nonetests/qemu-iotests/144.out +0
-0@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMTsize=536870912
{ 'execute': 'qmp_capabilities' }
{"return": {}}
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912
backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912
backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}}
=== Performing block-commit on active layer ===tests/qemu-iotests/144.out +0
-0@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2size=536870912 backing_file=TEST_DIR/
=== Performing Live Snapshot 2 ===
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } } -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912
backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912
backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}}
*** donetests/qemu-iotests/182.out +0
-0@@ -13,7 +13,7 @@ Is another process using the image[TEST_DIR/t.qcow2]?
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}
{'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } } -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120
backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120
backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"return": {}}
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } } {"return": {}}tests/qemu-iotests/242.out +0
-0@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: falsetests/qemu-iotests/242.out +0
-0@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/242.out +0
-0@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:tests/qemu-iotests/255.out +0
-0@@ -3,9 +3,9 @@ Finishing a commit job with background reads=== Create backing chain and start VM ===
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib === Start background read requests ===tests/qemu-iotests/255.out +0
-0@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled=== Create images and start VM ===
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib wrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
In reply to Vladimir Sementsov-OgievskiyQuoted 71 lines · Vladimir Sementsov-Ogievskiy
Quoted 55 lines · Denis Plotnikov
The test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.
Signed-off-by Denis Plotnikov <[email protected]>---
tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/287.out | 41 +++++++++++++ tests/qemu-iotests/group | 1 +3 files changed, 165 insertions(+)create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.outtests/qemu-iotests/287 new file +37
-0@@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Test case for an image using zstd compression +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# standard environment +. ./common.rc +. ./common.filtertests/qemu-iotests/287 line 47
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+ +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"` + +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null +RES=$?Hmm. This will not work for other architectures and for out of tree builds. Also, it checks config but not current binary (they may be out of sync, or even unrelated).
Quoted 2 lines
Probably better try to create image with zstd compression type and handle expected error.
What if the error is "unable to create an image with zstd", although it has to be?
I think the best way is to ask qemu binary whether it supports zstd, but it doesn't available by now (should be?)
Is there any other way to make sure that the std compression test has to be executed?
Quoted 28 lines
Quoted 19 lines
tests/qemu-iotests/287 new file +7
-0+if (($RES)); then + _notrun "ZSTD is disabled in the current configuration" +fi + +# Test: when compression is zlib the incompatible is unset +echo +echo "=== Testing compression type incompatible bit setting for zlib==="
tests/qemu-iotests/287 new file +8
-0+echo + +_make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: when compression differs from zlib the incompatible bit is set +echo +echo "=== Testing compression type incompatible bit setting for zstd==="
tests/qemu-iotests/287 line 68
Quoted 6 lines
tests/qemu-iotests/287 new file +6
-0+echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: an image can't be openned if compression type is zlib andopened
Quoted 16 lines
Quoted 12 lines
tests/qemu-iotests/287 new file +12
-0+# incompatible feature compression type is set +echo +echo "=== Testing zlib with incompatible bit set ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3 +# to make sure the bit was actually set +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must notQuoted 2 lines
be openned"
+fimay be better to instead keep error output and just check it..
I add the explicit message to reduce the investigating time of what happened and what should it be. If it isn't that important I'd rather leave it as is.
Quoted 34 lines
Quoted 34 lines
tests/qemu-iotests/287 new file +14
-0+ +# Test: an image can't be openned if compression type is NOT zlib and +# incompatible feature compression type is UNSET +echo +echo "=== Testing zstd with incompatible bit unset ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0 +# to make sure the bit was actually unset +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must notbe openned"
tests/qemu-iotests/287 new file +18
-0+fi +# Test: check compression type values +echo +echo "=== Testing compression type values ===" +echo +# zlib=0 +IMGOPTS='compression_type=zlib' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# zstd=1 +IMGOPTS='compression_type=zstd' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo +
Quoted 8 lines
tests/qemu-iotests/287 line 118
Quoted 5 lines
tests/qemu-iotests/287 new file +5
-0+CLUSTER_SIZE=65536 +IMGOPTS='compression_type=zstd' _make_test_img 64M +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_ioHmm output depends on default pattern. Better use "write -c -P
0x110 64k" (or any pattern you want), to make it explicit.
Good suggestion, I'll change that
Quoted 69 lines
Quoted 67 lines
tests/qemu-iotests/287 new file +5
-0+ +# success, all done +echo "*** done" +rm -f $seq.full +status=0tests/qemu-iotests/287.out new file +10
-0@@ -0,0 +1,41 @@ +QA output created by 287 + +=== Testing compression type incompatible bit setting for zlib === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [] + +=== Testing compression type incompatible bit setting for zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864compression_type=zstd
tests/qemu-iotests/287.out new file +10
-0+incompatible_features [3] + +=== Testing zlib with incompatible bit set === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [3] + +=== Testing zstd with incompatible bit unset === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864compression_type=zstd
tests/qemu-iotests/287.out new file +7
-0+incompatible_features [] + +=== Testing compression type values === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 + 0 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864compression_type=zstd
+ 1tests/qemu-iotests/287.out new file +4
-0+ +=== Testing reading and writing with zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864compression_type=zstd
tests/qemu-iotests/287.out new file +9
-0+wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +00000000: cd cd cd cd cd cd cd cd cd cd .......... +read 10/10 bytes at offset 0 +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +0000fffa: cd cd cd cd cd cd 00 00 ........ +read 8/8 bytes at offset 65530 +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** donetests/qemu-iotests/group +0
-0@@ -293,3 +293,4 @@283 auto quick
284 rw
286 rw quick
+287 auto quick
Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
Quoted 74 lines · Denis Plotnikov
Quoted 71 lines · Vladimir Sementsov-Ogievskiy
Quoted 55 lines · Denis Plotnikov
The test checks fulfilling qcow2 requiriements for the compression type feature and zstd compression type operability.
Signed-off-by Denis Plotnikov <[email protected]>---
tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/287.out | 41 +++++++++++++ tests/qemu-iotests/group | 1 +3 files changed, 165 insertions(+)create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.outtests/qemu-iotests/287 new file +37
-0@@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# +# Test case for an image using zstd compression +# +# Copyright (c) 2020 Virtuozzo International GmbH +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# standard environment +. ./common.rc +. ./common.filtertests/qemu-iotests/287 line 47
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+ +# This tests qocw2-specific low-level functionality +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"` + +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null +RES=$?Hmm. This will not work for other architectures and for out of tree builds. Also, it checks config but not current binary (they may be out of sync, or even unrelated).
Quoted 3 lines
Quoted 2 lines
Probably better try to create image with zstd compression type and handle expected error.
What if the error is "unable to create an image with zstd", although it has to be?
With such message test should fail. I expect something like "unknown option" error in case of not built in zstd, and I think it's OK to check such message and skip the test.
Quoted 1 line
I think the best way is to ask qemu binary whether it supports zstd, but it doesn't available by now (should be?)
And in this case the problem is the same: what if it will say that it doesn't support it, although it should support? We can't check such things anyway.
Quoted 43 lines
Is there any other way to make sure that the std compression test has to be executed?
Quoted 26 lines
Quoted 13 lines
tests/qemu-iotests/287 new file +11
-0+if (($RES)); then + _notrun "ZSTD is disabled in the current configuration" +fi + +# Test: when compression is zlib the incompatible is unset +echo +echo "=== Testing compression type incompatible bit setting for zlib ===" +echo + +_make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_featurestests/qemu-iotests/287 line 68
Quoted 10 lines
tests/qemu-iotests/287 new file +10
-0+ +# Test: when compression differs from zlib the incompatible bit is set +echo +echo "=== Testing compression type incompatible bit setting for zstd ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features + +# Test: an image can't be openned if compression type is zlib andopened
Quoted 15 lines
tests/qemu-iotests/287 line 81
Quoted 13 lines
tests/qemu-iotests/287 new file +13
-0+# incompatible feature compression type is set +echo +echo "=== Testing zlib with incompatible bit set ===" +echo + +IMGOPTS='compression_type=zlib' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3 +# to make sure the bit was actually set +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fimay be better to instead keep error output and just check it..
Quoted 2 lines
I add the explicit message to reduce the investigating time of what happened and what should it be. If it isn't that important I'd rather leave it as is.
of course, up to you
Quoted 108 lines
Quoted 33 lines
Quoted 33 lines
tests/qemu-iotests/287 new file +32
-0+ +# Test: an image can't be openned if compression type is NOT zlib and +# incompatible feature compression type is UNSET +echo +echo "=== Testing zstd with incompatible bit unset ===" +echo + +IMGOPTS='compression_type=zstd' _make_test_img 64M +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0 +# to make sure the bit was actually unset +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null +if (($?==0)); then + echo "Error: The image openned successfully. The image must not be openned" +fi +# Test: check compression type values +echo +echo "=== Testing compression type values ===" +echo +# zlib=0 +IMGOPTS='compression_type=zlib' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# zstd=1 +IMGOPTS='compression_type=zstd' _make_test_img 64M +od -j104 -N1 -An -vtu1 "$TEST_IMG" + +# Test: using zstd compression, write to and read from an image +echo +echo "=== Testing reading and writing with zstd ===" +echo +Quoted 8 lines
tests/qemu-iotests/287 line 118
Quoted 5 lines
tests/qemu-iotests/287 new file +5
-0+CLUSTER_SIZE=65536 +IMGOPTS='compression_type=zstd' _make_test_img 64M +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_ioHmm output depends on default pattern. Better use "write -c -P
0x110 64k" (or any pattern you want), to make it explicit.Good suggestion, I'll change that
Quoted 65 lines
Quoted 63 lines
tests/qemu-iotests/287 new file +5
-0+ +# success, all done +echo "*** done" +rm -f $seq.full +status=0tests/qemu-iotests/287.out new file +41
-0@@ -0,0 +1,41 @@ +QA output created by 287 + +=== Testing compression type incompatible bit setting for zlib === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [] + +=== Testing compression type incompatible bit setting for zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +incompatible_features [3] + +=== Testing zlib with incompatible bit set === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 +incompatible_features [3] + +=== Testing zstd with incompatible bit unset === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +incompatible_features [] + +=== Testing compression type values === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 + 0 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd + 1 + +=== Testing reading and writing with zstd === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd +wrote 65536/65536 bytes at offset 0 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +00000000: cd cd cd cd cd cd cd cd cd cd .......... +read 10/10 bytes at offset 0 +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +0000fffa: cd cd cd cd cd cd 00 00 ........ +read 8/8 bytes at offset 65530 +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** donetests/qemu-iotests/group +0
-0@@ -293,3 +293,4 @@283 auto quick
284 rw
286 rw quick
+287 auto quick
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v1 1/8] qcow2: introduce compression type feature
In reply to Eric BlakeQuoted 59 lines · Eric Blake
Quoted 2 lines · Denis Plotnikov
The patch adds some preparation parts for incompatible compression type feature to Qcow2 that indicates which allow to use different compression
to qcow2, allowing the use of different
Quoted 31 lines
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and can be changed only later by image conversion, thus compression type defines the only compression algorithm used for the image, and thus, for all image clusters.
The goal of the feature is to add support of other compression methods to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type are backward compatible with older qemu versions.
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++ block/qcow2.h | 31 ++++++++---include/block/block_int.h| 1 +
qapi/block-core.json | 22 +++++++-4 files changed, 150 insertions(+), 9 deletions(-)block/qcow2.c +1
-0@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options, return ret; } +static int validate_compression_type(BDRVQcow2State *s, Error **errp) +{block/qcow2.c static int · line 1256
Quoted 10 lines
block/qcow2.c +10
-0+ /* + * Sanity check + * according to qcow2 spec, the compression type is 1-byte field + * but in BDRVQcow2State the compression_type is enum sizeof(int) + * so, the max compression_type value is 255. + */ + if (s->compression_type > 0xff) { + error_setg(errp, "qcow2: compression type value is too big"); + return -EINVAL; + }Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104:
compression_type
105 - 111: padding, must be 0or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
Quoted 12 lines
block/qcow2.c static int qcow2_update_options(BlockDriverState *bs, QDict *options, · line 1266
Quoted 10 lines
block/qcow2.c +10
-0+ + switch (s->compression_type) { + case QCOW2_COMPRESSION_TYPE_ZLIB: + break; + + default: + error_setg(errp, "qcow2: unknown compression type: %u", + s->compression_type); + return -ENOTSUP; + }Having two checks feels redundant, compared to just letting the default catch all unrecognized values in that field.
Looks like it is.
Quoted 76 lines
Quoted 15 lines
block/qcow2.c +8
-0+ + /* + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB + * the incompatible feature flag must be set + */ + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) { + if (s->incompatible_features &QCOW2_INCOMPAT_COMPRESSION_TYPE ) { + error_setg(errp, "qcow2: Compression type incompatiblefeature "
+ "bit must not be set"); + return-EINVAL;
+ }
+ } else {Quoted 8 lines
+ if (!(s->incompatible_features &
QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must be set");
+ return-EINVAL;
+ }
+ }Matches what we documented in the spec.
Quoted 16 lines
block/qcow2.c +4
-0+ + return 0; +} +/* Called with
s->lockheld. */
static intcoroutine_fnqcow2_do_open(BlockDriverState *bs, QDict *options, int flags, Error **errp)block/qcow2.c +0
-0@@ -1357,6 +1401,26 @@ static int coroutine_fnqcow2_do_open (BlockDriverState *bs, QDict *options, s->compatible_features = header.compatible_features; s->autoclear_features = header.autoclear_features;+ /*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header. + * Distinguish them by the header length and use
+ * the only valid (default) compression type in that caseQuoted 8 lines
+ */
+ if (header.header_length >offsetof(QCowHeader, compression_type)) {
+ /*
+ * don't deal with endians sincecompression_typeis 1 byte long
+ */
+s->compression_type= header.compression_type;Changes if you go with my suggestion of just making the
compression_typefield occupy 8 bytes in the qcow2 header. (And if you want to keep it 1 byte, I still think the spec should call out explicit padding bytes).Quoted 23 lines
+ } else {
+s->compression_type=QCOW2_COMPRESSION_TYPE_ZLIB; + }block/qcow2.c +6
-0+ + ret = validate_compression_type(s, errp); + if (ret) { + goto fail; + } +if (
s->incompatible_features& ~QCOW2_INCOMPAT_MASK) { void*feature_table= NULL; qcow2_read_extensions(bs, header.header_length,ext_end,block/qcow2.c +5
-0@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs) total_size = bs->total_sectors * BDRV_SECTOR_SIZE; refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3); + ret = validate_compression_type(s, NULL); + + if (ret) { + goto fail; + } + *header = (QCowHeader) { /* Version 2 fields */
Quoted 8 lines
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2815
Quoted 6 lines
block/qcow2.c +1
-0.magic = cpu_to_be32(QCOW_MAGIC),@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs) .autoclear_features = cpu_to_be64(s->autoclear_features), .refcount_order = cpu_to_be32(s->refcount_order), .header_length = cpu_to_be32(header_length), + .compression_type = (uint8_t) s->compression_type,Is the cast necessary?
s->compression_type is enum, I thought it would be good to explicitly show that in the code. If it's totally useless I'll remove it.
Quoted 12 lines
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2914
Quoted 9 lines
block/qcow2.c +2
-0}; /* For older versions, write a shorter header */@@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs) .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR, .name = "lazy refcounts", }, + { + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,Ordering: please group all the incompatible bits side-by-side (this should come before the lazy refcount bit).
Quoted 4 lines
block/qcow2.c int qcow2_update_header(BlockDriverState *bs) · line 2916
Quoted 2 lines
block/qcow2.c +2
-0+ .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR, + .name = "compression type",This change breaks iotests; at least 31, 36, and 61 need updates (I've got a similar patch pending which fixes the fact that we forgot the autoclear bit [1]). You'll need to squash in fixes for those at the same time.
ok
Quoted 30 lines
[1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
block/qcow2.c qcow2_co_create(BlockdevCreateOptions · line 3482
Quoted 10 lines
block/qcow2.c +2
-0+ }, }; ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) .refcount_table_offset = cpu_to_be64(cluster_size), .refcount_table_clusters = cpu_to_be32(1), .refcount_order = cpu_to_be32(refcount_order), + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,Is the cast necessary?
Quoted 14 lines
block/qcow2.c +0
-0.header_length = cpu_to_be32(sizeof(*header)),};
@@ -3420,6 +3497,26 @@qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
}
+ if (qcow2_opts->has_compression_type&&
+qcow2_opts->compression_type!=QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+ if (qcow2_opts->compression_type>0xff) {
+ error_setg_errno(errp,-EINVAL, "Too big compression
type value");
+ goto out;
+ }
+
Quoted 8 lines
Quoted 6 lines
+ switch (
qcow2_opts->compression_type) {
+ default:
+error_setg_errno(errp, -EINVAL, "Unknown compression type");
+ goto out;
+ }This should probably be an assert that
qcow2_opts->compression_typeis in range, rather than a switch statement and error_setg. Callers ofqcow2_co_createshould not be handing us unknown values.
The intention was to express what has actually happened in the case of a "bad" image to easy the problem investigation.
Quoted 32 lines
Quoted 4 lines
+
+header->compression_type= (uint8_t)qcow2_opts->compression_type;Why the cast?
Quoted 13 lines
block/qcow2.c +5
-0+ + header->incompatible_features |= + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE); + } +ret =
blk_pwrite(blk, 0, header, cluster_size, 0);g_free(header);
if (ret < 0) {block/qcow2.c +0
-0@@ -3602,6 +3699,7 @@ static int coroutine_fnqcow2_co_create_opts(const char *filename, QemuOpts *opt
+++
b/block/qcow2.hblock/qcow2.c typedef struct QCowHeader { · line 154
Quoted 9 lines
block/qcow2.c +6
-0@@ -146,6 +146,12 @@ typedef struct QCowHeader { uint32_t refcount_order; uint32_t header_length; + + /* Additional fields */ + uint8_t compression_type; + + /* header must be a multiple of 8 */ + uint8_t padding[7]; } QEMU_PACKED QCowHeader;You're changing the size of this struct, which WILL break iotests (and even more than just the 3 I pointed out above for the feature name table).
Quoted 3 lines
/me looks ahead
Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to be squashed in here, where the change is made.
ok, but the patch will be pretty long
Quoted 28 lines
Quoted 15 lines
block/qcow2.c +2
-6typedef struct QEMU_PACKED QCowSnapshotHeader {@@ -213,16 +219,20 @@ enum { /* Incompatible feature bits */ enum { - QCOW2_INCOMPAT_DIRTY_BITNR = 0, - QCOW2_INCOMPAT_CORRUPT_BITNR = 1, - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2, - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR, - QCOW2_INCOMPAT_CORRUPT = 1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , - QCOW2_INCOMPAT_DATA_FILE = 1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , + QCOW2_INCOMPAT_DIRTY_BITNR = 0, + QCOW2_INCOMPAT_CORRUPT_BITNR = 1,block/qcow2.c enum { · line 234
Quoted 10 lines
block/qcow2.c +7
-0+ QCOW2_INCOMPAT_DATA_FILE_BITNR = 2, + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3, + QCOW2_INCOMPAT_DIRTY = 1 <<QCOW2_INCOMPAT_DIRTY_BITNR , + QCOW2_INCOMPAT_CORRUPT = 1 <<QCOW2_INCOMPAT_CORRUPT_BITNR , + QCOW2_INCOMPAT_DATA_FILE = 1 <<QCOW2_INCOMPAT_DATA_FILE_BITNR , + QCOW2_INCOMPAT_COMPRESSION_TYPE = + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,Uggh. I hate realigning = just because we added a longer name, especially when you then can't even keep things on one line because of length. If it were me, I'd leave the existing lines alone, and/or switch everything to just use 'BITNR = ' rather than trying to align =.
Quoted 1 line
Bikeshedding - since the new name is so long, can you get by with the shorter
QCOW2_INCOMPAT_COMPRESSION_BITNR(drop the_TYPE)?
Good idea, I'll certainly redo that part, because I don't like ether.
Denis
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Eric BlakeQuoted 46 lines · Eric Blake
Quoted 7 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 7 lines
block/qcow2-threads.c +7
-0+static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + + /* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer beginning
Quoted 15 lines
+ * to store compressed chunk size
+ */
+ char*d_buf= ((char *) dest) +ZSTD_LEN_BUF;block/qcow2-threads.c +12
-0+ + /* + * sanity check that we can store the compressed data length, + * and there is some space left for the compressor buffer + */ + if (dest_size <= ZSTD_LEN_BUF) { + return -ENOMEM; + } + + dest_size -= ZSTD_LEN_BUF; + + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 220
Quoted 10 lines
block/qcow2-threads.c +10
-0+ + if (ZSTD_isError(ret)) { + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) { + return -ENOMEM; + } else { + return -EIO; + } + } + + /* paraniod sanity check that we can store the commpressed size */paranoid, compressed
Quoted 5 lines
Quoted 3 lines
+ if (ret >
UINT_MAX) {
+ return-ENOMEM;
+ }This is pointless. Better is to ensure that we actually compressed data (the pigeonhole principle states that there are some inputs that MUST result in inflation, in order for most other inputs to result in compression). But that check was satisfied by checking for
ZSTD_error_dstSize_tooSmall, which is what happens for one of those uncompressible inputs. Namely, zstd will never return a result larger thandest_size, and sincedest_sizeis smaller thanUINT_MAXon entry, this check is pointless. But if you want something, I'd be okay with:assert(ret <= dest_size).
Taking into account that this is "just in case" and I'm trying to protect the first 4 bytes of the buffer from the overflow and
I can't imagine the situation when we deal with cluster sizes greater than UINT32_MAX but the input size is size_t which can be > UINT32_MAX on 64bit archs. I'd rather stick with
if (ret > UINT32_MAX) {
return -ENOMEM;
}
as Vladimir suggested.
I'm not sure that the assert is good here, since it stops the system operating and this isn't potential error but a limitation Does it work for you?
Denis
Quoted 54 lines
Quoted 41 lines
+++
b/docs/interop/qcow2.txt(quoted diff) +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===(quoted diff) +0
-0@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -(
cluster_bits- 8)):
Another compressed cluster may map to the tail of the final sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression + type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===docs/interop/qcow2.txt +10
-0+ +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1) + + 0: (default) zlib <http://zlib.net/>: + Byte 0 - x: the compressed data content + all the space provided used forcompressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data content == Snapshots ==
diff --gita/qapi/block-core.jsonb/qapi/block-core.jsonindex873fbef3b5..4b6e576c44100644qapi/block-core.json +0
-0@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image fileqapi/block-core.json line 4410
Quoted 10 lines
qapi/block-core.json +2
-1# # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }The spec and UI changes are okay.
Re: [PATCH v1 4/8] iotests: filter out compression_type
In reply to Eric BlakeQuoted 19 lines · Eric Blake
Quoted 2 lines · Denis Plotnikov
After adding compression type feature to qcow2 format, qemu framework commands reporting the image settingd, e.g. "qemu-img create", started
settings
Quoted 9 lines
reporting the compression type for the image which breaks the iotests output matching.
To fix it, add compression_type=zlib to the list of filtered image parameters.
Signed-off-by Denis Plotnikov <[email protected]>---
tests/qemu-iotests/common.filter | 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)This should be squashed in to the patch that caused the breakage (3/8, if I'm right).
Quoted 1 line
Quoted 13 lines
tests/qemu-iotests/common.filter _filter_img_create() · line 156
Quoted 11 lines
tests/qemu-iotests/common.filter +2
-1@@ -152,7 +152,8 @@ _filter_img_create() -e "s# refcount_bits=[0-9]\\+##g" \ -e "s# key-secret=[a-zA-Z0-9]\\+##g" \ -e "s# iter-time=[0-9]\\+##g" \ - -e "s# force_size=\\(on\\|off\\)##g" + -e "s# force_size=\\(on\\|off\\)##g" \ + -e "s# compression_type=zlib##g"Do you really want to hard-code just zlib, or should this be more generic as compression_type=[a-zA-Z0-9]\\+ as done on other lines like key-secret?
When I did this I meant additional implicit check that the default compression type is zlib. But non of the other items in the filter don't do it. So I'll change it to be consistent. Thanks!
Denis
Quoted 2 lines
Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
In reply to Vladimir Sementsov-OgievskiyQuoted 81 lines · Vladimir Sementsov-Ogievskiy
Quoted 80 lines · Denis Plotnikov
Quoted 77 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines · Denis Plotnikov
zstd significantly reduces cluster compression time. It provides better compression performance maintaining the same level of the compression ratio in comparison with zlib, which, at the moment, is the only compression method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2GThe test was conducted with brd disk to reduce the influence of disk subsystem to the test results.
The results is given in seconds.compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd] src.img [zlib|zstd]_compressed.img decompress cmd
time ./qemu-img convert -O qcow2 [zlib|zstd]_compressed.img uncompressed.imgcompression decompression zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %) user 65.0 15.8 5.3 2.5 sys 3.3 0.2 2.0 2.0Both ZLIB and ZSTD gave the same compression ratio: 1.57 compressed image size in both cases: 1.4G
Signed-off-by Denis Plotnikov <[email protected]>---
block/qcow2-threads.c | 122
+++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 7 +++
configure | 29 ++++++++++
docs/interop/qcow2.txt| 18 ++++++
qapi/block-core.json | 3 +-5 files changed, 178 insertions(+), 1 deletion(-)block/qcow2-threads.c +0
-0@@ -28,6 +28,11 @@ #define ZLIB_CONST #include <zlib.h>+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +4
-0+#include <zstd.h> +#include <zstd_errors.h> +#endif +#include "
qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"block/qcow2-threads.c +0
-0@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, return ret; }+#ifdef
CONFIG_ZSTDblock/qcow2-threads.c +2
-0+ +#define ZSTD_LEN_BUF 4block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 184
Quoted 10 lines
block/qcow2-threads.c +10
-0+ +/* +* qcow2_zstd_compress() +* +* Compress @src_size bytes of data using zstd compression method +* +* @dest - destination buffer, @dest_size bytes +* @src - source buffer, @src_size bytes +* +* Returns: compressed size on success This doesn't match
qcow2_co_compressdefinition. You should return 0 on success.
Quoted 149 lines
Quoted 2 lines
does it? I'd rather say it doesn't match to
qcow2_co_compressdescription in the function comment, which we can change actually, becauseqcow2_co_compressis used like:Oh, yes, you are right. Then we should change the comment.
Quoted 144 lines
block/qcow2.c:static
coroutine_fnintqcow2_co_pwritev_compressed_task(BlockDriverState *bs, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, size_t qiov_offset){
...
out_buf=g_malloc(s->cluster_size);
out_len=qcow2_co_compress(bs, out_buf, s->cluster_size - 1, buf, s->cluster_size);
if (out_len==-ENOMEM) {
/* could not compress: write normal cluster */
ret =qcow2_co_pwritev_part(bs, offset, bytes, qiov, qiov_offset, 0); if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len< 0) {
ret =-EINVAL;
goto fail;
}
qemu_co_mutex_lock(&s->lock);
ret =qcow2_alloc_compressed_cluster_offset(bs, offset, out_len, <<<<<<<<<<<< &cluster_offset);
...
}Quoted 39 lines
block/qcow2-threads.c static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size, · line 195
Quoted 13 lines
+ *
-ENOMEMdestination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */block/qcow2-threads.c +8
-0+ +static ssize_t qcow2_zstd_compress (void *dest,size_t dest_size, + const void *src, size_t src_size) +{ + size_t ret; + + /* +* steal ZSTD_LEN_BUF bytes in the very beginng of the buffer beginning
Quoted 23 lines
+ * to store compressed chunk size
+ */
+ char*d_buf= ((char *) dest) +ZSTD_LEN_BUF;block/qcow2-threads.c +20
-0+ + /* + * sanity check that we can store the compressed data length, + * and there is some space left for the compressor buffer + */ + if (dest_size <= ZSTD_LEN_BUF) { + return -ENOMEM; + } + + dest_size -= ZSTD_LEN_BUF; + + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5); + + if (ZSTD_isError(ret)) { + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) { + return -ENOMEM; + } else { + return -EIO; + } + }Quoted 8 lines
Quoted 6 lines
+
+ /* paraniod sanity check that we can store the commpressed size */
+ if (ret >UINT_MAX) {
+ return-ENOMEM;
+ }I'd use
UINT32_MAX, possibly even more paranoid)ok
Quoted 21 lines
Quoted 8 lines
+
+ /* store the compressed chunk size in the very beginning of the buffer */
+stl_be_p(dest, ret);
+
+ return ret +ZSTD_LEN_BUF;return 0;
Quoted 10 lines
block/qcow2-threads.c +6
-0+} + +/* +* qcow2_zstd_decompress() +* +* Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytesQuoted 8 lines
Quoted 6 lines
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ *-EIOon any error
+ */
+extra empty line.
will be removed
Quoted 26 lines
Quoted 26 lines
+static
ssize_tqcow2_zstd_decompress(void *dest,size_tdest_size, + const void *src,size_tsrc_size)block/qcow2-threads.c +20
-0+{ + /* + * zstd decompress wants to know the exact length of the data. + * For that purpose, on compression, the length is stored in + * the very beginning of the compressed buffer + */ + size_t s_size; + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF; + + /* + * sanity check that we can read 4 byte the content length and + * and there is some content to decompress + */ + if (src_size <= ZSTD_LEN_BUF) { + return -EIO; + } + + s_size = ldl_be_p(src); + + /* sanity check that the buffer is big enough to read thecontent from */
Quoted 8 lines
Quoted 6 lines
+ if (
src_size-ZSTD_LEN_BUF<s_size) {
+ return-EIO;
+ }
+
+ if (ZSTD_isError(
+ZSTD_decompress(dest, dest_size, s_buf, s_size))) {hmm, it fit into one line actually
will be fixed if so
If there is no other objections I'd leave it as is since
if (ZSTD_isError(ZSTD_decompress(dest, dest_size, s_buf, s_size))) { looks too messy to me although it fits into 80 chars
Quoted 228 lines
Quoted 111 lines
Quoted 103 lines
Quoted 103 lines
+ return
-EIO;
+ }block/qcow2-threads.c +5
-0+ + return 0; +} +#endif +static int
qcow2_compress_pool_func(void *opaque){
Qcow2CompressData *data = opaque;block/qcow2-threads.c +0
-0@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_compress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_compress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2-threads.c +0
-0@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size, fn = qcow2_zlib_decompress; break;+#ifdef
CONFIG_ZSTD
+ caseQCOW2_COMPRESSION_TYPE_ZSTD:
+ fn =qcow2_zstd_decompress;
+ break;
+#endif
default:
return-ENOTSUP;
}block/qcow2.c +0
-0@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp) switch (s->compression_type) {case
QCOW2_COMPRESSION_TYPE_ZLIB:block/qcow2.c +3
-0+#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: +#endif break;
default:block/qcow2.c +4
-0@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp) } switch (qcow2_opts->compression_type) { +#ifdef CONFIG_ZSTD + case QCOW2_COMPRESSION_TYPE_ZSTD: + break; +#endif default:
error_setg_errno(errp, -EINVAL, "Unknown compression type"); goto out;configure +0
-0@@ -444,6 +444,7 @@ opengl_dmabuf="no"cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""configure +0
-0@@ -1371,6 +1372,10 @@ for opt do ;;--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"configure +0
-0@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default isenabled if available:
(for reading bzip2-compressed dmg images) lzfse support of lzfse compression library (for reading lzfse-compressed dmg images) + zstd support of zstd compression library seccomp seccomp support
coroutine-pool coroutine freelist (better performance) glusterfs GlusterFS backendconfigure +0
-0@@ -2453,6 +2459,25 @@ EOFfi
fi
+#########################################configure +7
-0+# zstd check + +if test "$zstd" != "no" ; then + cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOFQuoted 8 lines
Quoted 6 lines
+ if
compile_prog"" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ zstd="yes"
+ else
+ if test "$zstd" = "yes"; then
+feature_not_found"zstd" "Install libzstd-devel"to correspond to style used around: s/libzstd-devel/libzstd devel/
Quoted 1 line
My intention was to give a package name for a user encountered this message to copy-paste it to "yum install".
What about apt users? :)
Quoted 2 lines
If this doesn't make things easier, because the package name can be changed in the future (although I doubt) I'll fix it. If the style isn't that important I'd rather leave it as is.
I don't care, keep it as is.
Quoted 60 lines
Quoted 60 lines
Quoted 47 lines
+ fi
+ zstd="no"
+ ficonfigure +2
-0+fi +##########################################
# libseccomp check @@ -6668,6 +6693,7 @@ echo "lzo support $lzo" echo "snappy support $snappy" echo "bzip2 support $bzip2" echo "lzfse support $lzfse" +echo "zstd support $zstd" echo "NUMA host support $numa" echo "libxml2 $libxml2" echo "tcmalloc support $tcmalloc"configure +0
-0@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then"\$ld_exported_symbols_list should have been set to 'yes'." fi
ficonfigure +3
-0+if test "$zstd" = "yes" ; then + echo "CONFIG_ZSTD=y" >> $config_host_mak +fiif test "$tcg_interpreter" = "yes"; then QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
docs/interop/qcow2.txt +0
-0@@ -208,6 +208,7 @@ version 2.Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===docs/interop/qcow2.txt +0
-0@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -(
cluster_bits- 8)):
Another compressed cluster may map to the tail of the final sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression + type used for the image (see compressed
cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing file (except if bit 0 in the Standard Cluster Descriptor is set). If there is no backing file or the backing file is smaller than the image, they shall read zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===docs/interop/qcow2.txt Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)): · line 593
Quoted 6 lines
docs/interop/qcow2.txt +6
-0+ +The compressed cluster data has a layout depending on the compression +type used for the image, as follows: + +Compressed data layout for the available compression types: +(x = data_space_length - 1)Note, that term '
data_space_lenght' is not defined in the spec..Hmm. And it's not trivial to define it correctly, all we have is offset and "number of additional sectors", and may be not all bytes of final sector are occupied... What about something like this:
Quoted 7 lines
Quoted 5 lines
In the following layout description byte 0 corresponds to byte at host cluster offset, as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy space starting from this offset, using additional 512-byte sectors defined by "Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes in the final sector. Let x be number of last byte of the layout.
Sounds too complicated for me, may be
x - data chunk length available to store a compressed cluster. (for more details see "Compressed Clusters Descriptor")Ok
Quoted 36 lines
Quoted 35 lines
Quoted 10 lines
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content + all the space provided used for compressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes + 4 - x: the compressed data contentMaybe, note that x+! == 4 + length, where length is value of first field of the layout.
Quoted 20 lines
== Snapshots ==
diff --gita/qapi/block-core.jsonb/qapi/block-core.jsonindex873fbef3b5..4b6e576c44100644qapi/block-core.json +2
-1@@ -4401,11 +4401,12 @@ # Compression type used in qcow2 image file # # @zlib: zlib compression, see <http://zlib.net/> +# @zstd: zstd compression, see <http://github.com/facebook/zstd> # # Since: 5.0 ## { 'enum': 'Qcow2CompressionType', - 'data': [ 'zlib' ] } + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] } ## # @BlockdevCreateOptionsQcow2: