migration: add sztd compression
cover migration: add sztd compression
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmark
Threads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------
compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31
time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------
compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%
Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compression
configure | 26 ++++ hmp.c | 8 ++ migration/migration.c | 45 ++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++---- migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++++++++++++++++++++++++++++++++++-------- qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 1 line
2.17.0
PATCH 1/3 migration: rework compression code for adding more data compressors
Also, the patch adds new migration parameter parameter: compress-type to be able choose between data compressors available.
By the moment, the only available data compressor is gzip (zlib)
--- migration/migration.c | 42 ++++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 +++------ migration/qemu-file.h | 17 +++- migration/ram.c | 196 +++++++++++++++++++++++++++++++----------- qapi/migration.json | 26 ++++-- 6 files changed, 236 insertions(+), 85 deletions(-)
migration/migration.c +40 -2
@@ -739,6 +739,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->max_postcopy_bandwidth = s->parameters.max_postcopy_bandwidth; params->has_max_cpu_throttle = true; params->max_cpu_throttle = s->parameters.max_cpu_throttle; + params->has_compress_type = true; + params->compress_type = s->parameters.compress_type;return params; }
@@ -1027,10 +1029,27 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,*/ static bool migrate_params_check (MigrationParameters *params,Error **errp) { +int max_compress_level =-1 ; + +if (params->has_compress_type) { +switch (params->compress_type) { +case COMPRESSION_TYPE_ZLIB : + max_compress_level =9 ; +break ; +default : +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"compress_type" , +"values: 0 - gzip" ); +return false; + } + } +if (params->has_compress_level && - (params->compress_level >9 )) { + (params->compress_level > max_compress_level)) { +char level_range_msg[30 ]; +snprintf (level_range_msg,30 ,"values from 0 to %d" , + max_compress_level);error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"compress_level" , -"is invalid, it should be in the range of 0 to 9" ); + level_range_msg);return false; }
@@ -1125,6 +1144,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params, *dest =migrate_get_current ()->parameters;/* TODO use QAPI_CLONE() instead of duplicating it inline */ +if (params->has_compress_type) { + dest->compress_type = params->compress_type; + }if (params->has_compress_level) { dest->compress_level = params->compress_level;
@@ -1272,6 +1294,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_max_cpu_throttle) { s->parameters.max_cpu_throttle = params->max_cpu_throttle; } +if (params->has_compress_type) { + s->parameters.compress_type = params->compress_type; + } }void qmp_migrate_set_parameters (MigrateSetParameters *params,Error **errp)
@@ -1938,6 +1963,15 @@ bool migrate_use_compression(void)return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS ]; } +int migrate_compress_type (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->parameters.compress_type; +} +int migrate_compress_level (void ) {MigrationState *s;
@@ -3234,6 +3268,9 @@ static Property migration_properties[] = { decompress_error_check, true),/* Migration parameters */ +DEFINE_PROP_UINT8 ("x-compress-type" , MigrationState, + parameters.compress_type, +COMPRESSION_TYPE_ZLIB ),DEFINE_PROP_UINT8 ("x-compress-level" , MigrationState, parameters.compress_level, DEFAULT_MIGRATE_COMPRESS_LEVEL),
@@ -3346,6 +3383,7 @@ static void migration_instance_init(Object *obj) params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true; params->has_max_cpu_throttle = true; + params->has_compress_type = true;qemu_sem_init (&ms->postcopy_pause_sem,0 );qemu_sem_init (&ms->postcopy_pause_rp_sem,0 );
migration/migration.h +1 -0
@@ -280,6 +280,7 @@ bool migrate_use_return_path(void);uint64_t ram_get_total_transferred_pages (void );bool migrate_use_compression (void ); +int migrate_compress_type (void );int migrate_compress_level (void );int migrate_compress_threads (void );int migrate_compress_wait_thread (void );
migration/qemu-file.c +14 -25
@@ -662,28 +662,10 @@ uint64_t qemu_get_be64(QEMUFile *f)return v; } -/* return the size after compression, or negative value on error */ -static int qemu_compress_data (z_stream *stream,uint8_t *dest,size_t dest_len, +static int qemu_compress_data (Compression *comp,uint8_t *dest,size_t dest_len,const uint8_t *source,size_t source_len) { -int err; - - err =deflateReset (stream); -if (err !=Z_OK ) { -return -1 ; - } - - stream->avail_in = source_len; - stream->next_in = (uint8_t *)source; - stream->avail_out = dest_len; - stream->next_out = dest; - - err =deflate (stream,Z_FINISH ); -if (err !=Z_STREAM_END ) { -return -1 ; - } - -return stream->next_out - dest; +return comp->process (comp, dest, dest_len, source, source_len); }/* Compress size bytes of data start at p and store the compressed
@@ -695,23 +677,30 @@ static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,* do fflush first, if f still has no space to save the compressed * data, return -1. */ -ssize_t qemu_put_compression_data (QEMUFile *f,z_stream *stream, +ssize_t qemu_put_compression_data (QEMUFile *f,Compression *comp,const uint8_t *p,size_t size) { -ssize_t blen =IO_BUF_SIZE - f->buf_index -sizeof (int32_t ); +int blen =IO_BUF_SIZE - f->buf_index -sizeof (int32_t ); +unsigned long bound; -if (blen <compressBound (size)) { + bound = comp->get_bound (size); + +if (blen < bound) {if (!qemu_file_is_writable (f)) { +error_report ("compression: qemu file is not writable" );return -1 ; } +qemu_fflush (f); blen =IO_BUF_SIZE -sizeof (int32_t ); -if (blen <compressBound (size)) { +if (blen < bound) { +error_report ("compression: io buffer is too small:%d needed: %lu" , +IO_BUF_SIZE , bound);return -1 ; } } - blen =qemu_compress_data (stream, f->buf + f->buf_index +sizeof (int32_t ), + blen =qemu_compress_data (comp, f->buf + f->buf_index +sizeof (int32_t ), blen, p, size);if (blen <0 ) {return -1 ;
migration/qemu-file.h +16 -1
@@ -115,6 +115,21 @@ typedef struct QEMUFileHooks {QEMURamSaveFunc *save_page; } QEMUFileHooks; +typedef enum CompressionType { +COMPRESSION_TYPE_ZLIB =0 , +}CompressionType ; + +struct Compression { +CompressionType type; +bool is_decompression; +void *stream; +int (*process)(struct Compression *comp,uint8_t *dest,size_t dest_len, +const uint8_t *source,size_t source_len); +unsigned long (*get_bound)(unsigned long ); +}; + +typedef struct Compression Compression ; +QEMUFile *qemu_fopen_ops (void *opaque,const QEMUFileOps *ops);void qemu_file_set_hooks (QEMUFile *f,const QEMUFileHooks *hooks);int qemu_get_fd (QEMUFile *f);
@@ -134,7 +149,7 @@ bool qemu_file_is_writable(QEMUFile *f);size_t qemu_peek_buffer (QEMUFile *f,uint8_t **buf,size_t size,size_t offset);size_t qemu_get_buffer_in_place (QEMUFile *f,uint8_t **buf,size_t size); -ssize_t qemu_put_compression_data (QEMUFile *f,z_stream *stream, +ssize_t qemu_put_compression_data (QEMUFile *f,Compression *comp,const uint8_t *p,size_t size);int qemu_put_qemu_file (QEMUFile *f_des,QEMUFile *f_src);
migration/ram.c +146 -50
293 lines not on this page · raw message
qapi/migration.json +19 -7
@@ -480,10 +480,15 @@ # # Migration parameters enumeration # +# @compress-type: Set the compression type to be used in live migration, +# the compression type is an integer from the list: +# 0 - gzip +# # @compress-level: Set the compression level to be used in live migration, -# the compression level is an integer between 0 and 9, where 0 means -# no compression, 1 means the best compression speed, and 9 means best -# compression ratio which will consume more CPU. +# the compression level is an integer between 0 and 9, +# where 0 means no compression, 1 means the best compression speed, +# and the highest value depending on the compression type means +# the best compression ratio which will consume more CPU. # # @compress-threads: Set compression thread count to be used in live migration, # the compression thread count is an integer between 1 and 255.
@@ -560,8 +565,8 @@ # Since: 2.4 ## { 'enum': 'MigrationParameter', - 'data': ['compress-level', 'compress-threads', 'decompress-threads', - 'compress-wait-thread', + 'data': ['compress-type', 'compress-level', 'compress-threads', + 'decompress-threads', 'compress-wait-thread', 'cpu-throttle-initial', 'cpu-throttle-increment', 'tls-creds', 'tls-hostname', 'max-bandwidth', 'downtime-limit', 'x-checkpoint-delay', 'block-incremental',
@@ -572,6 +577,9 @@ ## # @MigrateSetParameters: # +# @compress-type: Compression type is used for migration. +# Available types: 0 - gzip +# # @compress-level: compression level # # @compress-threads: compression thread count
@@ -653,7 +661,8 @@ # TODO either fuse back into MigrationParameters, or make # MigrationParameters members mandatory { 'struct': 'MigrateSetParameters', - 'data': { '*compress-level': 'int', + 'data': { '*compress-type': 'int', + '*compress-level': 'int', '*compress-threads': 'int', '*compress-wait-thread': 'bool', '*decompress-threads': 'int',
@@ -692,6 +701,8 @@ # # The optional members aren't actually optional. # +# @compress-type: compression type +# # @compress-level: compression level # # @compress-threads: compression thread count
@@ -769,7 +780,8 @@ # Since: 2.4 ## { 'struct': 'MigrationParameters', - 'data': { '*compress-level': 'uint8', + 'data': { '*compress-type': 'uint8', + '*compress-level': 'uint8', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',
Signature · 1 line
2.17.0
PATCH 2/3 hmp: add compress-type parameter to migration parameters
In reply to Denis Plotnikov--- hmp.c | 8 ++++++++ 1 file changed, 8 insertions(+)
hmp.c +8 -0
@@ -334,6 +334,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) params =qmp_query_migrate_parameters (NULL );if (params) { +assert (params->has_compress_type); +monitor_printf (mon,"%s: %u\n" , +MigrationParameter_str (MIGRATION_PARAMETER_COMPRESS_TYPE ), + params->compress_type);assert (params->has_compress_level); monitor_printf(mon, "%s: %u\n", MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_LEVEL),
@@ -1757,6 +1761,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_max_postcopy_bandwidth = true;visit_type_size (v, param, &p->max_postcopy_bandwidth, &err);break ; +case MIGRATION_PARAMETER_COMPRESS_TYPE : + p->has_compress_type = true; +visit_type_int (v, param, &p->compress_type, &err); +break ;default :assert (0 ); }
Signature · 1 line
2.17.0
PATCH 3/3 migration: add zstd compression
In reply to Denis Plotnikovzstd allows to migrate with less cpu consumption maintaining the the same level of data compression as qzip (zlib).
Compression level for zstd is set with migration parameter "compress-level" in the range 1 - 22. 1 - the best speed, 22 - the best compression.
Levels in the range of 20-22 should be used with care because they lead to significant growth of CPU and memory usage.
--- configure | 26 ++++++++++++ migration/migration.c | 5 ++- migration/qemu-file.h | 1 + migration/ram.c | 95 +++++++++++++++++++++++++++++++++++++++++++ qapi/migration.json | 6 +-- 5 files changed, 129 insertions(+), 4 deletions(-)
configure +26 -0
@@ -432,6 +432,7 @@ opengl_dmabuf="no" cpuid_h="no" avx2_opt="" zlib="yes" +zstd="yes" capstone="" lzo="" snappy=""
@@ -1301,6 +1302,8 @@ for opt do ;; --disable-zlib-test) zlib="no" ;; + --disable-zstd-test) zstd="no" + ;; --disable-lzo) lzo="no" ;; --enable-lzo) lzo="yes"
@@ -3586,6 +3589,29 @@ EOF fi fi +######################################### +# zstd check + +if test "$zstd" != "no" ; then + if $pkg_config --exists libzstd; then + zstd_cflags=$($pkg_config --cflags libzstd) + zstd_libs=$($pkg_config --libs libzstd) + QEMU_CFLAGS="$zstd_cflags $QEMU_CFLAGS" + LIBS="$zstd_libs $LIBS" + else + cat > $TMPC << EOF +#include <zstd.h> +int main(void) { ZSTD_versionNumber(); return 0; } +EOF + if compile_prog "" "-lzstd" ; then + LIBS="$LIBS -lzstd" + else + error_exit "zstd check failed" \ + "Make sure to have the zstd libs and headers installed." + fi + fi +fi + ########################################## # SHA command probe for modules if test "$modules" = yes; then
migration/migration.c +4 -1
@@ -1036,9 +1036,12 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)case COMPRESSION_TYPE_ZLIB : max_compress_level =9 ;break ; +case COMPRESSION_TYPE_ZSTD : + max_compress_level =22 ; +break ;default :error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"compress_type" , -"values: 0 - gzip" ); +"values: 0 - gzip, 1 - zstd" );return false; } }
migration/qemu-file.h +1 -0
@@ -117,6 +117,7 @@ typedef struct QEMUFileHooks {typedef enum CompressionType {COMPRESSION_TYPE_ZLIB =0 , +COMPRESSION_TYPE_ZSTD =1 , }CompressionType ;struct Compression {
migration/ram.c +95 -0
119 lines not on this page · raw message
qapi/migration.json +3 -3
@@ -482,10 +482,10 @@ # # @compress-type: Set the compression type to be used in live migration, # the compression type is an integer from the list: -# 0 - gzip +# 0 - gzip, 1 -zstd # # @compress-level: Set the compression level to be used in live migration, -# the compression level is an integer between 0 and 9, +# the compression level is an integer between 0 and gzip:9, zstd:22 # where 0 means no compression, 1 means the best compression speed, # and the highest value depending on the compression type means # the best compression ratio which will consume more CPU.
@@ -578,7 +578,7 @@ # @MigrateSetParameters: # # @compress-type: Compression type is used for migration. -# Available types: 0 - gzip +# Available types: 0 - gzip, 1 - zstd # # @compress-level: compression level #
Signature · 1 line
2.17.0
ping!
Quoted 56 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++ hmp.c | 8 ++ migration/migration.c | 45 ++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++---- migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++++++++++++++++++++++++++++++++++-------- qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
ping ping
Quoted 60 lines · Denis Plotnikov
ping!
Quoted 56 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++ hmp.c | 8 ++ migration/migration.c | 45 ++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++---- migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++++++++++++++++++++++++++++++++++-------- qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
ping ping ping!
Quoted 60 lines · Denis Plotnikov
ping!
Quoted 56 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++ hmp.c | 8 ++ migration/migration.c | 45 ++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++---- migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++++++++++++++++++++++++++++++++++-------- qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
cover migration: add sztd compression
ping!
Quoted 72 lines · Denis Plotnikov
ping ping ping ping ping!
Quoted 68 lines · Denis Plotnikov
ping ping ping ping!
Quoted 64 lines · Denis Plotnikov
ping ping ping!
Quoted 60 lines · Denis Plotnikov
ping!
Quoted 56 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++
hmp.c| 8 ++
migration/migration.c| 45 ++++++-
migration/migration.h| 1 +
migration/qemu-file.c| 39 ++----migration/qemu-file.h| 18 ++-
migration/ram.c| 291 ++++++++++++++++++++++++++++++++++--------
qapi/migration.json| 26 +++-
8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
cover migration: add sztd compression
In reply to a message not in the archiveping!
Quoted 72 lines · Denis Plotnikov
ping ping ping ping ping!
Quoted 68 lines · Denis Plotnikov
ping ping ping ping!
Quoted 64 lines · Denis Plotnikov
ping ping ping!
Quoted 60 lines · Denis Plotnikov
ping!
Quoted 56 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++
hmp.c| 8 ++
migration/migration.c| 45 ++++++-
migration/migration.h| 1 +
migration/qemu-file.c| 39 ++----migration/qemu-file.h| 18 ++-
migration/ram.c| 291 ++++++++++++++++++++++++++++++++++--------
qapi/migration.json| 26 +++-
8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
Quoted 87 lines · Denis Plotnikov
ping!
Quoted 83 lines · Denis Plotnikov
ping ping ping ping ping!
Quoted 79 lines · Denis Plotnikov
ping ping ping ping!
Quoted 75 lines · Denis Plotnikov
ping ping ping!
Quoted 71 lines · Denis Plotnikov
ping!
Quoted 67 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression.
It might be useful to employ the algorithm in VM migration to reduce CPU usage.
A user will be able to choose between those algorithms, therefor compress-type
migration parameter is added.Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------
compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31 time diff to zlib, sec -25 -41Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15
compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
time, sec 154 | 22 23 27 | 23 23 25
cpu idle, % 99 | 45 30 12 | 70 52 23
cpu idle diff to zlib | | -25% -22% -11%Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compressionconfigure | 26 ++++
hmp.c | 8 ++migration/migration.c | 45 ++++++- migration/migration.h | 1 + migration/qemu-file.c | 39 ++---- migration/qemu-file.h | 18 ++- migration/ram.c | 291 ++++++++++++++++++++++++++++++++++-------- qapi/migration.json | 26 +++- 8 files changed, 369 insertions(+), 85 deletions(-)
Signature · 2 lines
Best, Denis
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent
ERROR: zstd check failed
Make sure to have the zstd libs and headers installed.
# QEMU configure log Wed Mar 6 11:31:58 UTC 2019
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 636 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c:2:2: error: #error __linux__ not defined
#error __linux__ not defined
^~~~~
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 688 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c:2:2: error: #error __i386__ not defined
#error __i386__ not defined
^~~~~
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 691 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c:2:2: error: #error __ILP32__ not defined
#error __ILP32__ not defined
^~~~~
---
lines: 92 128 923 0
x86_64-w64-mingw32-gcc -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -g -liberty /usr/lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld: cannot find -liberty
collect2: error: ld returned 1 exit status
funcs: do_compiler do_cc compile_object main
lines: 92 122 1825 0
---
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Wstring-plus-int -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Wstring-plus-int'; did you mean '-Wstrict-aliasing'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Winitializer-overrides -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Winitializer-overrides'; did you mean '-Wno-suggest-override'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
---
lines: 92 122 2165 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -Werror -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c: In function 'main':config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
return printf("%zu", SIZE_MAX);
^~~~~
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
return printf("%zu", SIZE_MAX);
^~~~~cc1: all warnings being treated as errors
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2177 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g config-temp/qemu-conf.c:1:10: fatal error: sys/socket.h: No such file or directory
#include <sys/socket.h>
^~~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzo2 config-temp/qemu-conf.c:1:10: fatal error: lzo/lzo1x.h: No such file or directory
#include <lzo/lzo1x.h>
^~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2267 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsnappy config-temp/qemu-conf.c:1:10: fatal error: snappy-c.h: No such file or directory
#include <snappy-c.h>
^~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2304 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzfse config-temp/qemu-conf.c:1:10: fatal error: lzfse.h: No such file or directory
#include <lzfse.h>
^~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2387 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lxenstore -lxenctrl -lxenguest config-temp/qemu-conf.c:1:10: fatal error: xenctrl.h: No such file or directory
#include <xenctrl.h>
^~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2892 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpam config-temp/qemu-conf.c:1:10: fatal error: security/pam_appl.h: No such file or directory
#include <security/pam_appl.h>
^~~~~~~~~~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_object check_include main
lines: 92 122 627 2907 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c config-temp/qemu-conf.c:1:10: fatal error: ifaddrs.h: No such file or directory
#include <ifaddrs.h>
^~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3045 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -Dmain=SDL_main -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2 -Wno-undef -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -L/usr/x86_64-w64-mingw32/sys-root/mingw/lib -lmingw32 -lSDL2main -lSDL2 -mwindows config-temp/qemu-conf.c:5:2: error: #error No x11 support
#error No x11 support
^~~~~
In file included from /usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL.h:32,
from config-temp/qemu-conf.c:1:/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL_main.h:111:17: error: conflicting types for 'SDL_main'
#define main SDL_main
^~~~~~~~config-temp/qemu-conf.c:7:5: note: in expansion of macro 'main'
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3060 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lrdmacm -libverbs -libumad config-temp/qemu-conf.c:1:10: fatal error: rdma/rdma_cma.h: No such file or directory
#include <rdma/rdma_cma.h>
^~~~~~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3130 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsasl2 config-temp/qemu-conf.c:1:10: fatal error: sasl/sasl.h: No such file or directory
#include <sasl/sasl.h>
^~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3224 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g config-temp/qemu-conf.c:2:10: fatal error: xfs/xfs.h: No such file or directory
#include <xfs/xfs.h>
^~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lvdeplug config-temp/qemu-conf.c:1:10: fatal error: libvdeplug.h: No such file or directory
#include <libvdeplug.h>
^~~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3298 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lcap-ng config-temp/qemu-conf.c:1:10: fatal error: cap-ng.h: No such file or directory
#include <cap-ng.h>
^~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3394 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lbrlapi config-temp/qemu-conf.c:1:10: fatal error: brlapi.h: No such file or directory
#include <brlapi.h>
^~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
#include <curses.h>
^~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpdcurses config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
#include <curses.h>
^~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3489 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g config-temp/qemu-conf.c:1:10: fatal error: bluetooth/bluetooth.h: No such file or directory
#include <bluetooth/bluetooth.h>
^~~~~~~~~~~~~~~~~~~~~~~compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3609 0
x86_64-w64-mingw32-gcc -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lzstd config-temp/qemu-conf.c:1:10: fatal error: zstd.h: No such file or directory
#include <zstd.h>
^~~~~~~~compilation terminated.
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to [email protected]
Quoted 6 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
Please, could you comment on the series:
[PATCH v3 00/21] Multifd Migration Compression
That series integrated zstd/zlib compression on top of multifd, advantages over "old" compression code are:
- We don't have to copy data back and forth
- The unit of compression is 512KB instead of 4kb
- We "conserve" the compression state between packets (this is specially interesting for zstd, that uses dictionaries)
Quoted 25 lines
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
Quoted 7 lines
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%
I don't have handy results, but it looked for me like you:
- zstd has a way better latency than zlib (i.e. the packet cames sooner)
- And it compress much better
On the migration test (best possible case for a compressor, as we are writting just one byte of each page, and we write the same value in all pages):
- zlib: compress 512KB -> 2500 bytes
- zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it looked too small)
See that I posted another patch to "delete" the old compression code. Why?
- I have been unable to modify migration-test to test it and work reliablely (only way was to allow a really huge downtime)
- Even with slow networking (1Gigabit) I got really mixed results, because as it is so slow, the guest continue dirtying memory, and in my tests it was never a winner
Thanks, Juan.
Hi, Juan
I'll read the series soon. Thanks for sending that to me!
Denis
Quoted 72 lines · Juan Quintela
Quoted 6 lines · Denis Plotnikov
zstd date compression algorithm shows better performance on data compression. It might be useful to employ the algorithm in VM migration to reduce CPU usage. A user will be able to choose between those algorithms, therefor compress-type migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
Please, could you comment on the series:
[PATCH v3 00/21] Multifd Migration Compression
That series integrated zstd/zlib compression on top of multifd, advantages over "old" compression code are:
- We don't have to copy data back and forth
- The unit of compression is 512KB instead of 4kb
- We "conserve" the compression state between packets (this is specially interesting for zstd, that uses dictionaries)Quoted 25 lines
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data produced by pgbench performance benchmarkThreads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------compression level 1 5 | 1 5 compression ratio 6.92 7.05 | 6.69 6.89 cpu idle, % 82 83 | 86 80 time, sec 49 71 | 26 31time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
Quoted 7 lines
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------compression level - | 1 5 9 | 1 5 15 compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93 time, sec 154 | 22 23 27 | 23 23 25 cpu idle, % 99 | 45 30 12 | 70 52 23 cpu idle diff to zlib | | -25% -22% -11%I don't have handy results, but it looked for me like you:
- zstd has a way better latency than zlib (i.e. the packet cames sooner)
- And it compress much betterOn the migration test (best possible case for a compressor, as we are writting just one byte of each page, and we write the same value in all pages):
- zlib: compress 512KB -> 2500 bytes
- zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it looked too small)See that I posted another patch to "delete" the old compression code. Why?
- I have been unable to modify migration-test to test it and work reliablely (only way was to allow a really huge downtime)
- Even with slow networking (1Gigabit) I got really mixed results, because as it is so slow, the guest continue dirtying memory, and in my tests it was never a winnerThanks, Juan.