migration: use qatzip to acclerate the live migration
v2 cover migration: use qatzip to acclerate the live migration
Changes in V2:
1. Added CONFIG_QATZIP macro to control this feature
2. removed useless code changes
3. fixed typos
WIP: tests still under development
Hello,
qatzip https://github.com/intel/QATzip is a project that supply the zlib like api with the Intel QAT hardware. compress and decompress performance with small data block (4kb page) is low with qatzip API. so we compose multiple pages into a multi page address structure which is a buffer list, then we compose these pages together and then call compress and decompress. we also expand the io_buf_size since qat is fast have have stress to the IO_BUFF.
If there is no QAT hardware, qatzip will fall back to the software with proper configuration. that mean if host server have the QAT hardware, while destination server don't have QAT, the live migration still work well.
To use the qatzip, one parameter is added in both host and destination side:
(qemu) migrate_set_parameter compress-with-qat on if you proper eabled the qatzip, then you can benefit from the QAT acceleration.
Since the compress/decompress offloading to the QAT hardware, no CPU resource needed (only a periodic polling need some CPU resource).
For the performance, with the same threads scale, with QATzip (HW), the live migration will have 2~4 time of performance.
Plan: so far qatzip API support SVM,but no batch APIs and the requirement is raised. once the batch APIs ready, the perf expect to reach close to 10x.
you.chen (2):
migration: add build option to support qemu build with qatzip
migration: add support for qatzip compression when doing live
migration
meson.build | 11 +- meson_options.txt | 2 + migration/migration-hmp-cmds.c | 4 + migration/migration.c | 22 ++ migration/migration.h | 1 + migration/qemu-file.c | 15 +- migration/qemu-file.h | 5 + migration/ram.c | 675 +++++++++++++++++++++++++++++---- qapi/migration.json | 8 +- scripts/meson-buildoptions.sh | 3 + 10 files changed, 673 insertions(+), 73 deletions(-)
Signature · 1 line
2.27.0
PATCH v2 1/2 migration: add build option to support qemu build with qatzip
In this patch, we added qatzip build option in the meson_options.txt and meson-buildoptons.sh to support qemu build with qatzip.
If you installed qatzip and would like to use it for live migration, you could use "--enable-qatzip" during configure, it will check qatzip availablility from the pkg-config list (Please make sure you correctly set PKG_CONFIG_PATH to include qatzip.pc). If you simply use "configure" or use "configure --disable-qatzip", qatzip will not be enabled.
--- meson.build | 11 ++++++++++- meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 15 insertions(+), 1 deletion(-)
meson.build +10 -1
@@ -553,6 +553,13 @@ if have_system or have_tools endif zlib = dependency('zlib', required: true, kwargs: static_kwargs) +qatzip = not_found +if not get_option('qatzip').auto() + qatzip = dependency('qatzip', required: get_option('qatzip'), + method: 'pkg-config', + static: enable_static) +endif + libaio = not_found if not get_option('linux_aio').auto() or have_block libaio = cc.find_library('aio', has_headers: ['libaio.h'],
@@ -1863,6 +1870,7 @@ config_host_data.set('CONFIG_LIBISCSI', libiscsi.found()) config_host_data.set('CONFIG_LIBNFS', libnfs.found()) config_host_data.set('CONFIG_LIBSSH', libssh.found()) config_host_data.set('CONFIG_LINUX_AIO', libaio.found()) +config_host_data.set('CONFIG_QATZIP', qatzip.found()) config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found()) config_host_data.set('CONFIG_LIBPMEM', libpmem.found()) config_host_data.set('CONFIG_NUMA', numa.found())
@@ -3339,7 +3347,7 @@ libmigration = static_library('migration', sources: migration_files + genh, name_suffix: 'fa', build_by_default: false) migration = declare_dependency(link_with: libmigration, - dependencies: [zlib, qom, io]) + dependencies: [qatzip, zlib, qom, io]) softmmu_ss.add(migration) block_ss = block_ss.apply(config_host, strict: false)
@@ -3986,6 +3994,7 @@ summary_info += {'vde support': vde} summary_info += {'netmap support': have_netmap} summary_info += {'l2tpv3 support': have_l2tpv3} summary_info += {'Linux AIO support': libaio} +summary_info += {'qatzip compress support': qatzip} summary_info += {'Linux io_uring support': linux_io_uring} summary_info += {'ATTR/XATTR support': libattr} summary_info += {'RDMA support': rdma}
meson_options.txt +2 -0
@@ -191,6 +191,8 @@ option('smartcard', type : 'feature', value : 'auto', description: 'CA smartcard emulation support') option('snappy', type : 'feature', value : 'auto', description: 'snappy compression support') +option('qatzip', type : 'feature', value : 'auto', + description: 'qatzip compress support') option('spice', type : 'feature', value : 'auto', description: 'Spice server support') option('spice_protocol', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +3 -0
@@ -152,6 +152,7 @@ meson_options_help() { printf "%s\n" ' slirp-smbd use smbd (at path --smbd=*) in slirp networking' printf "%s\n" ' smartcard CA smartcard emulation support' printf "%s\n" ' snappy snappy compression support' + printf "%s\n" ' qatzip qatzip compression support' printf "%s\n" ' sndio sndio sound support' printf "%s\n" ' sparse sparse checker' printf "%s\n" ' spice Spice server support'
@@ -332,6 +333,8 @@ _meson_option_parse() { --disable-libvduse) printf "%s" -Dlibvduse=disabled ;; --enable-linux-aio) printf "%s" -Dlinux_aio=enabled ;; --disable-linux-aio) printf "%s" -Dlinux_aio=disabled ;; + --enable-qatzip) printf "%s" -Dqatzip=enabled ;; + --disable-qatzip) printf "%s" -Dqatzip=disabled ;; --enable-linux-io-uring) printf "%s" -Dlinux_io_uring=enabled ;; --disable-linux-io-uring) printf "%s" -Dlinux_io_uring=disabled ;; --enable-live-block-migration) printf "%s" -Dlive_block_migration=enabled ;;
Signature · 1 line
2.27.0
Re: [PATCH v2 1/2] migration: add build option to support qemu build with qatzip
Quoted 3 lines · you.chen
In this patch, we added qatzip build option in the meson_options.txt and
meson-buildoptons.shto support qemu build with qatzip.
If you installed qatzip and would like to use it for live migration, you could use "--enable-qatzip" during configure, it will check qatzip availablility from the pkg-config list (Please make sure you correctly setPKG_CONFIG_PATHto include qatzip.pc). If you simply use "configure" or use "configure --disable-qatzip", qatzip will not be enabled.
When neither specifying --disable-qatzip nor --enable-qatzip, the default should be auto-detection, not disablement ... or is this too instable to be enabled by default?
Quoted 14 lines
Co-developed-by dennis.wu <[email protected]>Signed-off-by you.chen <[email protected]>--- meson.build | 11 ++++++++++- meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 15 insertions(+), 1 deletion(-)meson.build +0
-0
meson.build if have_system or have_tools · line 557
Quoted 5 lines
meson.build +2
-0@@ -553,6 +553,13 @@ if have_system or have_tools endif zlib = dependency('zlib', required: true, kwargs: static_kwargs) +qatzip = not_found +if not get_option('qatzip').auto()
This looks weird - why ignoring the library when the option should be auto-detected? I think you likely want something like this instead:
if not get_option('qatzip').auto() or have_system
... so that the feature gets enabled automatically when system emulator binaries are built and the library is available.
Quoted 17 lines
meson.build +6
-0+ qatzip = dependency('qatzip', required: get_option('qatzip'), + method: 'pkg-config', + static: enable_static) +endif + libaio = not_found if not get_option('linux_aio').auto() or have_block libaio = cc.find_library('aio', has_headers: ['libaio.h'],@@ -1863,6 +1870,7 @@ config_host_data.set('CONFIG_LIBISCSI', libiscsi.found()) config_host_data.set('CONFIG_LIBNFS', libnfs.found()) config_host_data.set('CONFIG_LIBSSH', libssh.found()) config_host_data.set('CONFIG_LINUX_AIO', libaio.found()) +config_host_data.set('CONFIG_QATZIP', qatzip.found()) config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found()) config_host_data.set('CONFIG_LIBPMEM', libpmem.found()) config_host_data.set('CONFIG_NUMA', numa.found())
meson.build libmigration = static_library('migration', sources: migration_files + genh, · line 3350
Quoted 5 lines
meson.build +1
-1@@ -3339,7 +3347,7 @@ libmigration = static_library('migration', sources: migration_files + genh, name_suffix: 'fa', build_by_default: false) migration = declare_dependency(link_with: libmigration, - dependencies: [zlib, qom, io]) + dependencies: [qatzip, zlib, qom, io])
Not sure whether this is the right place to add this ... can't you add the dependency in the migration/meson.build file instead?
meson.build summary_info += {'vde support': vde} · line 3997
Quoted 8 lines
meson.build +1
-0softmmu_ss.add(migration) block_ss = block_ss.apply(config_host, strict: false)@@ -3986,6 +3994,7 @@ summary_info += {'vde support': vde} summary_info += {'netmap support': have_netmap} summary_info += {'l2tpv3 support': have_l2tpv3} summary_info += {'Linux AIO support': libaio} +summary_info += {'qatzip compress support': qatzip}
Could you please keep this closer to the other compression options in the summary instead? (lzo, bzip2, zstd, ...)
Quoted 29 lines
meson.build +0
-0summary_info += {'Linux io_uring support': linux_io_uring} summary_info += {'ATTR/XATTR support': libattr} summary_info += {'RDMA support': rdma}meson_options.txt +2
-0@@ -191,6 +191,8 @@ option('smartcard', type : 'feature', value : 'auto', description: 'CA smartcard emulation support') option('snappy', type : 'feature', value : 'auto', description: 'snappy compression support') +option('qatzip', type : 'feature', value : 'auto', + description: 'qatzip compress support') option('spice', type : 'feature', value : 'auto', description: 'Spice server support') option('spice_protocol', type : 'feature', value : 'auto',scripts/meson-buildoptions.sh +1
-0@@ -152,6 +152,7 @@ meson_options_help() { printf "%s\n" ' slirp-smbd use smbd (at path --smbd=*) in slirp networking' printf "%s\n" ' smartcard CA smartcard emulation support' printf "%s\n" ' snappy snappy compression support' + printf "%s\n" ' qatzip qatzip compression support' printf "%s\n" ' sndio sndio sound support' printf "%s\n" ' sparse sparse checker' printf "%s\n" ' spice Spice server support'
scripts/meson-buildoptions.sh _meson_option_parse() { · line 337
Quoted 5 lines
scripts/meson-buildoptions.sh +2
-0@@ -332,6 +333,8 @@ _meson_option_parse() { --disable-libvduse) printf "%s" -Dlibvduse=disabled ;; --enable-linux-aio) printf "%s" -Dlinux_aio=enabled ;; --disable-linux-aio) printf "%s" -Dlinux_aio=disabled ;; + --enable-qatzip) printf "%s" -Dqatzip=enabled ;; + --disable-qatzip) printf "%s" -Dqatzip=disabled ;;
The placement here looks rather random ... Could you please run scripts/meson-buildoptions.py to create this file instead?
scripts/meson-buildoptions.sh _meson_option_parse() { · line 340
Quoted 3 lines
scripts/meson-buildoptions.sh +0
-0--enable-linux-io-uring) printf "%s" -Dlinux_io_uring=enabled ;; --disable-linux-io-uring) printf "%s" -Dlinux_io_uring=disabled ;; --enable-live-block-migration) printf "%s" -Dlive_block_migration=enabled ;;
Thomas
PATCH v2 2/2 migration: add support for qatzip compression when doing live migration
In reply to you.chenAdd config and logics to use qatzip for page compression, in order to support qatzip compression better, we collect multipe pages together to do qatzip compression for best performance. And we use compile option CONFIG_QATZIP to determine whether should qatzip related code be compiled or not.
--- migration/migration-hmp-cmds.c | 4 + migration/migration.c | 22 ++ migration/migration.h | 1 + migration/qemu-file.c | 15 +- migration/qemu-file.h | 5 + migration/ram.c | 675 +++++++++++++++++++++++++++++---- qapi/migration.json | 8 +- 7 files changed, 658 insertions(+), 72 deletions(-)
migration/migration-hmp-cmds.c +4 -0
@@ -479,6 +479,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_compress_threads = true;visit_type_uint8 (v, param, &p->compress_threads, &err);break ; +case MIGRATION_PARAMETER_COMPRESS_WITH_QAT : + p->has_compress_with_qat = true; +visit_type_bool (v, param, &p->compress_with_qat, &err); +break ;case MIGRATION_PARAMETER_COMPRESS_WAIT_THREAD : p->has_compress_wait_thread = true;visit_type_bool (v, param, &p->compress_wait_thread, &err);
migration/migration.c +22 -0
@@ -962,6 +962,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->compress_level = s->parameters.compress_level; params->has_compress_threads = true; params->compress_threads = s->parameters.compress_threads; + params->has_compress_with_qat = true; + params->compress_with_qat = s->parameters.compress_with_qat; params->has_compress_wait_thread = true; params->compress_wait_thread = s->parameters.compress_wait_thread; params->has_decompress_threads = true;
@@ -1678,6 +1680,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params, dest->compress_threads = params->compress_threads; } +if (params->has_compress_with_qat) { + dest->compress_with_qat = params->compress_with_qat; + } +if (params->has_compress_wait_thread) { dest->compress_wait_thread = params->compress_wait_thread; }
@@ -1775,6 +1781,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) s->parameters.compress_threads = params->compress_threads; } +if (params->has_compress_with_qat) { + s->parameters.compress_with_qat = params->compress_with_qat; + } +if (params->has_compress_wait_thread) { s->parameters.compress_wait_thread = params->compress_wait_thread; }
@@ -2620,6 +2630,15 @@ bool migrate_use_compression(void)return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS ]; } +bool migrate_compress_with_qat (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return migrate_use_compression () && s->parameters.compress_with_qat; +} +int migrate_compress_level (void ) {MigrationState *s;
@@ -4451,6 +4470,8 @@ static Property migration_properties[] = {DEFINE_PROP_UINT8 ("x-compress-threads" , MigrationState, parameters.compress_threads,DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT ), +DEFINE_PROP_BOOL ("x-compress-with-qat" , MigrationState, + parameters.compress_with_qat, false),DEFINE_PROP_BOOL ("x-compress-wait-thread" , MigrationState, parameters.compress_wait_thread, true), DEFINE_PROP_UINT8("x-decompress-threads", MigrationState,
@@ -4580,6 +4601,7 @@ static void migration_instance_init(Object *obj) params->has_compress_level = true; params->has_compress_threads = true; params->has_compress_wait_thread = true; + params->has_compress_with_qat = true; params->has_decompress_threads = true; params->has_throttle_trigger_threshold = true; params->has_cpu_throttle_initial = true;
migration/migration.h +1 -0
@@ -443,6 +443,7 @@ bool migrate_use_return_path(void);uint64_t ram_get_total_transferred_pages (void );bool migrate_use_compression (void ); +bool migrate_compress_with_qat (void );int migrate_compress_level (void );int migrate_compress_threads (void );int migrate_compress_wait_thread (void );
migration/qemu-file.c +13 -2
@@ -26,12 +26,18 @@#include "qemu/madvise.h" #include "qemu/error-report.h" #include "qemu/iov.h" +#include "qemu/units.h" #include "migration.h" #include "qemu-file.h" #include "trace.h" #include "qapi/error.h" -#define IO_BUF_SIZE 32768 +#ifdef CONFIG_QATZIP +#define IO_BUF_SIZE (512 * KiB) +#else +#define IO_BUF_SIZE (32 * KiB) +#endif +#define MAX_IOV_SIZE MIN_CONST(IOV_MAX, 64)struct QEMUFile {
@@ -514,7 +520,7 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,return 0 ; } -static void add_buf_to_iovec (QEMUFile *f,size_t len) +void add_buf_to_iovec (QEMUFile *f,size_t len) {if (!add_to_iovec (f, f->buf + f->buf_index, len, false)) { f->buf_index += len;
@@ -524,6 +530,11 @@ static void add_buf_to_iovec(QEMUFile *f, size_t len) } } +uint8_t *qemu_get_pos (QEMUFile *f) +{ +return f->buf + f->buf_index; +} +void qemu_put_buffer_async (QEMUFile *f,const uint8_t *buf,size_t size,bool may_free) {
migration/qemu-file.h +5 -0
@@ -25,6 +25,9 @@#ifndef MIGRATION_QEMU_FILE_H #define MIGRATION_QEMU_FILE_H +#ifdef CONFIG_QATZIP +#include <qatzip.h> +#endif #include <zlib.h> #include "exec/cpu-common.h" #include "io/channel.h"
@@ -108,6 +111,8 @@ bool qemu_file_is_writable(QEMUFile *f);#include "migration/qemu-file-types.h" +void add_buf_to_iovec (QEMUFile *f,size_t len); +uint8_t *qemu_get_pos (QEMUFile *f);size_t qemu_peek_buffer (QEMUFile *f,uint8_t **buf,size_t size,size_t offset);size_t qemu_get_buffer_in_place (QEMUFile *f,uint8_t **buf,size_t size); ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
migration/ram.c +606 -69
867 lines not on this page · raw message
qapi/migration.json +7 -1
@@ -644,6 +644,8 @@ # no compression, 1 means the best compression speed, and 9 means best # compression ratio which will consume more CPU. # +# @compress-with-qat: compress with qat on and off. (Since 8.1) +# # @compress-threads: Set compression thread count to be used in live migration, # the compression thread count is an integer between 1 and 255. #
@@ -784,7 +786,7 @@ { 'enum': 'MigrationParameter', 'data': ['announce-initial', 'announce-max', 'announce-rounds', 'announce-step', - 'compress-level', 'compress-threads', 'decompress-threads', + 'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads', 'compress-wait-thread', 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', 'cpu-throttle-tailslow',
@@ -815,6 +817,8 @@ # # @compress-level: compression level # +# @compress-with-qat: compression with qat (Since 8.1) +# # @compress-threads: compression thread count # # @compress-wait-thread: Controls behavior when all compression threads are
@@ -954,6 +958,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',
@@ -1152,6 +1157,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',
Signature · 1 line
2.27.0
Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration
Quoted 5 lines · you.chen
Add config and logics to use qatzip for page compression, in order to support qatzip compression better, we collect multipe pages together to do qatzip compression for best performance. And we use compile option
CONFIG_QATZIPto determine whether should qatzip related code be compiled or not.Co-developed-by dennis.wu <[email protected]>Signed-off-by you.chen <[email protected]>
[...]
Quoted 40 lines
qapi/migration.json +6
-1@@ -644,6 +644,8 @@ # no compression, 1 means the best compression speed, and 9 means best # compression ratio which will consume more CPU. # +# @compress-with-qat: compress with qat on and off. (Since 8.1) +# # @compress-threads: Set compression thread count to be used in live migration, # the compression thread count is an integer between 1 and 255. #@@ -784,7 +786,7 @@ { 'enum': 'MigrationParameter', 'data': ['announce-initial', 'announce-max', 'announce-rounds', 'announce-step', - 'compress-level', 'compress-threads', 'decompress-threads', + 'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads', 'compress-wait-thread', 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', 'cpu-throttle-tailslow',@@ -815,6 +817,8 @@ # # @compress-level: compression level # +# @compress-with-qat: compression with qat (Since 8.1) +# # @compress-threads: compression thread count # # @compress-wait-thread: Controls behavior when all compression threads are@@ -954,6 +958,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',
qapi/migration.json line 1163
Quoted 7 lines
qapi/migration.json +1
-0@@ -1152,6 +1157,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',
We already have MigrationCapability compress
# @compress: Use multiple compression threads to accelerate live migration.
# This feature can help to reduce the migration traffic, by sending
# compressed pages. Please note that if compress and xbzrle are both
# on, compress only takes effect in the ram bulk stage, after that,
# it will be disabled and only xbzrle takes effect, this can help to
# minimize migration traffic. The feature is disabled by default.
# (since 2.4 )
and xbzrle
# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length Encoding).
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#
and MigrationParameters / MigrateSetParameters multifd-compression
# @multifd-compression: Which compression method to use.
# Defaults to none. (Since 5.0)
#
# @multifd-zlib-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 9, where 0 means no compression, 1 means the best
# compression speed, and 9 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 20, where 0 means no compression, 1 means the best
# compression speed, and 20 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)
where multifd-compression is
##
# @MultiFDCompression:
#
# An enumeration of multifd compression methods.
#
# @none: no compression.
# @zlib: use zlib compression method.
# @zstd: use zstd compression method.
#
# Since: 5.0
##How does this all fit together? It feels like a bunch of featured piled onto each other, then shaken well. Or am I confused?
I could use an abstract description of compression in migration.
Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration
Quoted 62 lines · Markus Armbruster
Quoted 7 lines · you.chen
Add config and logics to use qatzip for page compression, in order
to support qatzip compression better, we collect multipe pages
together to do qatzip compression for best performance.
And we use compile optionCONFIG_QATZIPto determine whether should qatzip related code be compiled or not.Co-developed-by dennis.wu <[email protected]>Signed-off-by you.chen <[email protected]>[...]
Quoted 40 lines
qapi/migration.json +6
-1@@ -644,6 +644,8 @@ # no compression, 1 means the best compression speed, and 9 means best # compression ratio which will consume more CPU. # +# @compress-with-qat: compress with qat on and off. (Since 8.1) +# # @compress-threads: Set compression thread count to be used in live migration, # the compression thread count is an integer between 1 and 255. #@@ -784,7 +786,7 @@ { 'enum': 'MigrationParameter', 'data': ['announce-initial', 'announce-max', 'announce-rounds', 'announce-step', - 'compress-level', 'compress-threads', 'decompress-threads', + 'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads', 'compress-wait-thread', 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', 'cpu-throttle-tailslow',@@ -815,6 +817,8 @@ # # @compress-level: compression level # +# @compress-with-qat: compression with qat (Since 8.1) +# # @compress-threads: compression thread count # # @compress-wait-thread: Controls behavior when all compression threads are@@ -954,6 +958,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',qapi/migration.json line 1163
Quoted 7 lines
qapi/migration.json +1
-0@@ -1152,6 +1157,7 @@ '*announce-rounds': 'size', '*announce-step': 'size', '*compress-level': 'uint8', + '*compress-with-qat': 'bool', '*compress-threads': 'uint8', '*compress-wait-thread': 'bool', '*decompress-threads': 'uint8',We already have MigrationCapability compress
Quoted 7 lines
# @compress: Use multiple compression threads to accelerate live migration.
# This feature can help to reduce the migration traffic, by sending
# compressed pages. Please note that if compress and xbzrle are both
# on, compress only takes effect in the ram bulk stage, after that,
# it will be disabled and only xbzrle takes effect, this can help to
# minimize migration traffic. The feature is disabled by default.
# (since 2.4 )
I had the patch to deprecate it on 8.1.
And now colo is using it. Sniff.
Quoted 6 lines
and xbzrle
# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length Encoding).
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#
Diferent can of worms, but I agree with you.
Quoted 21 lines
and MigrationParameters / MigrateSetParameters multifd-compression
# @multifd-compression: Which compression method to use.
# Defaults to none. (Since 5.0)
#
# @multifd-zlib-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 9, where 0 means no compression, 1 means the best
# compression speed, and 9 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 20, where 0 means no compression, 1 means the best
# compression speed, and 20 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)where multifd-compression is
Quoted 11 lines
##
# @MultiFDCompression:
#
# An enumeration of multifd compression methods.
#
# @none: no compression.
# @zlib: use zlib compression method.
# @zstd: use zstd compression method.
#
# Since: 5.0
##I think it belongs here as another compression method.
Later, Juan.
Quoted 4 lines
How does this all fit together? It feels like a bunch of featured piled onto each other, then shaken well. Or am I confused?
I could use an abstract description of compression in migration.
compression -> old code, uses threads and compress a page at a time
(i.e. takes more time to copy the code to the thread that what we got in return). Data is copied several times.
xbzrle: use a cache of already sent pages and sent the difference.
Not really very useful execpt if you are migrating over data centers. Here you trade memory and cpu consumption on the host for less network bandwidth used.
Current cache size default is a joke, but that is a complete different can of worms.
multifd compression: it is much better (I know that I am the author, but
the whole reason to create it was to address the shortcuts of the old compression code). highlights:
- compress 64 pages at at time, so get much better compression ratios.
- for each channel, it never resets the compression during migration, that means that it compress much better. For the few things that I know about compression, new methods are based heavily in dictionaries, so you need long sessions to get best effectivity.
- it minimizes the number of copies. zstd, no copies at all. zlib -> it makes one copy because hardware implmentations (s390x I am looking at you) make two passes through the data in some cases.
I plan to add xbzre in the near future to multifd compression so I can also deprecate it,
But someone will appears that really needs it O:-)
Later, Juan.
Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration
In reply to you.chenQuoted 7 lines · you.chen
Add config and logics to use qatzip for page compression, in order to
support qatzip compression better, we collect multipe pages together
to do qatzip compression for best performance.
And we use compile optionCONFIG_QATZIPto determine whether should qatzip related code be compiled or not.Co-developed-by dennis.wu <[email protected]>Signed-off-by you.chen <[email protected]>
two questions:
a - why are you using this? I guess that because it is faster, but you
are not telling.
b - why are you using it with compression thread instead of multifd
compression? probably just another method.
[...]
Quoted 11 lines
MigrationState *s;migration/migration.c +2
-0@@ -4451,6 +4470,8 @@ static Property migration_properties[] = {DEFINE_PROP_UINT8 ("x-compress-threads" , MigrationState, parameters.compress_threads,DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT ), +DEFINE_PROP_BOOL ("x-compress-with-qat" , MigrationState, + parameters.compress_with_qat, false),DEFINE_PROP_BOOL ("x-compress-wait-thread" , MigrationState, parameters.compress_wait_thread, true), DEFINE_PROP_UINT8("x-decompress-threads", MigrationState,
migration/migration.c static void migration_instance_init(Object *obj) · line 4607
Quoted 7 lines
migration/migration.c +1
-0@@ -4580,6 +4601,7 @@ static void migration_instance_init(Object *obj) params->has_compress_level = true; params->has_compress_threads = true; params->has_compress_wait_thread = true; + params->has_compress_with_qat = true; params->has_decompress_threads = true; params->has_throttle_trigger_threshold = true; params->has_cpu_throttle_initial = true;
If you use it as a parameter, this bits are ok, but I still think that it will work better with a multifd compression method.
migration/qemu-file.c line 39
Quoted 6 lines
migration/qemu-file.c +5
-1-#define IO_BUF_SIZE 32768 +#ifdef CONFIG_QATZIP +#define IO_BUF_SIZE (512 * KiB) +#else +#define IO_BUF_SIZE (32 * KiB) +#endif
1st part that it is already weird O:-)
I mean, we are supposed to not have bigger buffers.
migration/qemu-file.c static void add_buf_to_iovec(QEMUFile *f, size_t len) · line 537
Quoted 5 lines
migration/qemu-file.c +5
-0+uint8_t *qemu_get_pos (QEMUFile *f) +{ +return f->buf + f->buf_index; +} +
I know I shouldn't ask, but why do you need this.
/me looks later in the patch.
migration/ram.c static void __attribute__((constructor)) init_cpu_flag(void) · line 121
Quoted 8 lines
migration/ram.c +2
-0@@ -117,6 +117,20 @@ static void __attribute__((constructor)) init_cpu_flag(void)XBZRLECacheStats xbzrle_counters; +/* define the max page number to compress together */ +#define MULTI_PAGE_NUM 64
You get 64 pages to compress with just using multifd, nothing else needed.
Quoted 45 lines
migration/ram.c +27
-0+#define COMP_BUF_SIZE (TARGET_PAGE_SIZE * MULTI_PAGE_NUM * 2) +#define DECOMP_BUF_SIZE (TARGET_PAGE_SIZE * MULTI_PAGE_NUM) + +typedef struct MultiPageAddr { +/* real pages that will compress together */ +uint64_t pages; +/* the last index of the addr*/ +uint64_t last_idx; +/* each address might contain contineous pages*/ +uint64_t addr[MULTI_PAGE_NUM ]; +}MultiPageAddr ; +/* used by the search for pages to send */ struct PageSearchStatus {/* The migration channel used for a specific host page */ @@ -127,6 +141,12 @@ struct PageSearchStatus {RAMBlock *block;/* Current page to search from */ unsigned long page; +/* +* multi page search from current page +* for compress together with qatzip +* stream APIs +*/ +MultiPageAddr mpa;/* Set once we wrap around */ bool complete_round;/* Whether we're sending a host page */ @@ -506,6 +526,15 @@ struct CompressParam {/* internally used fields */ z_stream stream;uint8_t *originbuf; + +#ifdef CONFIG_QATZIP +/*multi page address for compression*/ +MultiPageAddr mpa; +QzSession_T qzsess; +uint8_t *decompbuf; +uint8_t *compbuf; +/* QzStream_T qzstream; */ +#endif };typedef struct CompressParam CompressParam ;
migration/ram.c struct DecompressParam { · line 552
Quoted 6 lines
migration/ram.c +3
-0@@ -518,6 +547,15 @@ struct DecompressParam {uint8_t *compbuf;int len;z_stream stream; +RAMBlock *block; + +#ifdef CONFIG_QATZIP
Aside. I would expect to detect at configure time if QATZIP is available.
migration/ram.c struct DecompressParam { · line 556
Quoted 4 lines
migration/ram.c +4
-0+/* decompress multi pages with qzlib*/ +QzSession_T qzsess; +/* QzStream_T qzstream; */ +uint8_t *decompbuf;/* buffer after decompress */
The comment is wrong. You uses this variable on do_compress_ram_page, that has nothing to do with decompression.
migration/ram.c struct DecompressParam { · line 559
Quoted 3 lines
migration/ram.c +2
-0+MultiPageAddr mpa;/* destination */ +#endif };
I am not going looking after this. We already have a mechanism to look for several pages.
See multifd_queue_page().
We have to options here:
- multifd_queue_page() is better, you need to generalize it and wrote this on top of that.
- yours is better. Then you need to write multifd one on top of that.
What I don't want is two different mechanisms to collect the next 64 dirty pages. I will even consider good that you _always_ search for multiple pages, let say 64 that are the ones that we have now in both multifd and your approach and you do a for loop in case that it is normal precopy.
Another note, I am not a big far of:
#ifdef CONFIG_QATZIP
if (migrate_compress_with_qat()) {
do_compress_ram_page_multiple(param->file, ¶m->qzsess,
param->decompbuf, param->compbuf, block, ¶m->mpa);
} else {
#endif
zero_page = do_compress_ram_page(param->file,
¶m->stream, block, offset, param->originbuf);
param->zero_page = zero_page;
#ifdef CONFIG_QATZIP
}#endif
Two ifdefs and an if mixed, ouch.
Suggestions:
- Just create a function that calls one or another.
- You add another function call to migration_ops, and you add the proper function there during initialization.
Another nitbit:
- You are dropping the zero page optimization, you are just compressing it.
- See how it is done on my zero pages on multifd on the list. a - we only sent the page address for a zero page (hard to win sending zero bytes for a page)
b - I don't know how QzSession_T works, but I guess that it is going
to be a loosy equivalent of how zlib works. Notice that compression code does:
do_compress_ram_page() -> qemu_put_compression_data() -> ...
static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
const uint8_t *source, size_t source_len)
{
int err;
err = deflateReset(stream);
if (err != Z_OK) {
return -1;
}
stream->avail_in = source_len;
stream->next_in = (uint8_t *)source;
stream->avail_out = dest_len;
stream->next_out = dest;
err = deflate(stream, Z_FINISH);
if (err != Z_STREAM_END) {
return -1;
}
return stream->next_out - dest;
}I can't see the equivalent of deflateReset on your code (remember that I don't understand qat). Rememeber that the number of threads for compression and decompression can be different, so it is not necessary that you have the same dicctionaries on destination on the thread that you are using. That is one of the reasons why multifd-zlib is much, much better at compression that compression threads.
a- it uses 64pages at a time
b- as recv thread is paired with the same send thread, we don't need to
reset the dictionaries, and that makes things much, much better. I thought it was a bug the 1st time that I saw that it compressed 64 pages to less than 64bytes O:-)
Later, Juan.