Skip to contentSkip to conversation

migration: use qatzip to acclerate the live migration

PATCH In review you.chen 4 participants 7 messages Apr 17, 2023 – Apr 20, 2023 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#2 lore raw

Re: [PATCH v2 1/2] migration: add build option to support qemu build with qatzip

In reply to you.chen
Quoted 3 lines · you.chen

In this patch, we added qatzip build option in the meson_options.txt and meson-buildoptons.sh to support qemu build with qatzip.
If you installed qatzip and would like to use it for live migration, you could use "--enable-qatzip" during configure, it will check qatzip availablility from the pkg-config list (Please make sure you correctly set PKG_CONFIG_PATH to include qatzip.pc). If you simply use "configure" or use "configure --disable-qatzip", qatzip will not be enabled.

When neither specifying --disable-qatzip nor --enable-qatzip, the default should be auto-detection, not disablement ... or is this too instable to be enabled by default?

Quoted 14 lines
---
  meson.build                   | 11 ++++++++++-
  meson_options.txt             |  2 ++
  scripts/meson-buildoptions.sh |  3 +++
  3 files changed, 15 insertions(+), 1 deletion(-)
meson.build +0 -0
index 29f8644d6d..aa7445e29e 100644

meson.build if have_system or have_tools · line 557

Quoted 5 lines
meson.build +2 -0
@@ -553,6 +553,13 @@ if have_system or have_tools
  endif
  zlib = dependency('zlib', required: true, kwargs: static_kwargs)
 
+qatzip = not_found
+if not get_option('qatzip').auto()

This looks weird - why ignoring the library when the option should be auto-detected? I think you likely want something like this instead:

if not get_option('qatzip').auto() or have_system

... so that the feature gets enabled automatically when system emulator binaries are built and the library is available.

Quoted 17 lines
meson.build +6 -0
+  qatzip = dependency('qatzip', required: get_option('qatzip'),
+                         method: 'pkg-config',
+                         static: enable_static)
+endif
+
  libaio = not_found
  if not get_option('linux_aio').auto() or have_block
    libaio = cc.find_library('aio', has_headers: ['libaio.h'],
@@ -1863,6 +1870,7 @@ config_host_data.set('CONFIG_LIBISCSI', libiscsi.found())
  config_host_data.set('CONFIG_LIBNFS', libnfs.found())
  config_host_data.set('CONFIG_LIBSSH', libssh.found())
  config_host_data.set('CONFIG_LINUX_AIO', libaio.found())
+config_host_data.set('CONFIG_QATZIP', qatzip.found())
  config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found())
  config_host_data.set('CONFIG_LIBPMEM', libpmem.found())
  config_host_data.set('CONFIG_NUMA', numa.found())

meson.build libmigration = static_library('migration', sources: migration_files + genh, · line 3350

Quoted 5 lines
meson.build +1 -1
@@ -3339,7 +3347,7 @@ libmigration = static_library('migration', sources: migration_files + genh,
                                name_suffix: 'fa',
                                build_by_default: false)
  migration = declare_dependency(link_with: libmigration,
-                               dependencies: [zlib, qom, io])
+                               dependencies: [qatzip, zlib, qom, io])

Not sure whether this is the right place to add this ... can't you add the dependency in the migration/meson.build file instead?

meson.build summary_info += {'vde support': vde} · line 3997

Quoted 8 lines
meson.build +1 -0
  softmmu_ss.add(migration)
 
  block_ss = block_ss.apply(config_host, strict: false)
@@ -3986,6 +3994,7 @@ summary_info += {'vde support':       vde}
  summary_info += {'netmap support':    have_netmap}
  summary_info += {'l2tpv3 support':    have_l2tpv3}
  summary_info += {'Linux AIO support': libaio}
+summary_info += {'qatzip compress support': qatzip}

Could you please keep this closer to the other compression options in the summary instead? (lzo, bzip2, zstd, ...)

Quoted 29 lines
meson.build +0 -0
  summary_info += {'Linux io_uring support': linux_io_uring}
  summary_info += {'ATTR/XATTR support': libattr}
  summary_info += {'RDMA support':      rdma}
meson_options.txt +2 -0
index fc9447d267..ef6d639876 100644
@@ -191,6 +191,8 @@ option('smartcard', type : 'feature', value : 'auto',
         description: 'CA smartcard emulation support')
  option('snappy', type : 'feature', value : 'auto',
         description: 'snappy compression support')
+option('qatzip', type : 'feature', value : 'auto',
+       description: 'qatzip compress support')
  option('spice', type : 'feature', value : 'auto',
         description: 'Spice server support')
  option('spice_protocol', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +1 -0
index 009fab1515..84d110197d 100644
@@ -152,6 +152,7 @@ meson_options_help() {
    printf "%s\n" '  slirp-smbd      use smbd (at path --smbd=*) in slirp networking'
    printf "%s\n" '  smartcard       CA smartcard emulation support'
    printf "%s\n" '  snappy          snappy compression support'
+  printf "%s\n" '  qatzip          qatzip compression support'
    printf "%s\n" '  sndio           sndio sound support'
    printf "%s\n" '  sparse          sparse checker'
    printf "%s\n" '  spice           Spice server support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 337

Quoted 5 lines
scripts/meson-buildoptions.sh +2 -0
@@ -332,6 +333,8 @@ _meson_option_parse() {
      --disable-libvduse) printf "%s" -Dlibvduse=disabled ;;
      --enable-linux-aio) printf "%s" -Dlinux_aio=enabled ;;
      --disable-linux-aio) printf "%s" -Dlinux_aio=disabled ;;
+    --enable-qatzip) printf "%s" -Dqatzip=enabled ;;
+    --disable-qatzip) printf "%s" -Dqatzip=disabled ;;

The placement here looks rather random ... Could you please run scripts/meson-buildoptions.py to create this file instead?

scripts/meson-buildoptions.sh _meson_option_parse() { · line 340

Quoted 3 lines
scripts/meson-buildoptions.sh +0 -0
      --enable-linux-io-uring) printf "%s" -Dlinux_io_uring=enabled ;;
      --disable-linux-io-uring) printf "%s" -Dlinux_io_uring=disabled ;;
      --enable-live-block-migration) printf "%s" -Dlive_block_migration=enabled ;;

Thomas

#4 lore raw

Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration

In reply to you.chen
Quoted 5 lines · you.chen

Add config and logics to use qatzip for page compression, in order to support qatzip compression better, we collect multipe pages together to do qatzip compression for best performance. And we use compile option CONFIG_QATZIP to determine whether should qatzip related code be compiled or not.

[...]

Quoted 40 lines
qapi/migration.json +6 -1
index c84fa10e86..6459927c7a 100644
@@ -644,6 +644,8 @@
 #                  no compression, 1 means the best compression speed, and 9 means best
 #                  compression ratio which will consume more CPU.
 #
+# @compress-with-qat: compress with qat on and off. (Since 8.1)
+#
 # @compress-threads: Set compression thread count to be used in live migration,
 #                    the compression thread count is an integer between 1 and 255.
 #
@@ -784,7 +786,7 @@
 { 'enum': 'MigrationParameter',
   'data': ['announce-initial', 'announce-max',
            'announce-rounds', 'announce-step',
-           'compress-level', 'compress-threads', 'decompress-threads',
+           'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads',
            'compress-wait-thread', 'throttle-trigger-threshold',
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'cpu-throttle-tailslow',
@@ -815,6 +817,8 @@
 #
 # @compress-level: compression level
 #
+# @compress-with-qat: compression with qat (Since 8.1)
+#
 # @compress-threads: compression thread count
 #
 # @compress-wait-thread: Controls behavior when all compression threads are
@@ -954,6 +958,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',

qapi/migration.json line 1163

Quoted 7 lines
qapi/migration.json +1 -0
@@ -1152,6 +1157,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',

We already have MigrationCapability compress

# @compress: Use multiple compression threads to accelerate live migration.
# This feature can help to reduce the migration traffic, by sending
# compressed pages. Please note that if compress and xbzrle are both
# on, compress only takes effect in the ram bulk stage, after that,
# it will be disabled and only xbzrle takes effect, this can help to
# minimize migration traffic. The feature is disabled by default.
# (since 2.4 )

and xbzrle

# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length Encoding).
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#

and MigrationParameters / MigrateSetParameters multifd-compression

# @multifd-compression: Which compression method to use.
# Defaults to none. (Since 5.0)
#
# @multifd-zlib-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 9, where 0 means no compression, 1 means the best
# compression speed, and 9 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 20, where 0 means no compression, 1 means the best
# compression speed, and 20 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)

where multifd-compression is

    ##
    # @MultiFDCompression:
    #
    # An enumeration of multifd compression methods.
    #
    # @none: no compression.
    # @zlib: use zlib compression method.
    # @zstd: use zstd compression method.
    #
    # Since: 5.0
    ##

How does this all fit together? It feels like a bunch of featured piled onto each other, then shaken well. Or am I confused?

I could use an abstract description of compression in migration.

you.chen

author
#0 lore raw

v2 cover migration: use qatzip to acclerate the live migration

Changes in V2:
1. Added CONFIG_QATZIP macro to control this feature

2. removed useless code changes

3. fixed typos

WIP: tests still under development

Hello,

qatzip https://github.com/intel/QATzip is a project that supply the zlib like api with the Intel QAT hardware. compress and decompress performance with small data block (4kb page) is low with qatzip API. so we compose multiple pages into a multi page address structure which is a buffer list, then we compose these pages together and then call compress and decompress. we also expand the io_buf_size since qat is fast have have stress to the IO_BUFF.

If there is no QAT hardware, qatzip will fall back to the software with proper configuration. that mean if host server have the QAT hardware, while destination server don't have QAT, the live migration still work well.

To use the qatzip, one parameter is added in both host and destination side:
(qemu) migrate_set_parameter compress-with-qat on if you proper eabled the qatzip, then you can benefit from the QAT acceleration.

Since the compress/decompress offloading to the QAT hardware, no CPU resource needed (only a periodic polling need some CPU resource).

For the performance, with the same threads scale, with QATzip (HW), the live migration will have 2~4 time of performance.

Plan: so far qatzip API support SVM,but no batch APIs and the requirement is raised. once the batch APIs ready, the perf expect to reach close to 10x.

you.chen (2):
migration: add build option to support qemu build with qatzip
migration: add support for qatzip compression when doing live
migration

 meson.build                    |  11 +-
 meson_options.txt              |   2 +
 migration/migration-hmp-cmds.c |   4 +
 migration/migration.c          |  22 ++
 migration/migration.h          |   1 +
 migration/qemu-file.c          |  15 +-
 migration/qemu-file.h          |   5 +
 migration/ram.c                | 675 +++++++++++++++++++++++++++++----
 qapi/migration.json            |   8 +-
 scripts/meson-buildoptions.sh  |   3 +
 10 files changed, 673 insertions(+), 73 deletions(-)
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH v2 1/2 migration: add build option to support qemu build with qatzip

In this patch, we added qatzip build option in the meson_options.txt and meson-buildoptons.sh to support qemu build with qatzip.
If you installed qatzip and would like to use it for live migration, you could use "--enable-qatzip" during configure, it will check qatzip availablility from the pkg-config list (Please make sure you correctly set PKG_CONFIG_PATH to include qatzip.pc). If you simply use "configure" or use "configure --disable-qatzip", qatzip will not be enabled.

---
 meson.build                   | 11 ++++++++++-
 meson_options.txt             |  2 ++
 scripts/meson-buildoptions.sh |  3 +++
 3 files changed, 15 insertions(+), 1 deletion(-)
meson.build +10 -1
index 29f8644d6d..aa7445e29e 100644
@@ -553,6 +553,13 @@ if have_system or have_tools
 endif
 zlib = dependency('zlib', required: true, kwargs: static_kwargs)
 
+qatzip = not_found
+if not get_option('qatzip').auto()
+  qatzip = dependency('qatzip', required: get_option('qatzip'),
+                         method: 'pkg-config',
+                         static: enable_static)
+endif
+
 libaio = not_found
 if not get_option('linux_aio').auto() or have_block
   libaio = cc.find_library('aio', has_headers: ['libaio.h'],
@@ -1863,6 +1870,7 @@ config_host_data.set('CONFIG_LIBISCSI', libiscsi.found())
 config_host_data.set('CONFIG_LIBNFS', libnfs.found())
 config_host_data.set('CONFIG_LIBSSH', libssh.found())
 config_host_data.set('CONFIG_LINUX_AIO', libaio.found())
+config_host_data.set('CONFIG_QATZIP', qatzip.found())
 config_host_data.set('CONFIG_LINUX_IO_URING', linux_io_uring.found())
 config_host_data.set('CONFIG_LIBPMEM', libpmem.found())
 config_host_data.set('CONFIG_NUMA', numa.found())
@@ -3339,7 +3347,7 @@ libmigration = static_library('migration', sources: migration_files + genh,
                               name_suffix: 'fa',
                               build_by_default: false)
 migration = declare_dependency(link_with: libmigration,
-                               dependencies: [zlib, qom, io])
+                               dependencies: [qatzip, zlib, qom, io])
 softmmu_ss.add(migration)
 
 block_ss = block_ss.apply(config_host, strict: false)
@@ -3986,6 +3994,7 @@ summary_info += {'vde support':       vde}
 summary_info += {'netmap support':    have_netmap}
 summary_info += {'l2tpv3 support':    have_l2tpv3}
 summary_info += {'Linux AIO support': libaio}
+summary_info += {'qatzip compress support': qatzip}
 summary_info += {'Linux io_uring support': linux_io_uring}
 summary_info += {'ATTR/XATTR support': libattr}
 summary_info += {'RDMA support':      rdma}
meson_options.txt +2 -0
index fc9447d267..ef6d639876 100644
@@ -191,6 +191,8 @@ option('smartcard', type : 'feature', value : 'auto',
        description: 'CA smartcard emulation support')
 option('snappy', type : 'feature', value : 'auto',
        description: 'snappy compression support')
+option('qatzip', type : 'feature', value : 'auto',
+       description: 'qatzip compress support')
 option('spice', type : 'feature', value : 'auto',
        description: 'Spice server support')
 option('spice_protocol', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +3 -0
index 009fab1515..84d110197d 100644
@@ -152,6 +152,7 @@ meson_options_help() {
   printf "%s\n" '  slirp-smbd      use smbd (at path --smbd=*) in slirp networking'
   printf "%s\n" '  smartcard       CA smartcard emulation support'
   printf "%s\n" '  snappy          snappy compression support'
+  printf "%s\n" '  qatzip          qatzip compression support'
   printf "%s\n" '  sndio           sndio sound support'
   printf "%s\n" '  sparse          sparse checker'
   printf "%s\n" '  spice           Spice server support'
@@ -332,6 +333,8 @@ _meson_option_parse() {
     --disable-libvduse) printf "%s" -Dlibvduse=disabled ;;
     --enable-linux-aio) printf "%s" -Dlinux_aio=enabled ;;
     --disable-linux-aio) printf "%s" -Dlinux_aio=disabled ;;
+    --enable-qatzip) printf "%s" -Dqatzip=enabled ;;
+    --disable-qatzip) printf "%s" -Dqatzip=disabled ;;
     --enable-linux-io-uring) printf "%s" -Dlinux_io_uring=enabled ;;
     --disable-linux-io-uring) printf "%s" -Dlinux_io_uring=disabled ;;
     --enable-live-block-migration) printf "%s" -Dlive_block_migration=enabled ;;
Signature · 1 line
2.27.0
author
#3 lore raw

PATCH v2 2/2 migration: add support for qatzip compression when doing live migration

In reply to you.chen

Add config and logics to use qatzip for page compression, in order to support qatzip compression better, we collect multipe pages together to do qatzip compression for best performance. And we use compile option CONFIG_QATZIP to determine whether should qatzip related code be compiled or not.

---
 migration/migration-hmp-cmds.c |   4 +
 migration/migration.c          |  22 ++
 migration/migration.h          |   1 +
 migration/qemu-file.c          |  15 +-
 migration/qemu-file.h          |   5 +
 migration/ram.c                | 675 +++++++++++++++++++++++++++++----
 qapi/migration.json            |   8 +-
 7 files changed, 658 insertions(+), 72 deletions(-)
migration/migration-hmp-cmds.c +4 -0
index 72519ea99f..2608f67469 100644
@@ -479,6 +479,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_compress_threads = true;
         visit_type_uint8(v, param, &p->compress_threads, &err);
         break;
+    case MIGRATION_PARAMETER_COMPRESS_WITH_QAT:
+        p->has_compress_with_qat = true;
+        visit_type_bool(v, param, &p->compress_with_qat, &err);
+        break;
     case MIGRATION_PARAMETER_COMPRESS_WAIT_THREAD:
         p->has_compress_wait_thread = true;
         visit_type_bool(v, param, &p->compress_wait_thread, &err);
migration/migration.c +22 -0
index ae2025d9d8..15ab6ecf57 100644
@@ -962,6 +962,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
     params->compress_level = s->parameters.compress_level;
     params->has_compress_threads = true;
     params->compress_threads = s->parameters.compress_threads;
+    params->has_compress_with_qat = true;
+    params->compress_with_qat = s->parameters.compress_with_qat;
     params->has_compress_wait_thread = true;
     params->compress_wait_thread = s->parameters.compress_wait_thread;
     params->has_decompress_threads = true;
@@ -1678,6 +1680,10 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
         dest->compress_threads = params->compress_threads;
     }
 
+    if (params->has_compress_with_qat) {
+        dest->compress_with_qat = params->compress_with_qat;
+    }
+
     if (params->has_compress_wait_thread) {
         dest->compress_wait_thread = params->compress_wait_thread;
     }
@@ -1775,6 +1781,10 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
         s->parameters.compress_threads = params->compress_threads;
     }
 
+    if (params->has_compress_with_qat) {
+        s->parameters.compress_with_qat = params->compress_with_qat;
+    }
+
     if (params->has_compress_wait_thread) {
         s->parameters.compress_wait_thread = params->compress_wait_thread;
     }
@@ -2620,6 +2630,15 @@ bool migrate_use_compression(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS];
 }
 
+bool migrate_compress_with_qat(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return migrate_use_compression() && s->parameters.compress_with_qat;
+}
+
 int migrate_compress_level(void)
 {
     MigrationState *s;
@@ -4451,6 +4470,8 @@ static Property migration_properties[] = {
     DEFINE_PROP_UINT8("x-compress-threads", MigrationState,
                       parameters.compress_threads,
                       DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT),
+    DEFINE_PROP_BOOL("x-compress-with-qat", MigrationState,
+                      parameters.compress_with_qat, false),
     DEFINE_PROP_BOOL("x-compress-wait-thread", MigrationState,
                       parameters.compress_wait_thread, true),
     DEFINE_PROP_UINT8("x-decompress-threads", MigrationState,
@@ -4580,6 +4601,7 @@ static void migration_instance_init(Object *obj)
     params->has_compress_level = true;
     params->has_compress_threads = true;
     params->has_compress_wait_thread = true;
+    params->has_compress_with_qat = true;
     params->has_decompress_threads = true;
     params->has_throttle_trigger_threshold = true;
     params->has_cpu_throttle_initial = true;
migration/migration.h +1 -0
index 2da2f8a164..fcb5a2f670 100644
@@ -443,6 +443,7 @@ bool migrate_use_return_path(void);
 uint64_t ram_get_total_transferred_pages(void);
 
 bool migrate_use_compression(void);
+bool migrate_compress_with_qat(void);
 int migrate_compress_level(void);
 int migrate_compress_threads(void);
 int migrate_compress_wait_thread(void);
migration/qemu-file.c +13 -2
index 102ab3b439..29d288e82e 100644
@@ -26,12 +26,18 @@
 #include "qemu/madvise.h"
 #include "qemu/error-report.h"
 #include "qemu/iov.h"
+#include "qemu/units.h"
 #include "migration.h"
 #include "qemu-file.h"
 #include "trace.h"
 #include "qapi/error.h"
 
-#define IO_BUF_SIZE 32768
+#ifdef CONFIG_QATZIP
+#define IO_BUF_SIZE (512 * KiB)
+#else
+#define IO_BUF_SIZE (32 * KiB)
+#endif
+
 #define MAX_IOV_SIZE MIN_CONST(IOV_MAX, 64)
 
 struct QEMUFile {
@@ -514,7 +520,7 @@ static int add_to_iovec(QEMUFile *f, const uint8_t *buf, size_t size,
     return 0;
 }
 
-static void add_buf_to_iovec(QEMUFile *f, size_t len)
+void add_buf_to_iovec(QEMUFile *f, size_t len)
 {
     if (!add_to_iovec(f, f->buf + f->buf_index, len, false)) {
         f->buf_index += len;
@@ -524,6 +530,11 @@ static void add_buf_to_iovec(QEMUFile *f, size_t len)
     }
 }
 
+uint8_t *qemu_get_pos(QEMUFile *f)
+{
+    return f->buf + f->buf_index;
+}
+
 void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, size_t size,
                            bool may_free)
 {
migration/qemu-file.h +5 -0
index 9d0155a2a1..391e02f197 100644
@@ -25,6 +25,9 @@
 #ifndef MIGRATION_QEMU_FILE_H
 #define MIGRATION_QEMU_FILE_H
 
+#ifdef CONFIG_QATZIP
+#include <qatzip.h>
+#endif
 #include <zlib.h>
 #include "exec/cpu-common.h"
 #include "io/channel.h"
@@ -108,6 +111,8 @@ bool qemu_file_is_writable(QEMUFile *f);
 
 #include "migration/qemu-file-types.h"
 
+void add_buf_to_iovec(QEMUFile *f, size_t len);
+uint8_t *qemu_get_pos(QEMUFile *f);
 size_t qemu_peek_buffer(QEMUFile *f, uint8_t **buf, size_t size, size_t offset);
 size_t qemu_get_buffer_in_place(QEMUFile *f, uint8_t **buf, size_t size);
 ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
migration/ram.c +606 -69

867 lines not on this page · raw message

qapi/migration.json +7 -1
index c84fa10e86..6459927c7a 100644
@@ -644,6 +644,8 @@
 #                  no compression, 1 means the best compression speed, and 9 means best
 #                  compression ratio which will consume more CPU.
 #
+# @compress-with-qat: compress with qat on and off. (Since 8.1)
+#
 # @compress-threads: Set compression thread count to be used in live migration,
 #                    the compression thread count is an integer between 1 and 255.
 #
@@ -784,7 +786,7 @@
 { 'enum': 'MigrationParameter',
   'data': ['announce-initial', 'announce-max',
            'announce-rounds', 'announce-step',
-           'compress-level', 'compress-threads', 'decompress-threads',
+           'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads',
            'compress-wait-thread', 'throttle-trigger-threshold',
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'cpu-throttle-tailslow',
@@ -815,6 +817,8 @@
 #
 # @compress-level: compression level
 #
+# @compress-with-qat: compression with qat (Since 8.1)
+#
 # @compress-threads: compression thread count
 #
 # @compress-wait-thread: Controls behavior when all compression threads are
@@ -954,6 +958,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',
@@ -1152,6 +1157,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',
Signature · 1 line
2.27.0
#6 lore raw

Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration

Quoted 7 lines · you.chen

Add config and logics to use qatzip for page compression, in order to
support qatzip compression better, we collect multipe pages together
to do qatzip compression for best performance.
And we use compile option CONFIG_QATZIP to determine whether should qatzip related code be compiled or not.

two questions:
a - why are you using this? I guess that because it is faster, but you
are not telling.
b - why are you using it with compression thread instead of multifd
compression? probably just another method.

[...]

Quoted 11 lines
     MigrationState *s;
migration/migration.c +2 -0
@@ -4451,6 +4470,8 @@ static Property migration_properties[] = {
     DEFINE_PROP_UINT8("x-compress-threads", MigrationState,
                       parameters.compress_threads,
                       DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT),
+    DEFINE_PROP_BOOL("x-compress-with-qat", MigrationState,
+                      parameters.compress_with_qat, false),
     DEFINE_PROP_BOOL("x-compress-wait-thread", MigrationState,
                       parameters.compress_wait_thread, true),
     DEFINE_PROP_UINT8("x-decompress-threads", MigrationState,

migration/migration.c static void migration_instance_init(Object *obj) · line 4607

Quoted 7 lines
migration/migration.c +1 -0
@@ -4580,6 +4601,7 @@ static void migration_instance_init(Object *obj)
     params->has_compress_level = true;
     params->has_compress_threads = true;
     params->has_compress_wait_thread = true;
+    params->has_compress_with_qat = true;
     params->has_decompress_threads = true;
     params->has_throttle_trigger_threshold = true;
     params->has_cpu_throttle_initial = true;

If you use it as a parameter, this bits are ok, but I still think that it will work better with a multifd compression method.

migration/qemu-file.c line 39

Quoted 6 lines
migration/qemu-file.c +5 -1
-#define IO_BUF_SIZE 32768
+#ifdef CONFIG_QATZIP
+#define IO_BUF_SIZE (512 * KiB)
+#else
+#define IO_BUF_SIZE (32 * KiB)
+#endif

1st part that it is already weird O:-)
I mean, we are supposed to not have bigger buffers.

migration/qemu-file.c static void add_buf_to_iovec(QEMUFile *f, size_t len) · line 537

Quoted 5 lines
migration/qemu-file.c +5 -0
+uint8_t *qemu_get_pos(QEMUFile *f)
+{
+    return f->buf + f->buf_index;
+}
+

I know I shouldn't ask, but why do you need this.
/me looks later in the patch.

migration/ram.c static void __attribute__((constructor)) init_cpu_flag(void) · line 121

Quoted 8 lines
migration/ram.c +2 -0
@@ -117,6 +117,20 @@ static void __attribute__((constructor)) init_cpu_flag(void)
 
 XBZRLECacheStats xbzrle_counters;
 
+/* define the max page number to compress together */
+#define MULTI_PAGE_NUM 64

You get 64 pages to compress with just using multifd, nothing else needed.

Quoted 45 lines
migration/ram.c +27 -0
+#define COMP_BUF_SIZE (TARGET_PAGE_SIZE *  MULTI_PAGE_NUM * 2)
+#define DECOMP_BUF_SIZE (TARGET_PAGE_SIZE *  MULTI_PAGE_NUM)
+
+typedef struct MultiPageAddr {
+    /* real pages that will compress together */
+    uint64_t pages;
+    /* the last index of the addr*/
+    uint64_t last_idx;
+    /* each address might contain contineous pages*/
+    uint64_t addr[MULTI_PAGE_NUM];
+} MultiPageAddr;
+
 /* used by the search for pages to send */
 struct PageSearchStatus {
     /* The migration channel used for a specific host page */
@@ -127,6 +141,12 @@ struct PageSearchStatus {
     RAMBlock    *block;
     /* Current page to search from */
     unsigned long page;
+    /*
+     * multi page search from current page
+     * for compress together with qatzip
+     * stream APIs
+     */
+    MultiPageAddr mpa;
     /* Set once we wrap around */
     bool         complete_round;
     /* Whether we're sending a host page */
@@ -506,6 +526,15 @@ struct CompressParam {
     /* internally used fields */
     z_stream stream;
     uint8_t *originbuf;
+
+#ifdef CONFIG_QATZIP
+    /*multi page address for compression*/
+    MultiPageAddr mpa;
+    QzSession_T qzsess;
+    uint8_t *decompbuf;
+    uint8_t *compbuf;
+    /* QzStream_T qzstream; */
+#endif
 };
 typedef struct CompressParam CompressParam;
 

migration/ram.c struct DecompressParam { · line 552

Quoted 6 lines
migration/ram.c +3 -0
@@ -518,6 +547,15 @@ struct DecompressParam {
     uint8_t *compbuf;
     int len;
     z_stream stream;
+    RAMBlock *block;
+
+#ifdef CONFIG_QATZIP

Aside. I would expect to detect at configure time if QATZIP is available.

migration/ram.c struct DecompressParam { · line 556

Quoted 4 lines
migration/ram.c +4 -0
+    /* decompress multi pages with qzlib*/
+    QzSession_T qzsess;
+    /* QzStream_T qzstream; */
+    uint8_t *decompbuf; /* buffer after decompress */

The comment is wrong. You uses this variable on do_compress_ram_page, that has nothing to do with decompression.

migration/ram.c struct DecompressParam { · line 559

Quoted 3 lines
migration/ram.c +2 -0
+    MultiPageAddr mpa;   /* destination */
+#endif
 };

I am not going looking after this. We already have a mechanism to look for several pages.

See multifd_queue_page().

We have to options here:
- multifd_queue_page() is better, you need to generalize it and wrote this on top of that.
- yours is better. Then you need to write multifd one on top of that.

What I don't want is two different mechanisms to collect the next 64 dirty pages. I will even consider good that you _always_ search for multiple pages, let say 64 that are the ones that we have now in both multifd and your approach and you do a for loop in case that it is normal precopy.

Another note, I am not a big far of:

#ifdef CONFIG_QATZIP
            if (migrate_compress_with_qat()) {
                do_compress_ram_page_multiple(param->file, &param->qzsess,
                    param->decompbuf, param->compbuf, block, &param->mpa);
            } else {
#endif
                zero_page = do_compress_ram_page(param->file,
                    &param->stream, block, offset, param->originbuf);
                param->zero_page = zero_page;
#ifdef CONFIG_QATZIP
            }

#endif

Two ifdefs and an if mixed, ouch.

Suggestions:
- Just create a function that calls one or another.
- You add another function call to migration_ops, and you add the proper function there during initialization.

Another nitbit:
- You are dropping the zero page optimization, you are just compressing it.
- See how it is done on my zero pages on multifd on the list. a - we only sent the page address for a zero page (hard to win sending zero bytes for a page)
b - I don't know how QzSession_T works, but I guess that it is going
to be a loosy equivalent of how zlib works. Notice that compression code does:

do_compress_ram_page() -> qemu_put_compression_data() -> ...

static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
const uint8_t *source, size_t source_len)

{
    int err;

    err = deflateReset(stream);
    if (err != Z_OK) {
        return -1;
    }

    stream->avail_in = source_len;
    stream->next_in = (uint8_t *)source;
    stream->avail_out = dest_len;
    stream->next_out = dest;

    err = deflate(stream, Z_FINISH);
    if (err != Z_STREAM_END) {
        return -1;
    }

    return stream->next_out - dest;
}

I can't see the equivalent of deflateReset on your code (remember that I don't understand qat). Rememeber that the number of threads for compression and decompression can be different, so it is not necessary that you have the same dicctionaries on destination on the thread that you are using. That is one of the reasons why multifd-zlib is much, much better at compression that compression threads.
a- it uses 64pages at a time
b- as recv thread is paired with the same send thread, we don't need to
reset the dictionaries, and that makes things much, much better. I thought it was a bug the 1st time that I saw that it compressed 64 pages to less than 64bytes O:-)

Later, Juan.

#5 lore raw

Re: [PATCH v2 2/2] migration: add support for qatzip compression when doing live migration

In reply to Markus Armbruster
Quoted 62 lines · Markus Armbruster
Quoted 7 lines · you.chen

Add config and logics to use qatzip for page compression, in order
to support qatzip compression better, we collect multipe pages
together to do qatzip compression for best performance.
And we use compile option CONFIG_QATZIP to determine whether should qatzip related code be compiled or not.

[...]

Quoted 40 lines
qapi/migration.json +6 -1
index c84fa10e86..6459927c7a 100644
@@ -644,6 +644,8 @@
 #                  no compression, 1 means the best compression speed, and 9 means best
 #                  compression ratio which will consume more CPU.
 #
+# @compress-with-qat: compress with qat on and off. (Since 8.1)
+#
 # @compress-threads: Set compression thread count to be used in live migration,
 #                    the compression thread count is an integer between 1 and 255.
 #
@@ -784,7 +786,7 @@
 { 'enum': 'MigrationParameter',
   'data': ['announce-initial', 'announce-max',
            'announce-rounds', 'announce-step',
-           'compress-level', 'compress-threads', 'decompress-threads',
+           'compress-level', 'compress-with-qat', 'compress-threads', 'decompress-threads',
            'compress-wait-thread', 'throttle-trigger-threshold',
            'cpu-throttle-initial', 'cpu-throttle-increment',
            'cpu-throttle-tailslow',
@@ -815,6 +817,8 @@
 #
 # @compress-level: compression level
 #
+# @compress-with-qat: compression with qat (Since 8.1)
+#
 # @compress-threads: compression thread count
 #
 # @compress-wait-thread: Controls behavior when all compression threads are
@@ -954,6 +958,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',

qapi/migration.json line 1163

Quoted 7 lines
qapi/migration.json +1 -0
@@ -1152,6 +1157,7 @@
             '*announce-rounds': 'size',
             '*announce-step': 'size',
             '*compress-level': 'uint8',
+            '*compress-with-qat': 'bool',
             '*compress-threads': 'uint8',
             '*compress-wait-thread': 'bool',
             '*decompress-threads': 'uint8',

We already have MigrationCapability compress

Quoted 7 lines

# @compress: Use multiple compression threads to accelerate live migration.
# This feature can help to reduce the migration traffic, by sending
# compressed pages. Please note that if compress and xbzrle are both
# on, compress only takes effect in the ram bulk stage, after that,
# it will be disabled and only xbzrle takes effect, this can help to
# minimize migration traffic. The feature is disabled by default.
# (since 2.4 )

I had the patch to deprecate it on 8.1.
And now colo is using it. Sniff.

Quoted 6 lines

and xbzrle

# @xbzrle: Migration supports xbzrle (Xor Based Zero Run Length Encoding).
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#

Diferent can of worms, but I agree with you.

Quoted 21 lines

and MigrationParameters / MigrateSetParameters multifd-compression

# @multifd-compression: Which compression method to use.
# Defaults to none. (Since 5.0)
#
# @multifd-zlib-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 9, where 0 means no compression, 1 means the best
# compression speed, and 9 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
# @multifd-zstd-level: Set the compression level to be used in live
# migration, the compression level is an integer between 0
# and 20, where 0 means no compression, 1 means the best
# compression speed, and 20 means best compression ratio which
# will consume more CPU.
# Defaults to 1. (Since 5.0)

where multifd-compression is

Quoted 11 lines
    ##
    # @MultiFDCompression:
    #
    # An enumeration of multifd compression methods.
    #
    # @none: no compression.
    # @zlib: use zlib compression method.
    # @zstd: use zstd compression method.
    #
    # Since: 5.0
    ##

I think it belongs here as another compression method.

Later, Juan.

Quoted 4 lines

How does this all fit together? It feels like a bunch of featured piled onto each other, then shaken well. Or am I confused?

I could use an abstract description of compression in migration.

compression -> old code, uses threads and compress a page at a time
(i.e. takes more time to copy the code to the thread that what we got in return). Data is copied several times.

xbzrle: use a cache of already sent pages and sent the difference.
Not really very useful execpt if you are migrating over data centers. Here you trade memory and cpu consumption on the host for less network bandwidth used.
Current cache size default is a joke, but that is a complete different can of worms.

multifd compression: it is much better (I know that I am the author, but
the whole reason to create it was to address the shortcuts of the old compression code). highlights:
- compress 64 pages at at time, so get much better compression ratios.
- for each channel, it never resets the compression during migration, that means that it compress much better. For the few things that I know about compression, new methods are based heavily in dictionaries, so you need long sessions to get best effectivity.
- it minimizes the number of copies. zstd, no copies at all. zlib -> it makes one copy because hardware implmentations (s390x I am looking at you) make two passes through the data in some cases.

I plan to add xbzre in the near future to multifd compression so I can also deprecate it,

But someone will appears that really needs it O:-)

Later, Juan.