Skip to contentSkip to conversation

Implement using Intel QAT to offload ZLIB

PATCH Superseded Yichen Wang 3 participants 11 messages Jun 27, 2024 – Jul 8, 2024 Arm/Amlogic Meson SoC support

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Implement using Intel QAT to offload ZLIB

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html

* Performance

We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but with two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a more realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         | 23.13         | 8749.94        |117.50   |186.49   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |254.35         |  771.87        |388.20   |144.40   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           | 54.52         | 3442.59        |414.59   |149.77   |
    |---------------|---------------|----------------|---------|---------|
    |none           | 12.45         |43739.60        |159.71   |204.96   |
    |---------------|---------------|----------------|---------|---------|

8 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         | 16.91         |12306.52        |186.37   |391.84   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |130.11         | 1508.89        |753.86   |289.35   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           | 27.57         | 6823.23        |786.83   |303.80   |
    |---------------|---------------|----------------|---------|---------|
    |none           | 11.82         |46072.63        |163.74   |238.56   |
    |---------------|---------------|----------------|---------|---------|

16 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         |18.64          |11044.52        | 573.61  |437.65   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |66.43          | 2955.79        |1469.68  |567.47   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           |14.17          |13290.66        |1504.08  |615.33   |
    |---------------|---------------|----------------|---------|---------|
    |none           |16.82          |32363.26        | 180.74  |217.17   |
    |---------------|---------------|----------------|---------|---------|

** Observations

- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method

 hw/core/qdev-properties-system.c |   6 +-
 meson.build                      |  10 +
 meson_options.txt                |   2 +
 migration/meson.build            |   1 +
 migration/migration-hmp-cmds.c   |   8 +
 migration/multifd-qatzip.c       | 382 +++++++++++++++++++++++++++++++
 migration/multifd.h              |   1 +
 migration/options.c              |  57 +++++
 migration/options.h              |   2 +
 qapi/migration.json              |  38 +++
 scripts/meson-buildoptions.sh    |   6 +
 tests/qtest/meson.build          |   4 +
 tests/qtest/migration-test.c     |  35 +++
 13 files changed, 551 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
Signature · 1 line
Yichen Wang
#1 lore raw

PATCH v3 1/4 meson: Introduce 'qatzip' feature to the build system

From: Bryan Zhang <[email protected]>

Add a 'qatzip' feature, which is automatically disabled, and which depends on the QATzip library if enabled.

---
 meson.build                   | 10 ++++++++++
 meson_options.txt             |  2 ++
 scripts/meson-buildoptions.sh |  6 ++++++
 3 files changed, 18 insertions(+)
meson.build +10 -0
index 97e00d6f59..009f07f506 100644
@@ -1219,6 +1219,14 @@ if not get_option('uadk').auto() or have_system
      uadk = declare_dependency(dependencies: [libwd, libwd_comp])
   endif
 endif
+
+qatzip = not_found
+if get_option('qatzip').enabled()
+  qatzip = dependency('qatzip', version: '>=1.1.2',
+                      required: get_option('qatzip'),
+                      method: 'pkg-config')
+endif
+
 virgl = not_found
 
 have_vhost_user_gpu = have_tools and host_os == 'linux' and pixman.found()
@@ -2353,6 +2361,7 @@ config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id)
 config_host_data.set('CONFIG_ZSTD', zstd.found())
 config_host_data.set('CONFIG_QPL', qpl.found())
 config_host_data.set('CONFIG_UADK', uadk.found())
+config_host_data.set('CONFIG_QATZIP', qatzip.found())
 config_host_data.set('CONFIG_FUSE', fuse.found())
 config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found())
 config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found())
@@ -4468,6 +4477,7 @@ summary_info += {'lzfse support':     liblzfse}
 summary_info += {'zstd support':      zstd}
 summary_info += {'Query Processing Library support': qpl}
 summary_info += {'UADK Library support': uadk}
+summary_info += {'qatzip support':    qatzip}
 summary_info += {'NUMA host support': numa}
 summary_info += {'capstone':          capstone}
 summary_info += {'libpmem support':   libpmem}
meson_options.txt +2 -0
index 7a79dd8970..3670e5058b 100644
@@ -263,6 +263,8 @@ option('qpl', type : 'feature', value : 'auto',
        description: 'Query Processing Library support')
 option('uadk', type : 'feature', value : 'auto',
        description: 'UADK Library support')
+option('qatzip', type: 'feature', value: 'disabled',
+       description: 'QATzip compression support')
 option('fuse', type: 'feature', value: 'auto',
        description: 'FUSE block device export')
 option('fuse_lseek', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +6 -0
index 58d49a447d..226605249e 100644
@@ -162,6 +162,8 @@ meson_options_help() {
   printf "%s\n" '  pixman          pixman support'
   printf "%s\n" '  plugins         TCG plugins via shared library loading'
   printf "%s\n" '  png             PNG support with libpng'
+  printf "%s\n" '  pvrdma          Enable PVRDMA support'
+  printf "%s\n" '  qatzip          QATzip compression support'
   printf "%s\n" '  qcow1           qcow1 image format support'
   printf "%s\n" '  qed             qed image format support'
   printf "%s\n" '  qga-vss         build QGA VSS support (broken with MinGW)'
@@ -428,6 +430,10 @@ _meson_option_parse() {
     --enable-png) printf "%s" -Dpng=enabled ;;
     --disable-png) printf "%s" -Dpng=disabled ;;
     --prefix=*) quote_sh "-Dprefix=$2" ;;
+    --enable-pvrdma) printf "%s" -Dpvrdma=enabled ;;
+    --disable-pvrdma) printf "%s" -Dpvrdma=disabled ;;
+    --enable-qatzip) printf "%s" -Dqatzip=enabled ;;
+    --disable-qatzip) printf "%s" -Dqatzip=disabled ;;
     --enable-qcow1) printf "%s" -Dqcow1=enabled ;;
     --disable-qcow1) printf "%s" -Dqcow1=disabled ;;
     --enable-qed) printf "%s" -Dqed=enabled ;;
Signature · 1 line
Yichen Wang
#2 lore raw

PATCH v3 2/4 migration: Add migration parameters for QATzip

In reply to Yichen Wang

From: Bryan Zhang <[email protected]>

Adds support for migration parameters to control QATzip compression level and to enable/disable software fallback when QAT hardware is unavailable. This is a preparatory commit for a subsequent commit that will actually use QATzip compression.

---
 migration/migration-hmp-cmds.c |  8 +++++
 migration/options.c            | 57 ++++++++++++++++++++++++++++++++++
 migration/options.h            |  2 ++
 qapi/migration.json            | 35 +++++++++++++++++++++
 4 files changed, 102 insertions(+)
migration/migration-hmp-cmds.c +8 -0
index 7d608d26e1..664e2390a3 100644
@@ -576,6 +576,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
         p->has_multifd_zlib_level = true;
         visit_type_uint8(v, param, &p->multifd_zlib_level, &err);
         break;
+    case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL:
+        p->has_multifd_qatzip_level = true;
+        visit_type_uint8(v, param, &p->multifd_qatzip_level, &err);
+        break;
+    case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK:
+        p->has_multifd_qatzip_sw_fallback = true;
+        visit_type_bool(v, param, &p->multifd_qatzip_sw_fallback, &err);
+        break;
     case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL:
         p->has_multifd_zstd_level = true;
         visit_type_uint8(v, param, &p->multifd_zstd_level, &err);
migration/options.c +57 -0

105 lines not on this page · raw message

migration/options.h +2 -0
index a2397026db..24d98c6a29 100644
@@ -78,6 +78,8 @@ uint64_t migrate_max_postcopy_bandwidth(void);
 int migrate_multifd_channels(void);
 MultiFDCompression migrate_multifd_compression(void);
 int migrate_multifd_zlib_level(void);
+int migrate_multifd_qatzip_level(void);
+bool migrate_multifd_qatzip_sw_fallback(void);
 int migrate_multifd_zstd_level(void);
 uint8_t migrate_throttle_trigger_threshold(void);
 const char *migrate_tls_authz(void);
qapi/migration.json +35 -0
index 0f24206bce..8c9f2a8aa7 100644
@@ -789,6 +789,16 @@
 #     speed, and 9 means best compression ratio which will consume
 #     more CPU. Defaults to 1.  (Since 5.0)
 #
+# @multifd-qatzip-level: Set the compression level to be used in live
+#     migration. The level is an integer between 1 and 9, where 1 means
+#     the best compression speed, and 9 means the best compression
+#     ratio which will consume more CPU. Defaults to 1. (Since 9.1)
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+#     is unavailable. Defaults to false. Software fallback performance
+#     is very poor compared to regular zlib, so be cautious about
+#     enabling this option. (Since 9.1)
+#
 # @multifd-zstd-level: Set the compression level to be used in live
 #     migration, the compression level is an integer between 0 and 20,
 #     where 0 means no compression, 1 means the best compression
@@ -849,6 +859,7 @@
            'xbzrle-cache-size', 'max-postcopy-bandwidth',
            'max-cpu-throttle', 'multifd-compression',
            'multifd-zlib-level', 'multifd-zstd-level',
+           'multifd-qatzip-level', 'multifd-qatzip-sw-fallback',
            'block-bitmap-mapping',
            { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] },
            'vcpu-dirty-limit',
@@ -964,6 +975,16 @@
 #     speed, and 9 means best compression ratio which will consume
 #     more CPU. Defaults to 1.  (Since 5.0)
 #
+# @multifd-qatzip-level: Set the compression level to be used in live
+#     migration. The level is an integer between 1 and 9, where 1 means
+#     the best compression speed, and 9 means the best compression
+#     ratio which will consume more CPU. Defaults to 1. (Since 9.1)
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+#     is unavailable. Defaults to false. Software fallback performance
+#     is very poor compared to regular zlib, so be cautious about
+#     enabling this option. (Since 9.1)
+#
 # @multifd-zstd-level: Set the compression level to be used in live
 #     migration, the compression level is an integer between 0 and 20,
 #     where 0 means no compression, 1 means the best compression
@@ -1037,6 +1058,8 @@
             '*max-cpu-throttle': 'uint8',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
+            '*multifd-qatzip-level': 'uint8',
+            '*multifd-qatzip-sw-fallback': 'bool',
             '*multifd-zstd-level': 'uint8',
             '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
@@ -1168,6 +1191,16 @@
 #     speed, and 9 means best compression ratio which will consume
 #     more CPU. Defaults to 1.  (Since 5.0)
 #
+# @multifd-qatzip-level: Set the compression level to be used in live
+#     migration. The level is an integer between 1 and 9, where 1 means
+#     the best compression speed, and 9 means the best compression
+#     ratio which will consume more CPU. Defaults to 1. (Since 9.1)
+#
+# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware
+#     is unavailable. Defaults to false. Software fallback performance
+#     is very poor compared to regular zlib, so be cautious about
+#     enabling this option. (Since 9.1)
+#
 # @multifd-zstd-level: Set the compression level to be used in live
 #     migration, the compression level is an integer between 0 and 20,
 #     where 0 means no compression, 1 means the best compression
@@ -1238,6 +1271,8 @@
             '*max-cpu-throttle': 'uint8',
             '*multifd-compression': 'MultiFDCompression',
             '*multifd-zlib-level': 'uint8',
+            '*multifd-qatzip-level': 'uint8',
+            '*multifd-qatzip-sw-fallback': 'bool',
             '*multifd-zstd-level': 'uint8',
             '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ],
             '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
Signature · 1 line
Yichen Wang
#3 lore raw

PATCH v3 3/4 migration: Introduce 'qatzip' compression method

In reply to Yichen Wang

From: Bryan Zhang <[email protected]>

Adds support for 'qatzip' as an option for the multifd compression method parameter, and implements using QAT for 'qatzip' compression and decompression.

---
 hw/core/qdev-properties-system.c |   6 +-
 migration/meson.build            |   1 +
 migration/multifd-qatzip.c       | 382 +++++++++++++++++++++++++++++++
 migration/multifd.h              |   1 +
 qapi/migration.json              |   3 +
 tests/qtest/meson.build          |   4 +
 6 files changed, 396 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
hw/core/qdev-properties-system.c +5 -1
index f13350b4fb..eb50d6ec5b 100644
@@ -659,7 +659,11 @@ const PropertyInfo qdev_prop_fdc_drive_type = {
 const PropertyInfo qdev_prop_multifd_compression = {
     .name = "MultiFDCompression",
     .description = "multifd_compression values, "
-                   "none/zlib/zstd/qpl/uadk",
+                   "none/zlib/zstd/qpl/uadk"
+#ifdef CONFIG_QATZIP
+                   "/qatzip"
+#endif
+                   ,
     .enum_table = &MultiFDCompression_lookup,
     .get = qdev_propinfo_get_enum,
     .set = qdev_propinfo_set_enum,
migration/meson.build +1 -0
index 5ce2acb41e..c9454c26ae 100644
@@ -41,6 +41,7 @@ system_ss.add(when: rdma, if_true: files('rdma.c'))
 system_ss.add(when: zstd, if_true: files('multifd-zstd.c'))
 system_ss.add(when: qpl, if_true: files('multifd-qpl.c'))
 system_ss.add(when: uadk, if_true: files('multifd-uadk.c'))
+system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c'))
 
 specific_ss.add(when: 'CONFIG_SYSTEM_ONLY',
                 if_true: files('ram.c',
migration/multifd-qatzip.c new file +382 -0

382 lines not on this page · raw message

migration/multifd.h +1 -0
index 0ecd6f47d7..2a3b904675 100644
@@ -40,6 +40,7 @@ MultiFDRecvData *multifd_get_recv_data(void);
 #define MULTIFD_FLAG_NOCOMP (0 << 1)
 #define MULTIFD_FLAG_ZLIB (1 << 1)
 #define MULTIFD_FLAG_ZSTD (2 << 1)
+#define MULTIFD_FLAG_QATZIP (3 << 1)
 #define MULTIFD_FLAG_QPL (4 << 1)
 #define MULTIFD_FLAG_UADK (8 << 1)
 
qapi/migration.json +3 -0
index 8c9f2a8aa7..ea62f983b1 100644
@@ -558,6 +558,8 @@
 #
 # @zstd: use zstd compression method.
 #
+# @qatzip: use qatzip compression method. (Since 9.1)
+#
 # @qpl: use qpl compression method.  Query Processing Library(qpl) is
 #       based on the deflate compression algorithm and use the Intel
 #       In-Memory Analytics Accelerator(IAA) accelerated compression
@@ -570,6 +572,7 @@
 { 'enum': 'MultiFDCompression',
   'data': [ 'none', 'zlib',
             { 'name': 'zstd', 'if': 'CONFIG_ZSTD' },
+            { 'name': 'qatzip', 'if': 'CONFIG_QATZIP'},
             { 'name': 'qpl', 'if': 'CONFIG_QPL' },
             { 'name': 'uadk', 'if': 'CONFIG_UADK' } ] }
 
tests/qtest/meson.build +4 -0
index 12792948ff..23e46144d7 100644
@@ -324,6 +324,10 @@ if gnutls.found()
   endif
 endif
 
+if qatzip.found()
+  migration_files += [qatzip]
+endif
+
 qtests = {
   'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'],
   'cdrom-test': files('boot-sector.c'),
Signature · 1 line
Yichen Wang
#4 lore raw

PATCH v3 4/4 tests/migration: Add integration test for 'qatzip' compression method

In reply to Yichen Wang

From: Bryan Zhang <[email protected]>

Adds an integration test for 'qatzip'.

---
 tests/qtest/migration-test.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
tests/qtest/migration-test.c +35 -0
index 571fc1334c..cc4a971e63 100644
@@ -32,6 +32,10 @@
 # endif /* CONFIG_TASN1 */
 #endif /* CONFIG_GNUTLS */
 
+#ifdef CONFIG_QATZIP
+#include <qatzip.h>
+#endif /* CONFIG_QATZIP */
+
 /* For dirty ring test; so far only x86_64 is supported */
 #if defined(__linux__) && defined(HOST_X86_64)
 #include "linux/kvm.h"
@@ -2992,6 +2996,22 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestState *from,
 }
 #endif /* CONFIG_ZSTD */
 
+#ifdef CONFIG_QATZIP
+static void *
+test_migrate_precopy_tcp_multifd_qatzip_start(QTestState *from,
+                                              QTestState *to)
+{
+    migrate_set_parameter_int(from, "multifd-qatzip-level", 2);
+    migrate_set_parameter_int(to, "multifd-qatzip-level", 2);
+
+    /* SW fallback is disabled by default, so enable it for testing. */
+    migrate_set_parameter_bool(from, "multifd-qatzip-sw-fallback", true);
+    migrate_set_parameter_bool(to, "multifd-qatzip-sw-fallback", true);
+
+    return test_migrate_precopy_tcp_multifd_start_common(from, to, "qatzip");
+}
+#endif
+
 #ifdef CONFIG_QPL
 static void *
 test_migrate_precopy_tcp_multifd_qpl_start(QTestState *from,
@@ -3089,6 +3109,17 @@ static void test_multifd_tcp_zstd(void)
 }
 #endif
 
+#ifdef CONFIG_QATZIP
+static void test_multifd_tcp_qatzip(void)
+{
+    MigrateCommon args = {
+        .listen_uri = "defer",
+        .start_hook = test_migrate_precopy_tcp_multifd_qatzip_start,
+    };
+    test_precopy_common(&args);
+}
+#endif
+
 #ifdef CONFIG_QPL
 static void test_multifd_tcp_qpl(void)
 {
@@ -4002,6 +4033,10 @@ int main(int argc, char **argv)
     migration_test_add("/migration/multifd/tcp/plain/zstd",
                        test_multifd_tcp_zstd);
 #endif
+#ifdef CONFIG_QATZIP
+    migration_test_add("/migration/multifd/tcp/plain/qatzip",
+                test_multifd_tcp_qatzip);
+#endif
 #ifdef CONFIG_QPL
     migration_test_add("/migration/multifd/tcp/plain/qpl",
                        test_multifd_tcp_qpl);
Signature · 1 line
Yichen Wang
#5 lore raw
In reply to Yichen Wang
Quoted 94 lines · Yichen Wang

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html

* Performance

We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but with two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a more realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         | 23.13         | 8749.94        |117.50   |186.49   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |254.35         |  771.87        |388.20   |144.40   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           | 54.52         | 3442.59        |414.59   |149.77   |
    |---------------|---------------|----------------|---------|---------|
    |none           | 12.45         |43739.60        |159.71   |204.96   |
    |---------------|---------------|----------------|---------|---------|

8 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         | 16.91         |12306.52        |186.37   |391.84   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |130.11         | 1508.89        |753.86   |289.35   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           | 27.57         | 6823.23        |786.83   |303.80   |
    |---------------|---------------|----------------|---------|---------|
    |none           | 11.82         |46072.63        |163.74   |238.56   |
    |---------------|---------------|----------------|---------|---------|

16 multifd threads:

    |---------------|---------------|----------------|---------|---------|
    |method         |time(sec)      |throughput(mbps)|send cpu%|recv cpu%|
    |---------------|---------------|----------------|---------|---------|
    |qatzip         |18.64          |11044.52        | 573.61  |437.65   |
    |---------------|---------------|----------------|---------|---------|
    |zlib           |66.43          | 2955.79        |1469.68  |567.47   |
    |---------------|---------------|----------------|---------|---------|
    |zstd           |14.17          |13290.66        |1504.08  |615.33   |
    |---------------|---------------|----------------|---------|---------|
    |none           |16.82          |32363.26        | 180.74  |217.17   |
    |---------------|---------------|----------------|---------|---------|

** Observations

Quoted 14 lines

- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.

Thanks for the results update.

It looks like the docs/migration/ file is still missing. It'll be great to have it in the next version or separately.

So how it compares with QPL (which got merged already)? They at least look like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.

I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even if the work should have been offloaded to the QAT chips iiuc.

I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about it outperforms others).

Thanks,

Quoted 27 lines

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method

 hw/core/qdev-properties-system.c |   6 +-
 meson.build                      |  10 +
 meson_options.txt                |   2 +
 migration/meson.build            |   1 +
 migration/migration-hmp-cmds.c   |   8 +
 migration/multifd-qatzip.c       | 382 +++++++++++++++++++++++++++++++
 migration/multifd.h              |   1 +
 migration/options.c              |  57 +++++
 migration/options.h              |   2 +
 qapi/migration.json              |  38 +++
 scripts/meson-buildoptions.sh    |   6 +
 tests/qtest/meson.build          |   4 +
 tests/qtest/migration-test.c     |  35 +++
 13 files changed, 551 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
Signature · 1 line
Yichen Wang
Signature · 1 line
Peter Xu
#6 lore raw
Quoted 165 lines

-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 28 lines · Yichen Wang

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html

* Performance

We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but with two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.

Quoted 6 lines

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a

more

Quoted 11 lines

realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).

Quoted 8 lines

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         | 23.13         | 8749.94        |117.50   |186.49

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |254.35         |  771.87        |388.20   |144.40

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           | 54.52         | 3442.59        |414.59   |149.77

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           | 12.45         |43739.60        |159.71   |204.96

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 4 lines

8 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         | 16.91         |12306.52        |186.37   |391.84

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |130.11         | 1508.89        |753.86   |289.35

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           | 27.57         | 6823.23        |786.83   |303.80

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           | 11.82         |46072.63        |163.74   |238.56

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 4 lines

16 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         |18.64          |11044.52        | 573.61  |437.65

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |66.43          | 2955.79        |1469.68  |567.47

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           |14.17          |13290.66        |1504.08  |615.33

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           |16.82          |32363.26        | 180.74  |217.17

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 17 lines

** Observations

- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.

Thanks for the results update.

It looks like the docs/migration/ file is still missing. It'll be great to
have it in the next version or separately.

Quoted 6 lines

So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.

I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.

1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA, different SKUs support different numbers of QAT and IAA, so some users do not have both QAT and IAA at the same time.

2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already used QAT cards to accelerate live migration.

3. In addition to compression, QAT and IAA also support various other features to better serve different workloads. Here is an introduction to the accelerators, including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf

For users who have both QAT and IAA, we recommend the following for choosing a live migration solution:

1. If the number of QAT devices is equal to or greater than the number of IAA devices and network bandwidth is limited, it is recommended to use the QATZip(QAT) solution.

2. In other scenarios, the QPL (IAA) solution can be used first.

Quoted 45 lines

I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.

I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about
it outperforms others).

Thanks,

Quoted 24 lines

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method

 hw/core/qdev-properties-system.c |   6 +-
 meson.build                      |  10 +
 meson_options.txt                |   2 +
 migration/meson.build            |   1 +
 migration/migration-hmp-cmds.c   |   8 +
 migration/multifd-qatzip.c       | 382 +++++++++++++++++++++++++++++++
 migration/multifd.h              |   1 +
 migration/options.c              |  57 +++++
 migration/options.h              |   2 +
 qapi/migration.json              |  38 +++
 scripts/meson-buildoptions.sh    |   6 +
 tests/qtest/meson.build          |   4 +
 tests/qtest/migration-test.c     |  35 +++
 13 files changed, 551 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
Quoted 3 lines

--
Yichen Wang

--
Peter Xu

#7 lore raw
Quoted 165 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 165 lines

-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 28 lines · Yichen Wang

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html

* Performance

We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but with two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.

Quoted 6 lines

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a

more

Quoted 11 lines

realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).

Quoted 8 lines

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         | 23.13         | 8749.94        |117.50   |186.49

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |254.35         |  771.87        |388.20   |144.40

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           | 54.52         | 3442.59        |414.59   |149.77

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           | 12.45         |43739.60        |159.71   |204.96

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 4 lines

8 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         | 16.91         |12306.52        |186.37   |391.84

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |130.11         | 1508.89        |753.86   |289.35

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           | 27.57         | 6823.23        |786.83   |303.80

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           | 11.82         |46072.63        |163.74   |238.56

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 4 lines

16 multifd threads:

    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |qatzip         |18.64          |11044.52        | 573.61  |437.65

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zlib           |66.43          | 2955.79        |1469.68  |567.47

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |zstd           |14.17          |13290.66        |1504.08  |615.33

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 1 line
    |none           |16.82          |32363.26        | 180.74  |217.17

|

Quoted 1 line
    |---------------|---------------|----------------|---------|--------

-|

Quoted 17 lines

** Observations

- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.

Thanks for the results update.

It looks like the docs/migration/ file is still missing. It'll be great to
have it in the next version or separately.

Quoted 6 lines
Quoted 6 lines

So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.

Yuan,

Quoted 3 lines

I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.

Yes, these are very helpful information. Please consider putting them into the cover letter if there's a repost, and perhaps also in the doc/ files.

Quoted 8 lines

1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA, different SKUs support different numbers of QAT and IAA, so some users do not have both QAT and IAA at the same time.

2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already used QAT cards to accelerate live migration.

Ah, this makes some sense to me.

So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.

So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?

If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper would be nice too.

Quoted 5 lines

3. In addition to compression, QAT and IAA also support various other features to better serve different workloads. Here is an introduction to the accelerators, including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf

Thanks for the link.

However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.

Two major (but pure..) questions:

1) Why high cpu usage?

I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?

The paper you provided above has this upfront:

When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...

Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?

2) TLS?

I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS (instead of compression) with the QATs?

Thanks,

Quoted 15 lines

For users who have both QAT and IAA, we recommend the following for choosing a live migration solution:

1. If the number of QAT devices is equal to or greater than the number of IAA devices and network bandwidth is limited, it is recommended to use the QATZip(QAT) solution.

2. In other scenarios, the QPL (IAA) solution can be used first.

Quoted 6 lines

I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.

[1]

Quoted 40 lines
Quoted 39 lines

I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about
it outperforms others).

Thanks,

Quoted 24 lines

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method

 hw/core/qdev-properties-system.c |   6 +-
 meson.build                      |  10 +
 meson_options.txt                |   2 +
 migration/meson.build            |   1 +
 migration/migration-hmp-cmds.c   |   8 +
 migration/multifd-qatzip.c       | 382 +++++++++++++++++++++++++++++++
 migration/multifd.h              |   1 +
 migration/options.c              |  57 +++++
 migration/options.h              |   2 +
 qapi/migration.json              |  38 +++
 scripts/meson-buildoptions.sh    |   6 +
 tests/qtest/meson.build          |   4 +
 tests/qtest/migration-test.c     |  35 +++
 13 files changed, 551 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
Quoted 3 lines

--
Yichen Wang

--
Peter Xu

Signature · 1 line
Peter Xu
#8 lore raw
Quoted 269 lines

-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Thursday, July 4, 2024 11:36 PM
To: Liu, Yuan1 <[email protected]>
Cc: Wang, Yichen <[email protected]>; Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; [email protected]; Hao Xiang <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 6 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 6 lines

-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-

André

Quoted 6 lines
Quoted 6 lines

Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren

(Jack)

Quoted 16 lines
Quoted 16 lines

Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 12 lines · Yichen Wang

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-

12/msg03761.html

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

* Performance

We present updated performance results. For circumstantial reasons,

v1

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but

with

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2

of

Quoted 1 line
Quoted 1 line
Quoted 1 line

the VM's RAM. Now we properly allocate and fill nearly all of the

VM's

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

RAM.

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB

memory.

Quoted 1 line
Quoted 1 line
Quoted 1 line

We prepare the machine's memory by powering it on, allocating a

large

Quoted 1 line
Quoted 1 line
Quoted 1 line

amount of memory (60GB) as a single buffer, and filling the buffer

with

Quoted 10 lines
Quoted 10 lines
Quoted 1 line

the repeated contents of the Silesia corpus[0]. This is in lieu of a

more

Quoted 8 lines

realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

compression methods, with varying numbers of multifd threads (4, 8,

and

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

16).

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

    |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |qatzip         | 23.13         | 8749.94        |117.50

|186.49

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zlib           |254.35         |  771.87        |388.20

|144.40

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zstd           | 54.52         | 3442.59        |414.59

|149.77

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |none           | 12.45         |43739.60        |159.71

|204.96

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

8 multifd threads:

    |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |qatzip         | 16.91         |12306.52        |186.37

|391.84

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zlib           |130.11         | 1508.89        |753.86

|289.35

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zstd           | 27.57         | 6823.23        |786.83

|303.80

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |none           | 11.82         |46072.63        |163.74

|238.56

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

16 multifd threads:

    |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |qatzip         |18.64          |11044.52        | 573.61

|437.65

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zlib           |66.43          | 2955.79        |1469.68

|567.47

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |zstd           |14.17          |13290.66        |1504.08

|615.33

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
    |none           |16.82          |32363.26        | 180.74

|217.17

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
    |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

** Observations

- In general, not using compression outperforms using compression in

a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8

workers,

Quoted 1 line
Quoted 1 line
Quoted 1 line

achieving a ~91% latency reduction over 'zlib' with 4 workers, and

a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16

workers,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is

highly

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

parallelizable.
- 'qatzip' compression uses considerably less CPU than other

compression

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16

workers.

Quoted 4 lines
Quoted 4 lines

Thanks for the results update.

It looks like the docs/migration/ file is still missing. It'll be

great

Quoted 7 lines
Quoted 7 lines

to
have it in the next version or separately.

So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on

why

Quoted 1 line
Quoted 1 line

Intel provides these two similar compressors. It would be great to

have

Quoted 1 line
Quoted 1 line

some answer and perhaps put into the doc.

Yuan,

Quoted 3 lines

I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.

Yes, these are very helpful information. Please consider putting them into
the cover letter if there's a repost, and perhaps also in the doc/ files.

Quoted 2 lines

1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA,

different

Quoted 1 line

SKUs support different numbers of QAT and IAA, so some users do not

have

Quoted 4 lines

both QAT and IAA at the same time.

2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already

used QAT

Quoted 1 line

cards to accelerate live migration.

Ah, this makes some sense to me.

So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.

Quoted 4 lines

So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?

Yes, users do not add a QAT card just for live migration. Users mainly use QAT-SRIOV technology to help cloud users offload compression and encryption.

Quoted 8 lines

If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if
there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper
would be nice too.

Yes, I will provide some performance data at some specific bandwidths(100Mbps/1Gbps/10Gbps). And add documentation to explain the advantages of using QAT

Quoted 23 lines
Quoted 2 lines

3. In addition to compression, QAT and IAA also support various other

features

Quoted 1 line

to better serve different workloads. Here is an introduction to the

accelerators,

Quoted 2 lines

including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-

libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf

Thanks for the link.

However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.

Two major (but pure..) questions:

1) Why high cpu usage?

I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?

Quoted 7 lines

The paper you provided above has this upfront:

When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...

Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?

Yes, it doesn't make sense, I will check this.

Quoted 6 lines

2) TLS?

I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS
(instead
of compression) with the QATs?

I'm sorry for not responding to the previous question about TLS. QAT has many related success cases (for example, OpenSSL).
https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/accelerating-openssl-brief.pdf

I will send a separate RFC or patch about this part because the software stacks of QAT compression and encryption are independent, so we discuss them separately.

Quoted 72 lines

Thanks,

Quoted 2 lines

For users who have both QAT and IAA, we recommend the following for

choosing a

Quoted 3 lines

live migration solution:

1. If the number of QAT devices is equal to or greater than the number

of IAA devices

Quoted 1 line

and network bandwidth is limited, it is recommended to use the

QATZip(QAT) solution.

Quoted 5 lines

2. In other scenarios, the QPL (IAA) solution can be used first.

Quoted 2 lines

I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp

is

Quoted 1 line
Quoted 1 line

mostly always better than qatzip, no matter on total migration time or

cpu

Quoted 3 lines
Quoted 3 lines

consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.

[1]

Quoted 2 lines
Quoted 2 lines

I think it may not be a problem to merge this series even if it

performs

Quoted 1 line
Quoted 1 line

slower at some criterias.. but I think we may still want to know when

this

Quoted 8 lines
Quoted 8 lines

should be used, or the good reason this should be merged (if it's not about
it outperforms others).

Thanks,

Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression

method

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
 hw/core/qdev-properties-system.c |   6 +-
 meson.build                      |  10 +
 meson_options.txt                |   2 +
 migration/meson.build            |   1 +
 migration/migration-hmp-cmds.c   |   8 +
 migration/multifd-qatzip.c       | 382

+++++++++++++++++++++++++++++++

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
 migration/multifd.h              |   1 +
 migration/options.c              |  57 +++++
 migration/options.h              |   2 +
 qapi/migration.json              |  38 +++
 scripts/meson-buildoptions.sh    |   6 +
 tests/qtest/meson.build          |   4 +
 tests/qtest/migration-test.c     |  35 +++
 13 files changed, 551 insertions(+), 1 deletion(-)
 create mode 100644 migration/multifd-qatzip.c
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

--
Yichen Wang

--
Peter Xu

--
Peter Xu

#9 lore raw
Quoted 290 lines · On Jul 5, 2024, at 1:32 AM, Liu, Yuan1 <[email protected]> wrote:
Quoted 270 lines

-----Original Message-----
From: Peter Xu <[email protected] <mailto:[email protected]>>
Sent: Thursday, July 4, 2024 11:36 PM
To: Liu, Yuan1 <[email protected] <mailto:[email protected]>>
Cc: Wang, Yichen <[email protected] <mailto:[email protected]>>; Paolo Bonzini <[email protected] <mailto:[email protected]>>; Daniel P. Berrangé <[email protected] <mailto:[email protected]>>; Eduardo Habkost <[email protected] <mailto:[email protected]>>; Marc-André Lureau
<[email protected] <mailto:[email protected]>>; Thomas Huth <[email protected] <mailto:[email protected]>>; Philippe Mathieu-Daudé <[email protected] <mailto:[email protected]>>; Fabiano Rosas <[email protected] <mailto:[email protected]>>; Eric Blake <[email protected] <mailto:[email protected]>>; Markus Armbruster <[email protected] <mailto:[email protected]>>; Laurent Vivier <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]>; Hao Xiang <[email protected] <mailto:[email protected]>>; Zou, Nanhai <[email protected] <mailto:[email protected]>>; Ho-Ren (Jack) Chuang <[email protected] <mailto:[email protected]>>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 6 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 6 lines

-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-

André

Quoted 6 lines
Quoted 6 lines

Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren

(Jack)

Quoted 16 lines
Quoted 16 lines

Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIB

Quoted 12 lines · Yichen Wang

v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations

v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-

12/msg03761.html

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

* Performance

We present updated performance results. For circumstantial reasons,

v1

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

presented performance on a low-bandwidth (1Gbps) network.

Here, we present updated results with a similar setup as before but

with

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

two main differences:

1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2

of

Quoted 1 line
Quoted 1 line
Quoted 1 line

the VM's RAM. Now we properly allocate and fill nearly all of the

VM's

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

RAM.

Thus, the test setup is as follows:

We perform multifd live migration over TCP using a VM with 64GB

memory.

Quoted 1 line
Quoted 1 line
Quoted 1 line

We prepare the machine's memory by powering it on, allocating a

large

Quoted 1 line
Quoted 1 line
Quoted 1 line

amount of memory (60GB) as a single buffer, and filling the buffer

with

Quoted 10 lines
Quoted 10 lines
Quoted 1 line

the repeated contents of the Silesia corpus[0]. This is in lieu of a

more

Quoted 8 lines

realistic memory snapshot, which proved troublesome to acquire.

We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.

We present the latency, throughput, and CPU usage results for all of

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

compression methods, with varying numbers of multifd threads (4, 8,

and

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

16).

[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia

** Results

4 multifd threads:

   |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |qatzip         | 23.13         | 8749.94        |117.50

|186.49

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zlib           |254.35         |  771.87        |388.20

|144.40

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zstd           | 54.52         | 3442.59        |414.59

|149.77

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |none           | 12.45         |43739.60        |159.71

|204.96

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

8 multifd threads:

   |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |qatzip         | 16.91         |12306.52        |186.37

|391.84

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zlib           |130.11         | 1508.89        |753.86

|289.35

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zstd           | 27.57         | 6823.23        |786.83

|303.80

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |none           | 11.82         |46072.63        |163.74

|238.56

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

16 multifd threads:

   |---------------|---------------|----------------|---------|----

----

Quoted 4 lines
Quoted 4 lines

-|

Quoted 1 line

|method |time(sec) |throughput(mbps)|send cpu%|recv

cpu%|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |qatzip         |18.64          |11044.52        | 573.61

|437.65

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zlib           |66.43          | 2955.79        |1469.68

|567.47

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |zstd           |14.17          |13290.66        |1504.08

|615.33

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 2 lines
Quoted 2 lines

-|

Quoted 1 line
   |none           |16.82          |32363.26        | 180.74

|217.17

Quoted 2 lines
Quoted 2 lines

|

Quoted 1 line
   |---------------|---------------|----------------|---------|----

----

Quoted 5 lines
Quoted 5 lines

-|

Quoted 4 lines

** Observations

- In general, not using compression outperforms using compression in

a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8

workers,

Quoted 1 line
Quoted 1 line
Quoted 1 line

achieving a ~91% latency reduction over 'zlib' with 4 workers, and

a

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16

workers,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is

highly

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

parallelizable.
- 'qatzip' compression uses considerably less CPU than other

compression

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16

workers.

Quoted 4 lines
Quoted 4 lines

Thanks for the results update.

It looks like the docs/migration/ file is still missing. It'll be

great

Quoted 7 lines
Quoted 7 lines

to
have it in the next version or separately.

So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on

why

Quoted 1 line
Quoted 1 line

Intel provides these two similar compressors. It would be great to

have

Quoted 1 line
Quoted 1 line

some answer and perhaps put into the doc.

Yuan,

Quoted 3 lines

I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.

Yes, these are very helpful information. Please consider putting them into
the cover letter if there's a repost, and perhaps also in the doc/ files.

Quoted 2 lines

1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA,

different

Quoted 1 line

SKUs support different numbers of QAT and IAA, so some users do not

have

Quoted 4 lines

both QAT and IAA at the same time.

2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already

used QAT

Quoted 1 line

cards to accelerate live migration.

Ah, this makes some sense to me.

So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.

Quoted 4 lines

So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?

Yes, users do not add a QAT card just for live migration. Users mainly use QAT-SRIOV technology to help cloud users offload compression and encryption.

Quoted 8 lines

If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if
there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper
would be nice too.

Quoted 3 lines

Yes, I will provide some performance data at some specific bandwidths(100Mbps/1Gbps/10Gbps). And add documentation to explain the advantages of using QAT

Just want to add some information here. So in ByteDance, the current generation server is quipped with 2*100Gb NIC. We reserve 10Gbps for control plane purposes which includes live migration here. So it is not about we are using “good network”, it is about not normal to use full bandwidth for control plane purposes. Hence we do have a requirements for QAT/IAA in these cases.

Quoted 119 lines
Quoted 23 lines
Quoted 2 lines

3. In addition to compression, QAT and IAA also support various other

features

Quoted 1 line

to better serve different workloads. Here is an introduction to the

accelerators,

Quoted 2 lines

including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-

libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf

Thanks for the link.

However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.

Two major (but pure..) questions:

1) Why high cpu usage?

I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?

Quoted 7 lines

The paper you provided above has this upfront:

When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...

Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?

Yes, it doesn't make sense, I will check this.

Quoted 6 lines

2) TLS?

I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS
(instead
of compression) with the QATs?

I'm sorry for not responding to the previous question about TLS. QAT has many related success cases (for example, OpenSSL).
https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/accelerating-openssl-brief.pdf

I will send a separate RFC or patch about this part because the software stacks of QAT compression and encryption are independent, so we discuss them separately.

Quoted 72 lines

Thanks,

Quoted 2 lines

For users who have both QAT and IAA, we recommend the following for

choosing a

Quoted 3 lines

live migration solution:

1. If the number of QAT devices is equal to or greater than the number

of IAA devices

Quoted 1 line

and network bandwidth is limited, it is recommended to use the

QATZip(QAT) solution.

Quoted 5 lines

2. In other scenarios, the QPL (IAA) solution can be used first.

Quoted 2 lines

I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp

is

Quoted 1 line
Quoted 1 line

mostly always better than qatzip, no matter on total migration time or

cpu

Quoted 3 lines
Quoted 3 lines

consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.

[1]

Quoted 2 lines
Quoted 2 lines

I think it may not be a problem to merge this series even if it

performs

Quoted 1 line
Quoted 1 line

slower at some criterias.. but I think we may still want to know when

this

Quoted 8 lines
Quoted 8 lines

should be used, or the good reason this should be merged (if it's not about
it outperforms others).

Thanks,

Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression

method

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/core/qdev-properties-system.c |   6 +-
meson.build                      |  10 +
meson_options.txt                |   2 +
migration/meson.build            |   1 +
migration/migration-hmp-cmds.c   |   8 +
migration/multifd-qatzip.c       | 382

+++++++++++++++++++++++++++++++

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
migration/multifd.h              |   1 +
migration/options.c              |  57 +++++
migration/options.h              |   2 +
qapi/migration.json              |  38 +++
scripts/meson-buildoptions.sh    |   6 +
tests/qtest/meson.build          |   4 +
tests/qtest/migration-test.c     |  35 +++
13 files changed, 551 insertions(+), 1 deletion(-)

create mode 100644 migration/multifd-qatzip.c

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

--
Yichen Wang

--
Peter Xu

--
Peter Xu

#10 lore raw
Quoted 6 lines · Yichen Wang

Just want to add some information here. So in ByteDance, the current generation server is quipped with 2*100Gb NIC. We reserve 10Gbps for control plane purposes which includes live migration here. So it is not about we are using “good network”, it is about not normal to use full bandwidth for control plane purposes. Hence we do have a requirements for QAT/IAA in these cases.

Yes, this makes sense.

But then you may then also want to figure out the high cpu consumption of those cards, to not interrupt more important workloads?

I saw there's a new version posted, I didn't see an explanation of the cpu consumption issue mentioned. Meanwhile I also see that the docs/ update is missing.

Would you consider adding both by replying to the new version?

Signature · 1 line
Peter Xu