Implement using Intel QAT to offload ZLIB
v3 cover Implement using Intel QAT to offload ZLIB
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations
v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.
v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html
* Performance
We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but with two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a more realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia
** Results
4 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip | 23.13 | 8749.94 |117.50 |186.49 |
|---------------|---------------|----------------|---------|---------|
|zlib |254.35 | 771.87 |388.20 |144.40 |
|---------------|---------------|----------------|---------|---------|
|zstd | 54.52 | 3442.59 |414.59 |149.77 |
|---------------|---------------|----------------|---------|---------|
|none | 12.45 |43739.60 |159.71 |204.96 |
|---------------|---------------|----------------|---------|---------|8 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip | 16.91 |12306.52 |186.37 |391.84 |
|---------------|---------------|----------------|---------|---------|
|zlib |130.11 | 1508.89 |753.86 |289.35 |
|---------------|---------------|----------------|---------|---------|
|zstd | 27.57 | 6823.23 |786.83 |303.80 |
|---------------|---------------|----------------|---------|---------|
|none | 11.82 |46072.63 |163.74 |238.56 |
|---------------|---------------|----------------|---------|---------|16 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip |18.64 |11044.52 | 573.61 |437.65 |
|---------------|---------------|----------------|---------|---------|
|zlib |66.43 | 2955.79 |1469.68 |567.47 |
|---------------|---------------|----------------|---------|---------|
|zstd |14.17 |13290.66 |1504.08 |615.33 |
|---------------|---------------|----------------|---------|---------|
|none |16.82 |32363.26 | 180.74 |217.17 |
|---------------|---------------|----------------|---------|---------|** Observations
- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method
hw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++ migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.c
Signature · 1 line
Yichen Wang
PATCH v3 1/4 meson: Introduce 'qatzip' feature to the build system
From: Bryan Zhang <[email protected]>
Add a 'qatzip' feature, which is automatically disabled, and which depends on the QATzip library if enabled.
--- meson.build | 10 ++++++++++ meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 6 ++++++ 3 files changed, 18 insertions(+)
meson.build +10 -0
@@ -1219,6 +1219,14 @@ if not get_option('uadk').auto() or have_system uadk = declare_dependency(dependencies: [libwd, libwd_comp]) endif endif + +qatzip = not_found +if get_option('qatzip').enabled() + qatzip = dependency('qatzip', version: '>=1.1.2', + required: get_option('qatzip'), + method: 'pkg-config') +endif + virgl = not_found have_vhost_user_gpu = have_tools and host_os == 'linux' and pixman.found()
@@ -2353,6 +2361,7 @@ config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id) config_host_data.set('CONFIG_ZSTD', zstd.found()) config_host_data.set('CONFIG_QPL', qpl.found()) config_host_data.set('CONFIG_UADK', uadk.found()) +config_host_data.set('CONFIG_QATZIP', qatzip.found()) config_host_data.set('CONFIG_FUSE', fuse.found()) config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found()) config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found())
@@ -4468,6 +4477,7 @@ summary_info += {'lzfse support': liblzfse} summary_info += {'zstd support': zstd} summary_info += {'Query Processing Library support': qpl} summary_info += {'UADK Library support': uadk} +summary_info += {'qatzip support': qatzip} summary_info += {'NUMA host support': numa} summary_info += {'capstone': capstone} summary_info += {'libpmem support': libpmem}
meson_options.txt +2 -0
@@ -263,6 +263,8 @@ option('qpl', type : 'feature', value : 'auto', description: 'Query Processing Library support') option('uadk', type : 'feature', value : 'auto', description: 'UADK Library support') +option('qatzip', type: 'feature', value: 'disabled', + description: 'QATzip compression support') option('fuse', type: 'feature', value: 'auto', description: 'FUSE block device export') option('fuse_lseek', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +6 -0
@@ -162,6 +162,8 @@ meson_options_help() { printf "%s\n" ' pixman pixman support' printf "%s\n" ' plugins TCG plugins via shared library loading' printf "%s\n" ' png PNG support with libpng' + printf "%s\n" ' pvrdma Enable PVRDMA support' + printf "%s\n" ' qatzip QATzip compression support' printf "%s\n" ' qcow1 qcow1 image format support' printf "%s\n" ' qed qed image format support' printf "%s\n" ' qga-vss build QGA VSS support (broken with MinGW)'
@@ -428,6 +430,10 @@ _meson_option_parse() { --enable-png) printf "%s" -Dpng=enabled ;; --disable-png) printf "%s" -Dpng=disabled ;; --prefix=*) quote_sh "-Dprefix=$2" ;; + --enable-pvrdma) printf "%s" -Dpvrdma=enabled ;; + --disable-pvrdma) printf "%s" -Dpvrdma=disabled ;; + --enable-qatzip) printf "%s" -Dqatzip=enabled ;; + --disable-qatzip) printf "%s" -Dqatzip=disabled ;; --enable-qcow1) printf "%s" -Dqcow1=enabled ;; --disable-qcow1) printf "%s" -Dqcow1=disabled ;; --enable-qed) printf "%s" -Dqed=enabled ;;
Signature · 1 line
Yichen Wang
PATCH v3 2/4 migration: Add migration parameters for QATzip
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds support for migration parameters to control QATzip compression level and to enable/disable software fallback when QAT hardware is unavailable. This is a preparatory commit for a subsequent commit that will actually use QATzip compression.
--- migration/migration-hmp-cmds.c | 8 +++++ migration/options.c | 57 ++++++++++++++++++++++++++++++++++ migration/options.h | 2 ++ qapi/migration.json | 35 +++++++++++++++++++++ 4 files changed, 102 insertions(+)
migration/migration-hmp-cmds.c +8 -0
@@ -576,6 +576,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zlib_level = true;visit_type_uint8 (v, param, &p->multifd_zlib_level, &err);break ; +case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL : + p->has_multifd_qatzip_level = true; +visit_type_uint8 (v, param, &p->multifd_qatzip_level, &err); +break ; +case MIGRATION_PARAMETER_MULTIFD_QATZIP_SW_FALLBACK : + p->has_multifd_qatzip_sw_fallback = true; +visit_type_bool (v, param, &p->multifd_qatzip_sw_fallback, &err); +break ;case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL : p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);
migration/options.c +57 -0
105 lines not on this page · raw message
migration/options.h +2 -0
@@ -78,6 +78,8 @@ uint64_t migrate_max_postcopy_bandwidth(void);int migrate_multifd_channels (void );MultiFDCompression migrate_multifd_compression (void );int migrate_multifd_zlib_level (void ); +int migrate_multifd_qatzip_level (void ); +bool migrate_multifd_qatzip_sw_fallback (void );int migrate_multifd_zstd_level (void );uint8_t migrate_throttle_trigger_threshold (void );const char *migrate_tls_authz (void );
qapi/migration.json +35 -0
@@ -789,6 +789,16 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.1) +# +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware +# is unavailable. Defaults to false. Software fallback performance +# is very poor compared to regular zlib, so be cautious about +# enabling this option. (Since 9.1) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -849,6 +859,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', + 'multifd-qatzip-level', 'multifd-qatzip-sw-fallback', 'block-bitmap-mapping', { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit',
@@ -964,6 +975,16 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.1) +# +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware +# is unavailable. Defaults to false. Software fallback performance +# is very poor compared to regular zlib, so be cautious about +# enabling this option. (Since 9.1) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -1037,6 +1058,8 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-qatzip-sw-fallback': 'bool', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
@@ -1168,6 +1191,16 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.1) +# +# @multifd-qatzip-sw-fallback: Enable software fallback if QAT hardware +# is unavailable. Defaults to false. Software fallback performance +# is very poor compared to regular zlib, so be cautious about +# enabling this option. (Since 9.1) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -1238,6 +1271,8 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', + '*multifd-qatzip-sw-fallback': 'bool', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
Signature · 1 line
Yichen Wang
PATCH v3 3/4 migration: Introduce 'qatzip' compression method
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds support for 'qatzip' as an option for the multifd compression method parameter, and implements using QAT for 'qatzip' compression and decompression.
--- hw/core/qdev-properties-system.c | 6 +- migration/meson.build | 1 + migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++ migration/multifd.h | 1 + qapi/migration.json | 3 + tests/qtest/meson.build | 4 + 6 files changed, 396 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.c
hw/core/qdev-properties-system.c +5 -1
@@ -659,7 +659,11 @@ const PropertyInfo qdev_prop_fdc_drive_type = {const PropertyInfo qdev_prop_multifd_compression = { .name ="MultiFDCompression" , .description ="multifd_compression values, " -"none/zlib/zstd/qpl/uadk" , +"none/zlib/zstd/qpl/uadk" +#ifdef CONFIG_QATZIP +"/qatzip" +#endif + , .enum_table = &MultiFDCompression_lookup, .get = qdev_propinfo_get_enum, .set = qdev_propinfo_set_enum,
migration/meson.build +1 -0
@@ -41,6 +41,7 @@ system_ss.add(when: rdma, if_true: files('rdma.c')) system_ss.add(when: zstd, if_true: files('multifd-zstd.c')) system_ss.add(when: qpl, if_true: files('multifd-qpl.c')) system_ss.add(when: uadk, if_true: files('multifd-uadk.c')) +system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c')) specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', if_true: files('ram.c',
migration/multifd-qatzip.c new file +382 -0
382 lines not on this page · raw message
migration/multifd.h +1 -0
@@ -40,6 +40,7 @@ MultiFDRecvData *multifd_get_recv_data(void);#define MULTIFD_FLAG_NOCOMP (0 << 1)#define MULTIFD_FLAG_ZLIB (1 << 1)#define MULTIFD_FLAG_ZSTD (2 << 1) +#define MULTIFD_FLAG_QATZIP (3 << 1)#define MULTIFD_FLAG_QPL (4 << 1)#define MULTIFD_FLAG_UADK (8 << 1)
qapi/migration.json +3 -0
@@ -558,6 +558,8 @@ # # @zstd: use zstd compression method. # +# @qatzip: use qatzip compression method. (Since 9.1) +# # @qpl: use qpl compression method. Query Processing Library(qpl) is # based on the deflate compression algorithm and use the Intel # In-Memory Analytics Accelerator(IAA) accelerated compression
@@ -570,6 +572,7 @@ { 'enum': 'MultiFDCompression', 'data': [ 'none', 'zlib', { 'name': 'zstd', 'if': 'CONFIG_ZSTD' }, + { 'name': 'qatzip', 'if': 'CONFIG_QATZIP'}, { 'name': 'qpl', 'if': 'CONFIG_QPL' }, { 'name': 'uadk', 'if': 'CONFIG_UADK' } ] }
tests/qtest/meson.build +4 -0
@@ -324,6 +324,10 @@ if gnutls.found() endif endif +if qatzip.found() + migration_files += [qatzip] +endif + qtests = { 'bios-tables-test': [io, 'boot-sector.c', 'acpi-utils.c', 'tpm-emu.c'], 'cdrom-test': files('boot-sector.c'),
Signature · 1 line
Yichen Wang
PATCH v3 4/4 tests/migration: Add integration test for 'qatzip' compression method
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds an integration test for 'qatzip'.
--- tests/qtest/migration-test.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
tests/qtest/migration-test.c +35 -0
@@ -32,6 +32,10 @@# endif /* CONFIG_TASN1 */ #endif /* CONFIG_GNUTLS */ +#ifdef CONFIG_QATZIP +#include <qatzip.h> +#endif /* CONFIG_QATZIP */ +/* For dirty ring test; so far only x86_64 is supported */ #if defined(__linux__) && defined(HOST_X86_64 )#include "linux/kvm.h"
@@ -2992,6 +2996,22 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestState *from, }#endif /* CONFIG_ZSTD */ +#ifdef CONFIG_QATZIP +static void * +test_migrate_precopy_tcp_multifd_qatzip_start (QTestState *from, +QTestState *to) +{ +migrate_set_parameter_int (from,"multifd-qatzip-level" ,2 ); +migrate_set_parameter_int (to,"multifd-qatzip-level" ,2 ); + +/* SW fallback is disabled by default, so enable it for testing. */ +migrate_set_parameter_bool (from,"multifd-qatzip-sw-fallback" , true); +migrate_set_parameter_bool (to,"multifd-qatzip-sw-fallback" , true); + +return test_migrate_precopy_tcp_multifd_start_common (from, to,"qatzip" ); +} +#endif +#ifdef CONFIG_QPL static void * test_migrate_precopy_tcp_multifd_qpl_start(QTestState *from,
@@ -3089,6 +3109,17 @@ static void test_multifd_tcp_zstd(void) }#endif +#ifdef CONFIG_QATZIP +static void test_multifd_tcp_qatzip (void ) +{ +MigrateCommon args = { + .listen_uri ="defer" , + .start_hook = test_migrate_precopy_tcp_multifd_qatzip_start, + }; +test_precopy_common (&args); +} +#endif +#ifdef CONFIG_QPL static void test_multifd_tcp_qpl (void ) {
@@ -4002,6 +4033,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/tcp/plain/zstd" , test_multifd_tcp_zstd);#endif +#ifdef CONFIG_QATZIP +migration_test_add ("/migration/multifd/tcp/plain/qatzip" , + test_multifd_tcp_qatzip); +#endif #ifdef CONFIG_QPL migration_test_add ("/migration/multifd/tcp/plain/qpl" , test_multifd_tcp_qpl);
Signature · 1 line
Yichen Wang
Quoted 94 lines · Yichen Wang
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentationsv2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html* Performance
We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but with two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a more realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia** Results
4 multifd threads:
|---------------|---------------|----------------|---------|---------| |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%| |---------------|---------------|----------------|---------|---------| |qatzip | 23.13 | 8749.94 |117.50 |186.49 | |---------------|---------------|----------------|---------|---------| |zlib |254.35 | 771.87 |388.20 |144.40 | |---------------|---------------|----------------|---------|---------| |zstd | 54.52 | 3442.59 |414.59 |149.77 | |---------------|---------------|----------------|---------|---------| |none | 12.45 |43739.60 |159.71 |204.96 | |---------------|---------------|----------------|---------|---------|8 multifd threads:
|---------------|---------------|----------------|---------|---------| |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%| |---------------|---------------|----------------|---------|---------| |qatzip | 16.91 |12306.52 |186.37 |391.84 | |---------------|---------------|----------------|---------|---------| |zlib |130.11 | 1508.89 |753.86 |289.35 | |---------------|---------------|----------------|---------|---------| |zstd | 27.57 | 6823.23 |786.83 |303.80 | |---------------|---------------|----------------|---------|---------| |none | 11.82 |46072.63 |163.74 |238.56 | |---------------|---------------|----------------|---------|---------|16 multifd threads:
|---------------|---------------|----------------|---------|---------| |method |time(sec) |throughput(mbps)|send cpu%|recv cpu%| |---------------|---------------|----------------|---------|---------| |qatzip |18.64 |11044.52 | 573.61 |437.65 | |---------------|---------------|----------------|---------|---------| |zlib |66.43 | 2955.79 |1469.68 |567.47 | |---------------|---------------|----------------|---------|---------| |zstd |14.17 |13290.66 |1504.08 |615.33 | |---------------|---------------|----------------|---------|---------| |none |16.82 |32363.26 | 180.74 |217.17 | |---------------|---------------|----------------|---------|---------|** Observations
Quoted 14 lines
- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.
Thanks for the results update.
It looks like the docs/migration/ file is still missing. It'll be great to have it in the next version or separately.
So how it compares with QPL (which got merged already)? They at least look like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.
I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even if the work should have been offloaded to the QAT chips iiuc.
I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about it outperforms others).
Thanks,
Quoted 27 lines
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression methodhw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++ migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.cSignature · 1 line
Yichen Wang
Signature · 1 line
Peter Xu
Quoted 165 lines
-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 28 lines · Yichen Wang
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentationsv2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html* Performance
We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but with two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.Quoted 6 lines
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a
more
Quoted 11 lines
realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).
Quoted 8 lines
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia** Results
4 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip | 23.13 | 8749.94 |117.50 |186.49|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |254.35 | 771.87 |388.20 |144.40|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd | 54.52 | 3442.59 |414.59 |149.77|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none | 12.45 |43739.60 |159.71 |204.96|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 4 lines
8 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip | 16.91 |12306.52 |186.37 |391.84|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |130.11 | 1508.89 |753.86 |289.35|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd | 27.57 | 6823.23 |786.83 |303.80|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none | 11.82 |46072.63 |163.74 |238.56|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 4 lines
16 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip |18.64 |11044.52 | 573.61 |437.65|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |66.43 | 2955.79 |1469.68 |567.47|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd |14.17 |13290.66 |1504.08 |615.33|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none |16.82 |32363.26 | 180.74 |217.17|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 17 lines
** Observations
- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.Thanks for the results update.
It looks like the docs/migration/ file is still missing. It'll be great to
have it in the next version or separately.
Quoted 6 lines
So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.
I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.
1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA, different SKUs support different numbers of QAT and IAA, so some users do not have both QAT and IAA at the same time.
2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already used QAT cards to accelerate live migration.
3. In addition to compression, QAT and IAA also support various other features to better serve different workloads. Here is an introduction to the accelerators, including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf
For users who have both QAT and IAA, we recommend the following for choosing a live migration solution:
1. If the number of QAT devices is equal to or greater than the number of IAA devices and network bandwidth is limited, it is recommended to use the QATZip(QAT) solution.
2. In other scenarios, the QPL (IAA) solution can be used first.
Quoted 45 lines
I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about
it outperforms others).Thanks,
Quoted 24 lines
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression methodhw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++ migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.cQuoted 3 lines
--
Yichen Wang--
Peter Xu
Quoted 165 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 165 lines
-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 28 lines · Yichen Wang
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentationsv2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html* Performance
We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but with two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.Quoted 6 lines
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a
more
Quoted 11 lines
realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).
Quoted 8 lines
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia** Results
4 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip | 23.13 | 8749.94 |117.50 |186.49|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |254.35 | 771.87 |388.20 |144.40|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd | 54.52 | 3442.59 |414.59 |149.77|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none | 12.45 |43739.60 |159.71 |204.96|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 4 lines
8 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip | 16.91 |12306.52 |186.37 |391.84|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |130.11 | 1508.89 |753.86 |289.35|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd | 27.57 | 6823.23 |786.83 |303.80|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none | 11.82 |46072.63 |163.74 |238.56|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 4 lines
16 multifd threads:
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|qatzip |18.64 |11044.52 | 573.61 |437.65|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zlib |66.43 | 2955.79 |1469.68 |567.47|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|zstd |14.17 |13290.66 |1504.08 |615.33|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 1 line
|none |16.82 |32363.26 | 180.74 |217.17|
Quoted 1 line
|---------------|---------------|----------------|---------|---------|
Quoted 17 lines
** Observations
- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.Thanks for the results update.
It looks like the docs/migration/ file is still missing. It'll be great to
have it in the next version or separately.
Quoted 6 lines
Quoted 6 lines
So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused on why Intel provides these two similar compressors. It would be great to have some answer and perhaps put into the doc.
Yuan,
Quoted 3 lines
I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.
Yes, these are very helpful information. Please consider putting them into the cover letter if there's a repost, and perhaps also in the doc/ files.
Quoted 8 lines
1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA, different SKUs support different numbers of QAT and IAA, so some users do not have both QAT and IAA at the same time.
2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already used QAT cards to accelerate live migration.
Ah, this makes some sense to me.
So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.
So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?
If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper would be nice too.
Quoted 5 lines
3. In addition to compression, QAT and IAA also support various other features to better serve different workloads. Here is an introduction to the accelerators, including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf
Thanks for the link.
However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.
Two major (but pure..) questions:
1) Why high cpu usage?
I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?
The paper you provided above has this upfront:
When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...
Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?
2) TLS?
I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS (instead of compression) with the QATs?
Thanks,
Quoted 15 lines
For users who have both QAT and IAA, we recommend the following for choosing a live migration solution:
1. If the number of QAT devices is equal to or greater than the number of IAA devices and network bandwidth is limited, it is recommended to use the QATZip(QAT) solution.
2. In other scenarios, the QPL (IAA) solution can be used first.
Quoted 6 lines
I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp is mostly always better than qatzip, no matter on total migration time or cpu consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.
[1]
Quoted 40 lines
Quoted 39 lines
I think it may not be a problem to merge this series even if it performs slower at some criterias.. but I think we may still want to know when this should be used, or the good reason this should be merged (if it's not about
it outperforms others).Thanks,
Quoted 24 lines
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression methodhw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382 +++++++++++++++++++++++++++++++ migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.cQuoted 3 lines
--
Yichen Wang--
Peter Xu
Signature · 1 line
Peter Xu
Quoted 269 lines
-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Thursday, July 4, 2024 11:36 PM
To: Liu, Yuan1 <[email protected]>
Cc: Wang, Yichen <[email protected]>; Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; [email protected]; Hao Xiang <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren (Jack) Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 6 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 6 lines
-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André
Quoted 6 lines
Quoted 6 lines
Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren
(Jack)
Quoted 16 lines
Quoted 16 lines
Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 12 lines · Yichen Wang
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentationsv2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
12/msg03761.html
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* Performance
We present updated performance results. For circumstantial reasons,
v1
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but
with
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2of
Quoted 1 line
Quoted 1 line
Quoted 1 line
the VM's RAM. Now we properly allocate and fill nearly all of the
VM's
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
RAM.
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB
memory.
Quoted 1 line
Quoted 1 line
Quoted 1 line
We prepare the machine's memory by powering it on, allocating a
large
Quoted 1 line
Quoted 1 line
Quoted 1 line
amount of memory (60GB) as a single buffer, and filling the buffer
with
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
the repeated contents of the Silesia corpus[0]. This is in lieu of a
more
Quoted 8 lines
realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
compression methods, with varying numbers of multifd threads (4, 8,
and
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
16).
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia** Results
4 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip | 23.13 | 8749.94 |117.50|186.49
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |254.35 | 771.87 |388.20|144.40
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd | 54.52 | 3442.59 |414.59|149.77
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none | 12.45 |43739.60 |159.71|204.96
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
8 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip | 16.91 |12306.52 |186.37|391.84
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |130.11 | 1508.89 |753.86|289.35
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd | 27.57 | 6823.23 |786.83|303.80
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none | 11.82 |46072.63 |163.74|238.56
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
16 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip |18.64 |11044.52 | 573.61|437.65
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |66.43 | 2955.79 |1469.68|567.47
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd |14.17 |13290.66 |1504.08|615.33
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none |16.82 |32363.26 | 180.74|217.17
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
** Observations
- In general, not using compression outperforms using compression in
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8workers,
Quoted 1 line
Quoted 1 line
Quoted 1 line
achieving a ~91% latency reduction over 'zlib' with 4 workers, and
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16workers,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression ishighly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
parallelizable.
- 'qatzip' compression uses considerably less CPU than othercompression
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16workers.
Quoted 4 lines
Quoted 4 lines
Thanks for the results update.
It looks like the docs/migration/ file is still missing. It'll be
great
Quoted 7 lines
Quoted 7 lines
to
have it in the next version or separately.So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused onwhy
Quoted 1 line
Quoted 1 line
Intel provides these two similar compressors. It would be great to
have
Quoted 1 line
Quoted 1 line
some answer and perhaps put into the doc.
Yuan,
Quoted 3 lines
I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.
Yes, these are very helpful information. Please consider putting them into
the cover letter if there's a repost, and perhaps also in the doc/ files.Quoted 2 lines
1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA,
different
Quoted 1 line
SKUs support different numbers of QAT and IAA, so some users do not
have
Quoted 4 lines
both QAT and IAA at the same time.
2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already
used QAT
Quoted 1 line
cards to accelerate live migration.
Ah, this makes some sense to me.
So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.
Quoted 4 lines
So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?
Yes, users do not add a QAT card just for live migration. Users mainly use QAT-SRIOV technology to help cloud users offload compression and encryption.
Quoted 8 lines
If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if
there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper
would be nice too.
Yes, I will provide some performance data at some specific bandwidths(100Mbps/1Gbps/10Gbps). And add documentation to explain the advantages of using QAT
Quoted 23 lines
Quoted 2 lines
3. In addition to compression, QAT and IAA also support various other
features
Quoted 1 line
to better serve different workloads. Here is an introduction to the
accelerators,
Quoted 2 lines
including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf
Thanks for the link.
However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.
Two major (but pure..) questions:
1) Why high cpu usage?
I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?
Quoted 7 lines
The paper you provided above has this upfront:
When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...
Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?
Yes, it doesn't make sense, I will check this.
Quoted 6 lines
2) TLS?
I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS
(instead
of compression) with the QATs?
I'm sorry for not responding to the previous question about TLS. QAT has many related success cases (for example, OpenSSL).
https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/accelerating-openssl-brief.pdf
I will send a separate RFC or patch about this part because the software stacks of QAT compression and encryption are independent, so we discuss them separately.
Quoted 72 lines
Thanks,
Quoted 2 lines
For users who have both QAT and IAA, we recommend the following for
choosing a
Quoted 3 lines
live migration solution:
1. If the number of QAT devices is equal to or greater than the number
of IAA devices
Quoted 1 line
and network bandwidth is limited, it is recommended to use the
QATZip(QAT) solution.
Quoted 5 lines
2. In other scenarios, the QPL (IAA) solution can be used first.
Quoted 2 lines
I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp
is
Quoted 1 line
Quoted 1 line
mostly always better than qatzip, no matter on total migration time or
cpu
Quoted 3 lines
Quoted 3 lines
consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.[1]
Quoted 2 lines
Quoted 2 lines
I think it may not be a problem to merge this series even if it
performs
Quoted 1 line
Quoted 1 line
slower at some criterias.. but I think we may still want to know when
this
Quoted 8 lines
Quoted 8 lines
should be used, or the good reason this should be merged (if it's not about
it outperforms others).Thanks,
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compressionmethod
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382+++++++++++++++++++++++++++++++
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-) create mode 100644 migration/multifd-qatzip.cQuoted 7 lines
Quoted 6 lines
Quoted 3 lines
--
Yichen Wang--
Peter Xu--
Peter Xu
Quoted 290 lines · On Jul 5, 2024, at 1:32 AM, Liu, Yuan1 <[email protected]> wrote:
Quoted 270 lines
-----Original Message-----
From: Peter Xu <[email protected] <mailto:[email protected]>>
Sent: Thursday, July 4, 2024 11:36 PM
To: Liu, Yuan1 <[email protected] <mailto:[email protected]>>
Cc: Wang, Yichen <[email protected] <mailto:[email protected]>>; Paolo Bonzini <[email protected] <mailto:[email protected]>>; Daniel P. Berrangé <[email protected] <mailto:[email protected]>>; Eduardo Habkost <[email protected] <mailto:[email protected]>>; Marc-André Lureau
<[email protected] <mailto:[email protected]>>; Thomas Huth <[email protected] <mailto:[email protected]>>; Philippe Mathieu-Daudé <[email protected] <mailto:[email protected]>>; Fabiano Rosas <[email protected] <mailto:[email protected]>>; Eric Blake <[email protected] <mailto:[email protected]>>; Markus Armbruster <[email protected] <mailto:[email protected]>>; Laurent Vivier <[email protected] <mailto:[email protected]>>; [email protected] <mailto:[email protected]>; Hao Xiang <[email protected] <mailto:[email protected]>>; Zou, Nanhai <[email protected] <mailto:[email protected]>>; Ho-Ren (Jack) Chuang <[email protected] <mailto:[email protected]>>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 6 lines · On Thu, Jul 04, 2024 at 03:15:51AM +0000, Liu, Yuan1 wrote:
Quoted 6 lines
-----Original Message-----
From: Peter Xu <[email protected]>
Sent: Wednesday, July 3, 2024 3:16 AM
To: Wang, Yichen <[email protected]>
Cc: Paolo Bonzini <[email protected]>; Daniel P. Berrangé <[email protected]>; Eduardo Habkost <[email protected]>; Marc-André
Quoted 6 lines
Quoted 6 lines
Lureau <[email protected]>; Thomas Huth <[email protected]>; Philippe Mathieu-Daudé <[email protected]>; Fabiano Rosas <[email protected]>; Eric Blake <[email protected]>; Markus Armbruster <[email protected]>; Laurent Vivier <[email protected]>; qemu- [email protected]; Hao Xiang <[email protected]>; Liu, Yuan1 <[email protected]>; Zou, Nanhai <[email protected]>; Ho-Ren
(Jack)
Quoted 16 lines
Quoted 16 lines
Chuang <[email protected]>
Subject: Re: [PATCH v3 0/4] Implement using Intel QAT to offload ZLIBQuoted 12 lines · Yichen Wang
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentationsv2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
12/msg03761.html
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
* Performance
We present updated performance results. For circumstantial reasons,
v1
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but
with
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2of
Quoted 1 line
Quoted 1 line
Quoted 1 line
the VM's RAM. Now we properly allocate and fill nearly all of the
VM's
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
RAM.
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB
memory.
Quoted 1 line
Quoted 1 line
Quoted 1 line
We prepare the machine's memory by powering it on, allocating a
large
Quoted 1 line
Quoted 1 line
Quoted 1 line
amount of memory (60GB) as a single buffer, and filling the buffer
with
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
the repeated contents of the Silesia corpus[0]. This is in lieu of a
more
Quoted 8 lines
realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
compression methods, with varying numbers of multifd threads (4, 8,
and
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
16).
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia** Results
4 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip | 23.13 | 8749.94 |117.50|186.49
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |254.35 | 771.87 |388.20|144.40
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd | 54.52 | 3442.59 |414.59|149.77
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none | 12.45 |43739.60 |159.71|204.96
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
8 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip | 16.91 |12306.52 |186.37|391.84
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |130.11 | 1508.89 |753.86|289.35
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd | 27.57 | 6823.23 |786.83|303.80
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none | 11.82 |46072.63 |163.74|238.56
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
16 multifd threads:
|---------------|---------------|----------------|---------|--------
Quoted 4 lines
Quoted 4 lines
-|
Quoted 1 line
|method |time(sec) |throughput(mbps)|send cpu%|recv
cpu%|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|qatzip |18.64 |11044.52 | 573.61|437.65
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zlib |66.43 | 2955.79 |1469.68|567.47
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|zstd |14.17 |13290.66 |1504.08|615.33
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 2 lines
Quoted 2 lines
-|
Quoted 1 line
|none |16.82 |32363.26 | 180.74|217.17
Quoted 2 lines
Quoted 2 lines
|
Quoted 1 line
|---------------|---------------|----------------|---------|--------
Quoted 5 lines
Quoted 5 lines
-|
Quoted 4 lines
** Observations
- In general, not using compression outperforms using compression in
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8workers,
Quoted 1 line
Quoted 1 line
Quoted 1 line
achieving a ~91% latency reduction over 'zlib' with 4 workers, and
a
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16workers,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression ishighly
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
parallelizable.
- 'qatzip' compression uses considerably less CPU than othercompression
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16workers.
Quoted 4 lines
Quoted 4 lines
Thanks for the results update.
It looks like the docs/migration/ file is still missing. It'll be
great
Quoted 7 lines
Quoted 7 lines
to
have it in the next version or separately.So how it compares with QPL (which got merged already)? They at least look
like both supported on an Intel platform, so an user whoever wants to compress the RAM could start to look at both. I'm utterly confused onwhy
Quoted 1 line
Quoted 1 line
Intel provides these two similar compressors. It would be great to
have
Quoted 1 line
Quoted 1 line
some answer and perhaps put into the doc.
Yuan,
Quoted 3 lines
I would like to explain some of the reasons why we want to merge the two QAT and IAA solutions into the community.
Yes, these are very helpful information. Please consider putting them into
the cover letter if there's a repost, and perhaps also in the doc/ files.Quoted 2 lines
1. Although Intel Xeon Sapphire Rapids supports both QAT and IAA,
different
Quoted 1 line
SKUs support different numbers of QAT and IAA, so some users do not
have
Quoted 4 lines
both QAT and IAA at the same time.
2. QAT products include PCIe cards, which are compatible with older Xeon products and other non-Xeon products. And some users have already
used QAT
Quoted 1 line
cards to accelerate live migration.
Ah, this makes some sense to me.
So a previous question always haunted me, where I wondered why an user who bought all these fancy and expensive processors with QAT, would still like to not invest on a better network of 50G or more, but stick with 10Gs ancient NICs and switches.
Quoted 4 lines
So what you're saying is logically in some old clusters with old chips and old network solutions, it's possible that user buys these PCIe cards separately so it may help with that old infra migrate VMs faster. Is that the case?
Yes, users do not add a QAT card just for live migration. Users mainly use QAT-SRIOV technology to help cloud users offload compression and encryption.
Quoted 8 lines
If so, we may still want some numbers showing how this performs in a network-limited environment, and how that helps users to migrate. Sorry if
there's some back-and-forth requirement asking for these numbers, but I think these are still important information when an user would like to decide whether to use these features. Again, put that into docs/ if proper
would be nice too.
Quoted 3 lines
Yes, I will provide some performance data at some specific bandwidths(100Mbps/1Gbps/10Gbps). And add documentation to explain the advantages of using QAT
Just want to add some information here. So in ByteDance, the current generation server is quipped with 2*100Gb NIC. We reserve 10Gbps for control plane purposes which includes live migration here. So it is not about we are using “good network”, it is about not normal to use full bandwidth for control plane purposes. Hence we do have a requirements for QAT/IAA in these cases.
Quoted 119 lines
Quoted 23 lines
Quoted 2 lines
3. In addition to compression, QAT and IAA also support various other
features
Quoted 1 line
to better serve different workloads. Here is an introduction to the
accelerators,
Quoted 2 lines
including usage scenarios of QAT and IAA.
https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2022-12/storage-engines-4th-gen-xeon-brief.pdf
Thanks for the link.
However this doesn't look like a reason to support it in migration? It needs to help migration in some form or another, no matter how many features it provides.. since migration may not consume them.
Two major (but pure..) questions:
1) Why high cpu usage?
I raised this question below [1] too but I didn't yet get an answer. Ror 8-chan multifd, it's ~390% (QAT) v.s. ~240% (nocomp), even if 46Gbps bw for the latter... so when throttled it will be even lower?
Quoted 7 lines
The paper you provided above has this upfront:
When a CPU can offload storage functions to built-in accelerators, it frees up cores for business-critical workloads...
Isn't that a major feature to be able to "offload" things? Why cpu isn't freed even if the offload happened?
Yes, it doesn't make sense, I will check this.
Quoted 6 lines
2) TLS?
I think I asked before, I apologize if any of you've already answered and if I forgot.. but have any of you looked into offload TLS
(instead
of compression) with the QATs?I'm sorry for not responding to the previous question about TLS. QAT has many related success cases (for example, OpenSSL).
https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/accelerating-openssl-brief.pdfI will send a separate RFC or patch about this part because the software stacks of QAT compression and encryption are independent, so we discuss them separately.
Quoted 72 lines
Thanks,
Quoted 2 lines
For users who have both QAT and IAA, we recommend the following for
choosing a
Quoted 3 lines
live migration solution:
1. If the number of QAT devices is equal to or greater than the number
of IAA devices
Quoted 1 line
and network bandwidth is limited, it is recommended to use the
QATZip(QAT) solution.
Quoted 5 lines
2. In other scenarios, the QPL (IAA) solution can be used first.
Quoted 2 lines
I am honestly curious too on whether are you planning to use it in production. It looks like if the network resources are rich, no-comp
is
Quoted 1 line
Quoted 1 line
mostly always better than qatzip, no matter on total migration time or
cpu
Quoted 3 lines
Quoted 3 lines
consumption. I'm pretty surprised that it'll take that much resources even
if the work should have been offloaded to the QAT chips iiuc.[1]
Quoted 2 lines
Quoted 2 lines
I think it may not be a problem to merge this series even if it
performs
Quoted 1 line
Quoted 1 line
slower at some criterias.. but I think we may still want to know when
this
Quoted 8 lines
Quoted 8 lines
should be used, or the good reason this should be merged (if it's not about
it outperforms others).Thanks,
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compressionmethod
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/core/qdev-properties-system.c | 6 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 8 + migration/multifd-qatzip.c | 382+++++++++++++++++++++++++++++++
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
migration/multifd.h | 1 + migration/options.c | 57 +++++ migration/options.h | 2 + qapi/migration.json | 38 +++ scripts/meson-buildoptions.sh | 6 + tests/qtest/meson.build | 4 + tests/qtest/migration-test.c | 35 +++ 13 files changed, 551 insertions(+), 1 deletion(-)create mode 100644
migration/multifd-qatzip.cQuoted 7 lines
Quoted 6 lines
Quoted 3 lines
--
Yichen Wang--
Peter Xu--
Peter Xu
Quoted 6 lines · Yichen Wang
Just want to add some information here. So in ByteDance, the current generation server is quipped with 2*100Gb NIC. We reserve 10Gbps for control plane purposes which includes live migration here. So it is not about we are using “good network”, it is about not normal to use full bandwidth for control plane purposes. Hence we do have a requirements for QAT/IAA in these cases.
Yes, this makes sense.
But then you may then also want to figure out the high cpu consumption of those cards, to not interrupt more important workloads?
I saw there's a new version posted, I didn't see an explanation of the cpu consumption issue mentioned. Meanwhile I also see that the docs/ update is missing.
Would you consider adding both by replying to the new version?
Signature · 1 line
Peter Xu