Implement QATzip compression method
v8 cover Implement QATzip compression method
v8:
- Rebase changes on top of 2eefd4fcec4b8fe41ceee2a8f00cdec1fe81b75c;
- Fix typo and grammars in documentation and comments;
v7:
- Rebase changes on top of 0173b97a219c63062972744682eba46c560fb7f3
- Added QAT memory requirement introduction in documentations;
- Change the configuration options detection to auto in meson;
- Enhance the auto-fallback mechanism to align with QPL behavior;
v6:
- Rebase changes on top of 4ea7e9cd882f1574c129d67431784fecc426d23b
- Address comments, typos, and styling issues
- Re-factor QAT setup()/cleanup() logic
v5:
- Rebase changes on top of 59084feb256c617063e0dbe7e64821ae8852d7cf
- Add documentations about migration with qatzip accerlation
- Remove multifd-qatzip-sw-fallback option
v4:
- Rebase changes on top of 1a2d52c7fcaeaaf4f2fe8d4d5183dccaeab67768
- Move the IOV initialization to qatzip implementation
- Only use qatzip to compress normal pages
v3:
- Rebase changes on top of master
- Merge two patches per Fabiano Rosas's comment
- Add versions into comments and documentations
v2:
- Rebase changes on top of recent multifd code changes.
- Use QATzip API 'qzMalloc' and 'qzFree' to allocate QAT buffers.
- Remove parameter tuning and use QATzip's defaults for better performance.
- Add parameter to enable QAT software fallback.
v1:
https://lists.nongnu.org/archive/html/qemu-devel/2023-12/msg03761.html
* Performance
We present updated performance results. For circumstantial reasons, v1 presented performance on a low-bandwidth (1Gbps) network.
Here, we present updated results with a similar setup as before but with two main differences:
1. Our machines have a ~50Gbps connection, tested using 'iperf3'.
2. We had a bug in our memory allocation causing us to only use ~1/2 of the VM's RAM. Now we properly allocate and fill nearly all of the VM's RAM.
Thus, the test setup is as follows:
We perform multifd live migration over TCP using a VM with 64GB memory. We prepare the machine's memory by powering it on, allocating a large amount of memory (60GB) as a single buffer, and filling the buffer with the repeated contents of the Silesia corpus[0]. This is in lieu of a more realistic memory snapshot, which proved troublesome to acquire.
We analyze CPU usage by averaging the output of 'top' every second during migration. This is admittedly imprecise, but we feel that it accurately portrays the different degrees of CPU usage of varying compression methods.
We present the latency, throughput, and CPU usage results for all of the compression methods, with varying numbers of multifd threads (4, 8, and 16).
[0] The Silesia corpus can be accessed here:
https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia
** Results
4 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip | 23.13 | 8749.94 |117.50 |186.49 |
|---------------|---------------|----------------|---------|---------|
|zlib |254.35 | 771.87 |388.20 |144.40 |
|---------------|---------------|----------------|---------|---------|
|zstd | 54.52 | 3442.59 |414.59 |149.77 |
|---------------|---------------|----------------|---------|---------|
|none | 12.45 |43739.60 |159.71 |204.96 |
|---------------|---------------|----------------|---------|---------|8 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip | 16.91 |12306.52 |186.37 |391.84 |
|---------------|---------------|----------------|---------|---------|
|zlib |130.11 | 1508.89 |753.86 |289.35 |
|---------------|---------------|----------------|---------|---------|
|zstd | 27.57 | 6823.23 |786.83 |303.80 |
|---------------|---------------|----------------|---------|---------|
|none | 11.82 |46072.63 |163.74 |238.56 |
|---------------|---------------|----------------|---------|---------|16 multifd threads:
|---------------|---------------|----------------|---------|---------|
|method |time(sec) |throughput(mbps)|send cpu%|recv cpu%|
|---------------|---------------|----------------|---------|---------|
|qatzip |18.64 |11044.52 | 573.61 |437.65 |
|---------------|---------------|----------------|---------|---------|
|zlib |66.43 | 2955.79 |1469.68 |567.47 |
|---------------|---------------|----------------|---------|---------|
|zstd |14.17 |13290.66 |1504.08 |615.33 |
|---------------|---------------|----------------|---------|---------|
|none |16.82 |32363.26 | 180.74 |217.17 |
|---------------|---------------|----------------|---------|---------|** Observations
- In general, not using compression outperforms using compression in a non-network-bound environment.
- 'qatzip' outperforms other compression workers with 4 and 8 workers, achieving a ~91% latency reduction over 'zlib' with 4 workers, and a
~58% latency reduction over 'zstd' with 4 workers.
- 'qatzip' maintains comparable performance with 'zstd' at 16 workers, showing a ~32% increase in latency. This performance difference
becomes more noticeable with more workers, as CPU compression is highly parallelizable.
- 'qatzip' compression uses considerably less CPU than other compression methods. At 8 workers, 'qatzip' demonstrates a ~75% reduction in
compression CPU usage compared to 'zstd' and 'zlib'.
- 'qatzip' decompression CPU usage is less impressive, and is even slightly worse than 'zstd' and 'zlib' CPU usage at 4 and 16 workers.
Bryan Zhang (4):
meson: Introduce 'qatzip' feature to the build system
migration: Add migration parameters for QATzip
migration: Introduce 'qatzip' compression method tests/migration: Add integration test for 'qatzip' compression method
Yuan Liu (1):
docs/migration: add qatzip compression feature
docs/devel/migration/features.rst | 1 + docs/devel/migration/qatzip-compression.rst | 165 ++++++++ hw/core/qdev-properties-system.c | 2 +- meson.build | 10 + meson_options.txt | 2 + migration/meson.build | 1 + migration/migration-hmp-cmds.c | 4 + migration/multifd-qatzip.c | 394 ++++++++++++++++++++ migration/multifd.h | 5 +- migration/options.c | 34 ++ migration/options.h | 1 + qapi/migration.json | 21 ++ scripts/meson-buildoptions.sh | 3 + tests/qtest/migration-test.c | 27 ++ 14 files changed, 667 insertions(+), 3 deletions(-) create mode 100644 docs/devel/migration/qatzip-compression.rst create mode 100644 migration/multifd-qatzip.c
Signature · 1 line
Yichen Wang
PATCH v8 1/5 docs/migration: add qatzip compression feature
From: Yuan Liu <[email protected]>
add Intel QATzip compression method introduction
--- docs/devel/migration/features.rst | 1 + docs/devel/migration/qatzip-compression.rst | 165 ++++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 docs/devel/migration/qatzip-compression.rst
docs/devel/migration/features.rst +1 -0
@@ -14,3 +14,4 @@ Migration has plenty of features to support different use cases. CPR qpl-compression uadk-compression + qatzip-compression
docs/devel/migration/qatzip-compression.rst new file +165 -0
165 lines not on this page · raw message
Signature · 1 line
Yichen Wang
Re: [PATCH v8 1/5] docs/migration: add qatzip compression feature
Quoted 8 lines · Yichen Wang
From: Yuan Liu <[email protected]>
add Intel QATzip compression method introduction
Reviewed-by Nanhai Zou <[email protected]>Reviewed-by Peter Xu <[email protected]>Signed-off-by Yuan Liu <[email protected]>Signed-off-by Yichen Wang <[email protected]>
PATCH v8 2/5 meson: Introduce 'qatzip' feature to the build system
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Add a 'qatzip' feature, which is automatically disabled, and which depends on the QATzip library if enabled.
--- meson.build | 10 ++++++++++ meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 15 insertions(+)
meson.build +10 -0
@@ -1262,6 +1262,14 @@ if not get_option('uadk').auto() or have_system uadk = declare_dependency(dependencies: [libwd, libwd_comp]) endif endif + +qatzip = not_found +if not get_option('qatzip').auto() or have_system + qatzip = dependency('qatzip', version: '>=1.1.2', + required: get_option('qatzip'), + method: 'pkg-config') +endif + virgl = not_found have_vhost_user_gpu = have_tools and host_os == 'linux' and pixman.found()
@@ -2412,6 +2420,7 @@ config_host_data.set('CONFIG_STATX_MNT_ID', has_statx_mnt_id) config_host_data.set('CONFIG_ZSTD', zstd.found()) config_host_data.set('CONFIG_QPL', qpl.found()) config_host_data.set('CONFIG_UADK', uadk.found()) +config_host_data.set('CONFIG_QATZIP', qatzip.found()) config_host_data.set('CONFIG_FUSE', fuse.found()) config_host_data.set('CONFIG_FUSE_LSEEK', fuse_lseek.found()) config_host_data.set('CONFIG_SPICE_PROTOCOL', spice_protocol.found())
@@ -4535,6 +4544,7 @@ summary_info += {'lzfse support': liblzfse} summary_info += {'zstd support': zstd} summary_info += {'Query Processing Library support': qpl} summary_info += {'UADK Library support': uadk} +summary_info += {'qatzip support': qatzip} summary_info += {'NUMA host support': numa} summary_info += {'capstone': capstone} summary_info += {'libpmem support': libpmem}
meson_options.txt +2 -0
@@ -261,6 +261,8 @@ option('qpl', type : 'feature', value : 'auto', description: 'Query Processing Library support') option('uadk', type : 'feature', value : 'auto', description: 'UADK Library support') +option('qatzip', type: 'feature', value: 'auto', + description: 'QATzip compression support') option('fuse', type: 'feature', value: 'auto', description: 'FUSE block device export') option('fuse_lseek', type : 'feature', value : 'auto',
scripts/meson-buildoptions.sh +3 -0
@@ -163,6 +163,7 @@ meson_options_help() { printf "%s\n" ' pixman pixman support' printf "%s\n" ' plugins TCG plugins via shared library loading' printf "%s\n" ' png PNG support with libpng' + printf "%s\n" ' qatzip QATzip compression support' printf "%s\n" ' qcow1 qcow1 image format support' printf "%s\n" ' qed qed image format support' printf "%s\n" ' qga-vss build QGA VSS support (broken with MinGW)'
@@ -427,6 +428,8 @@ _meson_option_parse() { --enable-png) printf "%s" -Dpng=enabled ;; --disable-png) printf "%s" -Dpng=disabled ;; --prefix=*) quote_sh "-Dprefix=$2" ;; + --enable-qatzip) printf "%s" -Dqatzip=enabled ;; + --disable-qatzip) printf "%s" -Dqatzip=disabled ;; --enable-qcow1) printf "%s" -Dqcow1=enabled ;; --disable-qcow1) printf "%s" -Dqcow1=disabled ;; --enable-qed) printf "%s" -Dqed=enabled ;;
Signature · 1 line
Yichen Wang
PATCH v8 3/5 migration: Add migration parameters for QATzip
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds support for migration parameters to control QATzip compression level and to enable/disable software fallback when QAT hardware is unavailable. This is a preparatory commit for a subsequent commit that will actually use QATzip compression.
--- migration/migration-hmp-cmds.c | 4 ++++ migration/options.c | 34 ++++++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 ++++++++++++++++++ 4 files changed, 57 insertions(+)
migration/migration-hmp-cmds.c +4 -0
@@ -576,6 +576,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zlib_level = true;visit_type_uint8 (v, param, &p->multifd_zlib_level, &err);break ; +case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL : + p->has_multifd_qatzip_level = true; +visit_type_uint8 (v, param, &p->multifd_qatzip_level, &err); +break ;case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL : p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);
migration/options.c +34 -0
@@ -55,6 +55,13 @@#define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE/* 0: means nocompress, 1: best speed, ... 9: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1 +/* +* 1: best speed, ... 9: best compress ratio +* There is some nuance here. Refer to QATzip documentation to understand +* the mapping of QATzip levels to standard deflate levels. +*/ +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1 +/* 0: means nocompress, 1: best speed, ... 20: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1
@@ -123,6 +130,9 @@ Property migration_properties[] = {DEFINE_PROP_UINT8 ("multifd-zlib-level" , MigrationState, parameters.multifd_zlib_level,DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL ), +DEFINE_PROP_UINT8 ("multifd-qatzip-level" , MigrationState, + parameters.multifd_qatzip_level, +DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL ),DEFINE_PROP_UINT8 ("multifd-zstd-level" , MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),
@@ -787,6 +797,13 @@ int migrate_multifd_zlib_level(void)return s->parameters.multifd_zlib_level; } +int migrate_multifd_qatzip_level (void ) +{ +MigrationState *s =migrate_get_current (); + +return s->parameters.multifd_qatzip_level; +} +int migrate_multifd_zstd_level (void ) {MigrationState *s =migrate_get_current ();
@@ -892,6 +909,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_compression = s->parameters.multifd_compression; params->has_multifd_zlib_level = true; params->multifd_zlib_level = s->parameters.multifd_zlib_level; + params->has_multifd_qatzip_level = true; + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; params->has_xbzrle_cache_size = true;
@@ -946,6 +965,7 @@ void migrate_params_init(MigrationParameters *params) params->has_multifd_channels = true; params->has_multifd_compression = true; params->has_multifd_zlib_level = true; + params->has_multifd_qatzip_level = true; params->has_multifd_zstd_level = true; params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true;
@@ -1038,6 +1058,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)return false; } +if (params->has_multifd_qatzip_level && + ((params->multifd_qatzip_level >9 ) || + (params->multifd_qatzip_level <1 ))) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"multifd_qatzip_level" , +"a value between 1 and 9" ); +return false; + } +if (params->has_multifd_zstd_level && (params->multifd_zstd_level >20 )) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level",
@@ -1195,6 +1223,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + dest->multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { dest->multifd_zlib_level = params->multifd_zlib_level; }
@@ -1315,6 +1346,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + s->parameters.multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { s->parameters.multifd_zlib_level = params->multifd_zlib_level; }
migration/options.h +1 -0
@@ -78,6 +78,7 @@ uint64_t migrate_max_postcopy_bandwidth(void);int migrate_multifd_channels (void );MultiFDCompression migrate_multifd_compression (void );int migrate_multifd_zlib_level (void ); +int migrate_multifd_qatzip_level (void );int migrate_multifd_zstd_level (void );uint8_t migrate_throttle_trigger_threshold (void );const char *migrate_tls_authz (void );
qapi/migration.json +18 -0
@@ -792,6 +792,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -852,6 +857,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', + 'multifd-qatzip-level', 'block-bitmap-mapping', { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit',
@@ -967,6 +973,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -1040,6 +1051,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
@@ -1171,6 +1183,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression
@@ -1241,6 +1258,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',
Signature · 1 line
Yichen Wang
Re: [PATCH v8 3/5] migration: Add migration parameters for QATzip
Quoted 4 lines · Yichen Wang
Adds support for migration parameters to control QATzip compression level and to enable/disable software fallback when QAT hardware is unavailable. This is a preparatory commit for a subsequent commit that will actually use QATzip compression.
* Is the check whether "QAT hardware is available" happening in this patch? (I couldn't spot it).
* The informatory notice "This is a preparatory commit for..." could be moved to the cover-letter, instead of commit message. (Not sure how it helps to log it in git history)
Quoted 197 lines
Signed-off-by Bryan Zhang <[email protected]>Signed-off-by Hao Xiang <[email protected]>Signed-off-by Yichen Wang <[email protected]>--- migration/migration-hmp-cmds.c | 4 ++++ migration/options.c | 34 ++++++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 ++++++++++++++++++ 4 files changed, 57 insertions(+)migration/migration-hmp-cmds.c +4
-0@@ -576,6 +576,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zlib_level = true;visit_type_uint8 (v, param, &p->multifd_zlib_level, &err);break ; +case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL : + p->has_multifd_qatzip_level = true; +visit_type_uint8 (v, param, &p->multifd_qatzip_level, &err); +break ;case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL : p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);migration/options.c +34
-0@@ -55,6 +55,13 @@#define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE/* 0: means nocompress, 1: best speed, ... 9: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1 +/* +* 1: best speed, ... 9: best compress ratio +* There is some nuance here. Refer to QATzip documentation to understand +* the mapping of QATzip levels to standard deflate levels. +*/ +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1 +/* 0: means nocompress, 1: best speed, ... 20: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1@@ -123,6 +130,9 @@ Property migration_properties[] = {DEFINE_PROP_UINT8 ("multifd-zlib-level" , MigrationState, parameters.multifd_zlib_level,DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL ), +DEFINE_PROP_UINT8 ("multifd-qatzip-level" , MigrationState, + parameters.multifd_qatzip_level, +DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL ),DEFINE_PROP_UINT8 ("multifd-zstd-level" , MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),@@ -787,6 +797,13 @@ int migrate_multifd_zlib_level(void)return s->parameters.multifd_zlib_level; } +int migrate_multifd_qatzip_level (void ) +{ +MigrationState *s =migrate_get_current (); + +return s->parameters.multifd_qatzip_level; +} +int migrate_multifd_zstd_level (void ) {MigrationState *s =migrate_get_current ();@@ -892,6 +909,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_compression = s->parameters.multifd_compression; params->has_multifd_zlib_level = true; params->multifd_zlib_level = s->parameters.multifd_zlib_level; + params->has_multifd_qatzip_level = true; + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; params->has_xbzrle_cache_size = true;@@ -946,6 +965,7 @@ void migrate_params_init(MigrationParameters *params) params->has_multifd_channels = true; params->has_multifd_compression = true; params->has_multifd_zlib_level = true; + params->has_multifd_qatzip_level = true; params->has_multifd_zstd_level = true; params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true;@@ -1038,6 +1058,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)return false; } +if (params->has_multifd_qatzip_level && + ((params->multifd_qatzip_level >9 ) || + (params->multifd_qatzip_level <1 ))) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"multifd_qatzip_level" , +"a value between 1 and 9" ); +return false; + } +if (params->has_multifd_zstd_level && (params->multifd_zstd_level >20 )) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level",@@ -1195,6 +1223,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + dest->multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { dest->multifd_zlib_level = params->multifd_zlib_level; }@@ -1315,6 +1346,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + s->parameters.multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { s->parameters.multifd_zlib_level = params->multifd_zlib_level; }migration/options.h +1
-0@@ -78,6 +78,7 @@ uint64_t migrate_max_postcopy_bandwidth(void);int migrate_multifd_channels (void );MultiFDCompression migrate_multifd_compression (void );int migrate_multifd_zlib_level (void ); +int migrate_multifd_qatzip_level (void );int migrate_multifd_zstd_level (void );uint8_t migrate_throttle_trigger_threshold (void );const char *migrate_tls_authz (void );qapi/migration.json +18
-1@@ -792,6 +792,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -852,6 +857,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', + 'multifd-qatzip-level', 'block-bitmap-mapping', { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit',@@ -967,6 +973,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -1040,6 +1051,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',@@ -1171,6 +1183,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -1241,6 +1258,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', --
Quoted 1 line
Yichen Wang
'multifd-qatzip-level' related changes look okay.
Thank you.
---
- Prasad
Re: [External] Re: [PATCH v8 3/5] migration: Add migration parameters for QATzip
Quoted 9 lines · Prasad Pandit
Quoted 4 lines · Yichen Wang
Adds support for migration parameters to control QATzip compression level and to enable/disable software fallback when QAT hardware is unavailable. This is a preparatory commit for a subsequent commit that will actually use QATzip compression.
* Is the check whether "QAT hardware is available" happening in this patch? (I couldn't spot it).
After discussing with Intel folks, I decided to align to the existing QPL behavior. In QPL, the code path of compression will always go through regardless. When acceleration hardware is initialized properly, use it. If failed, fallback to software path automatically. So in this QAT case, I do the same. The line of "ret = qzInit(&q->sess, true);" will do the auto software fallback.
Quoted 4 lines
* The informatory notice "This is a preparatory commit for..." could be moved to the cover-letter, instead of commit message. (Not sure how it helps to log it in git history)
Oh, this line is purely for breaking the big patch into two commits. This one plus the following commit [4/5] together implements the full feature. I can remove this from the commit message if you prefer.
Quoted 206 lines
Quoted 197 lines
Signed-off-by Bryan Zhang <[email protected]>Signed-off-by Hao Xiang <[email protected]>Signed-off-by Yichen Wang <[email protected]>--- migration/migration-hmp-cmds.c | 4 ++++ migration/options.c | 34 ++++++++++++++++++++++++++++++++++ migration/options.h | 1 + qapi/migration.json | 18 ++++++++++++++++++ 4 files changed, 57 insertions(+)migration/migration-hmp-cmds.c +4
-0@@ -576,6 +576,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zlib_level = true;visit_type_uint8 (v, param, &p->multifd_zlib_level, &err);break ; +case MIGRATION_PARAMETER_MULTIFD_QATZIP_LEVEL : + p->has_multifd_qatzip_level = true; +visit_type_uint8 (v, param, &p->multifd_qatzip_level, &err); +break ;case MIGRATION_PARAMETER_MULTIFD_ZSTD_LEVEL : p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);migration/options.c +34
-0@@ -55,6 +55,13 @@#define DEFAULT_MIGRATE_MULTIFD_COMPRESSION MULTIFD_COMPRESSION_NONE/* 0: means nocompress, 1: best speed, ... 9: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL 1 +/* +* 1: best speed, ... 9: best compress ratio +* There is some nuance here. Refer to QATzip documentation to understand +* the mapping of QATzip levels to standard deflate levels. +*/ +#define DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL 1 +/* 0: means nocompress, 1: best speed, ... 20: best compress ratio */ #define DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL 1@@ -123,6 +130,9 @@ Property migration_properties[] = {DEFINE_PROP_UINT8 ("multifd-zlib-level" , MigrationState, parameters.multifd_zlib_level,DEFAULT_MIGRATE_MULTIFD_ZLIB_LEVEL ), +DEFINE_PROP_UINT8 ("multifd-qatzip-level" , MigrationState, + parameters.multifd_qatzip_level, +DEFAULT_MIGRATE_MULTIFD_QATZIP_LEVEL ),DEFINE_PROP_UINT8 ("multifd-zstd-level" , MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL),@@ -787,6 +797,13 @@ int migrate_multifd_zlib_level(void)return s->parameters.multifd_zlib_level; } +int migrate_multifd_qatzip_level (void ) +{ +MigrationState *s =migrate_get_current (); + +return s->parameters.multifd_qatzip_level; +} +int migrate_multifd_zstd_level (void ) {MigrationState *s =migrate_get_current ();@@ -892,6 +909,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_compression = s->parameters.multifd_compression; params->has_multifd_zlib_level = true; params->multifd_zlib_level = s->parameters.multifd_zlib_level; + params->has_multifd_qatzip_level = true; + params->multifd_qatzip_level = s->parameters.multifd_qatzip_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; params->has_xbzrle_cache_size = true;@@ -946,6 +965,7 @@ void migrate_params_init(MigrationParameters *params) params->has_multifd_channels = true; params->has_multifd_compression = true; params->has_multifd_zlib_level = true; + params->has_multifd_qatzip_level = true; params->has_multifd_zstd_level = true; params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true;@@ -1038,6 +1058,14 @@ bool migrate_params_check(MigrationParameters *params, Error **errp)return false; } +if (params->has_multifd_qatzip_level && + ((params->multifd_qatzip_level >9 ) || + (params->multifd_qatzip_level <1 ))) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"multifd_qatzip_level" , +"a value between 1 and 9" ); +return false; + } +if (params->has_multifd_zstd_level && (params->multifd_zstd_level >20 )) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "multifd_zstd_level",@@ -1195,6 +1223,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + dest->multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { dest->multifd_zlib_level = params->multifd_zlib_level; }@@ -1315,6 +1346,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } +if (params->has_multifd_qatzip_level) { + s->parameters.multifd_qatzip_level = params->multifd_qatzip_level; + }if (params->has_multifd_zlib_level) { s->parameters.multifd_zlib_level = params->multifd_zlib_level; }migration/options.h +1
-0@@ -78,6 +78,7 @@ uint64_t migrate_max_postcopy_bandwidth(void);int migrate_multifd_channels (void );MultiFDCompression migrate_multifd_compression (void );int migrate_multifd_zlib_level (void ); +int migrate_multifd_qatzip_level (void );int migrate_multifd_zstd_level (void );uint8_t migrate_throttle_trigger_threshold (void );const char *migrate_tls_authz (void );qapi/migration.json +18
-1@@ -792,6 +792,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -852,6 +857,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level', 'multifd-zstd-level', + 'multifd-qatzip-level', 'block-bitmap-mapping', { 'name': 'x-vcpu-dirty-limit-period', 'features': ['unstable'] }, 'vcpu-dirty-limit',@@ -967,6 +973,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -1040,6 +1051,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64',@@ -1171,6 +1183,11 @@ # speed, and 9 means best compression ratio which will consume # more CPU. Defaults to 1. (Since 5.0) # +# @multifd-qatzip-level: Set the compression level to be used in live +# migration. The level is an integer between 1 and 9, where 1 means +# the best compression speed, and 9 means the best compression +# ratio which will consume more CPU. Defaults to 1. (Since 9.2) +# # @multifd-zstd-level: Set the compression level to be used in live # migration, the compression level is an integer between 0 and 20, # where 0 means no compression, 1 means the best compression@@ -1241,6 +1258,7 @@ '*max-cpu-throttle': 'uint8', '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', + '*multifd-qatzip-level': 'uint8', '*multifd-zstd-level': 'uint8', '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ], '*x-vcpu-dirty-limit-period': { 'type': 'uint64', --Quoted 1 line
Yichen Wang
'multifd-qatzip-level' related changes look okay.
Reviewed-by Prasad Pandit <[email protected]>Thank you.
---
- Prasad
Re: [External] Re: [PATCH v8 3/5] migration: Add migration parameters for QATzip
Hi,
Quoted 6 lines · Yichen Wang
After discussing with Intel folks, I decided to align to the existing QPL behavior. In QPL, the code path of compression will always go through regardless. When acceleration hardware is initialized properly, use it. If failed, fallback to software path automatically. So in this QAT case, I do the same. The line of "ret =
qzInit(&q->sess, true);" will do the auto software fallback.
* I see.
Quoted 3 lines
Oh, this line is purely for breaking the big patch into two commits. This one plus the following commit [4/5] together implements the full feature. I can remove this from the commit message if you prefer.
* Yes, that'll be nice.
Thank you.
---
- Prasad
PATCH v8 4/5 migration: Introduce 'qatzip' compression method
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds support for 'qatzip' as an option for the multifd compression method parameter, and implements using QAT for 'qatzip' compression and decompression.
--- hw/core/qdev-properties-system.c | 2 +- migration/meson.build | 1 + migration/multifd-qatzip.c | 394 +++++++++++++++++++++++++++++++ migration/multifd.h | 5 +- qapi/migration.json | 3 + 5 files changed, 402 insertions(+), 3 deletions(-) create mode 100644 migration/multifd-qatzip.c
hw/core/qdev-properties-system.c +1 -1
@@ -659,7 +659,7 @@ const PropertyInfo qdev_prop_fdc_drive_type = {const PropertyInfo qdev_prop_multifd_compression = { .name ="MultiFDCompression" , .description ="multifd_compression values, " -"none/zlib/zstd/qpl/uadk" , +"none/zlib/zstd/qpl/uadk/qatzip" , .enum_table = &MultiFDCompression_lookup, .get = qdev_propinfo_get_enum, .set = qdev_propinfo_set_enum,
migration/meson.build +1 -0
@@ -41,6 +41,7 @@ system_ss.add(when: rdma, if_true: files('rdma.c')) system_ss.add(when: zstd, if_true: files('multifd-zstd.c')) system_ss.add(when: qpl, if_true: files('multifd-qpl.c')) system_ss.add(when: uadk, if_true: files('multifd-uadk.c')) +system_ss.add(when: qatzip, if_true: files('multifd-qatzip.c')) specific_ss.add(when: 'CONFIG_SYSTEM_ONLY', if_true: files('ram.c',
migration/multifd-qatzip.c new file +394 -0
394 lines not on this page · raw message
migration/multifd.h +3 -2
@@ -34,14 +34,15 @@ MultiFDRecvData *multifd_get_recv_data(void);/* Multifd Compression flags */ #define MULTIFD_FLAG_SYNC (1 << 0) -/* We reserve 4 bits for compression methods */ -#define MULTIFD_FLAG_COMPRESSION_MASK (0xf << 1) +/* We reserve 5 bits for compression methods */ +#define MULTIFD_FLAG_COMPRESSION_MASK (0x1f << 1)/* we need to be compatible. Before compression value was 0 */ #define MULTIFD_FLAG_NOCOMP (0 << 1)#define MULTIFD_FLAG_ZLIB (1 << 1)#define MULTIFD_FLAG_ZSTD (2 << 1)#define MULTIFD_FLAG_QPL (4 << 1)#define MULTIFD_FLAG_UADK (8 << 1) +#define MULTIFD_FLAG_QATZIP (16 << 1)/* This value needs to be a multiple of qemu_target_page_size() */ #define MULTIFD_PACKET_SIZE (512 * 1024)
qapi/migration.json +3 -0
@@ -561,6 +561,8 @@ # # @zstd: use zstd compression method. # +# @qatzip: use qatzip compression method. (Since 9.2) +# # @qpl: use qpl compression method. Query Processing Library(qpl) is # based on the deflate compression algorithm and use the Intel # In-Memory Analytics Accelerator(IAA) accelerated compression and
@@ -573,6 +575,7 @@ { 'enum': 'MultiFDCompression', 'data': [ 'none', 'zlib', { 'name': 'zstd', 'if': 'CONFIG_ZSTD' }, + { 'name': 'qatzip', 'if': 'CONFIG_QATZIP'}, { 'name': 'qpl', 'if': 'CONFIG_QPL' }, { 'name': 'uadk', 'if': 'CONFIG_UADK' } ] }
Signature · 1 line
Yichen Wang
Re: [PATCH v8 4/5] migration: Introduce 'qatzip' compression method
Hello,
Quoted 63 lines · Yichen Wang
migration/multifd-qatzip.c new file +63
-0+static int qatzip_send_setup (MultiFDSendParams *p,Error **errp) +{ +QatzipData *q; +QzSessionParamsDeflate_T params; +const char *err_msg; +int ret; + + q =g_new0 (QatzipData,1 ); + p->compress_data = q; +/* We need one extra place for the packet header */ + p->iov = g_new0(struct iovec, 2); + +/* +* Initialize QAT device with software fallback by default. This allows +* QATzip to use CPU path when QAT hardware reaches maximum throughput. +*/ + ret =qzInit (&q->sess, true); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzInit failed" ; + goto err; + } + + ret =qzGetDefaultsDeflate (¶ms); +if (ret !=QZ_OK ) { + err_msg ="qzGetDefaultsDeflate failed" ; + goto err; + } + +/* Make sure to use configured QATzip compression level. */ + params.common_params.comp_lvl =migrate_multifd_qatzip_level (); + ret =qzSetupSessionDeflate (&q->sess, ¶ms); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzSetupSessionDeflate failed" ; + goto err; + } + +if (MULTIFD_PACKET_SIZE >UINT32_MAX ) { + err_msg ="packet size too large for QAT" ; + goto err; + } + + q->in_len =MULTIFD_PACKET_SIZE ; +/* +* PINNED_MEM is an enum from qatzip headers, which means to use +* kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device +* is not available or software fallback is used, the malloc flag needs to +* be set as COMMON_MEM. +*/ + q->in_buf =qzMalloc (q->in_len,0 ,PINNED_MEM ); +if (!q->in_buf) { + q->in_buf =qzMalloc (q->in_len,0 ,COMMON_MEM ); +if (!q->in_buf) { + err_msg ="qzMalloc failed" ; + goto err; + } + } + + q->out_len =qzMaxCompressedLength (MULTIFD_PACKET_SIZE , &q->sess); + q->out_buf =qzMalloc (q->out_len,0 ,PINNED_MEM ); +if (!q->out_buf) { + q->out_buf =qzMalloc (q->out_len,0 ,COMMON_MEM ); +if (!q->out_buf) { + err_msg ="qzMalloc failed" ;
migration/multifd-qatzip.c line 124
Quoted 10 lines
migration/multifd-qatzip.c new file +10
-0+ goto err; + } + } + +return 0 ; + +err: +error_setg (errp,"multifd %u: [sender] %s" , p->id, err_msg); +return -1 ; +}
* Need to release (g_free OR qatzip_send_cleanup) allocated memory in the error (err:) path.
Quoted 58 lines
migration/multifd-qatzip.c new file +58
-0+static int qatzip_recv_setup (MultiFDRecvParams *p,Error **errp) +{ +QatzipData *q; +QzSessionParamsDeflate_T params; +const char *err_msg; +int ret; + + q =g_new0 (QatzipData,1 ); + p->compress_data = q; + +/* +* Initialize QAT device with software fallback by default. This allows +* QATzip to use CPU path when QAT hardware reaches maximum throughput. +*/ + ret =qzInit (&q->sess, true); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzInit failed" ; + goto err; + } + + ret =qzGetDefaultsDeflate (¶ms); +if (ret !=QZ_OK ) { + err_msg ="qzGetDefaultsDeflate failed" ; + goto err; + } + + ret =qzSetupSessionDeflate (&q->sess, ¶ms); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzSetupSessionDeflate failed" ; + goto err; + } + +/* +* Reserve extra spaces for the incoming packets. Current implementation +* doesn't send uncompressed pages in case the compression gets too big. +*/ + q->in_len =MULTIFD_PACKET_SIZE *2 ; +/* +* PINNED_MEM is an enum from qatzip headers, which means to use +* kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device +* is not available or software fallback is used, the malloc flag needs to +* be set as COMMON_MEM. +*/ + q->in_buf =qzMalloc (q->in_len,0 ,PINNED_MEM ); +if (!q->in_buf) { + q->in_buf =qzMalloc (q->in_len,0 ,COMMON_MEM ); +if (!q->in_buf) { + err_msg ="qzMalloc failed" ; + goto err; + } + } + + q->out_len =MULTIFD_PACKET_SIZE ; + q->out_buf =qzMalloc (q->out_len,0 ,PINNED_MEM ); +if (!q->out_buf) { + q->out_buf =qzMalloc (q->out_len,0 ,COMMON_MEM ); +if (!q->out_buf) { + err_msg ="qzMalloc failed" ;
migration/multifd-qatzip.c line 291
Quoted 10 lines
migration/multifd-qatzip.c new file +10
-0+ goto err; + } + } + +return 0 ; + +err: +error_setg (errp,"multifd %u: [receiver] %s" , p->id, err_msg); +return -1 ; +}
* Need to release (g_free OR qatzip_recv_cleanup) allocated memory in the error (err:) path.
Thank you.
---
- Prasad
Re: [External] Re: [PATCH v8 4/5] migration: Introduce 'qatzip' compression method
Quoted 72 lines · Prasad Pandit
Hello,
Quoted 68 lines · Yichen Wang
migration/multifd-qatzip.c new file +68
-0+static int qatzip_send_setup (MultiFDSendParams *p,Error **errp) +{ +QatzipData *q; +QzSessionParamsDeflate_T params; +const char *err_msg; +int ret; + + q =g_new0 (QatzipData,1 ); + p->compress_data = q; +/* We need one extra place for the packet header */ + p->iov = g_new0(struct iovec, 2); + +/* +* Initialize QAT device with software fallback by default. This allows +* QATzip to use CPU path when QAT hardware reaches maximum throughput. +*/ + ret =qzInit (&q->sess, true); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzInit failed" ; + goto err; + } + + ret =qzGetDefaultsDeflate (¶ms); +if (ret !=QZ_OK ) { + err_msg ="qzGetDefaultsDeflate failed" ; + goto err; + } + +/* Make sure to use configured QATzip compression level. */ + params.common_params.comp_lvl =migrate_multifd_qatzip_level (); + ret =qzSetupSessionDeflate (&q->sess, ¶ms); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzSetupSessionDeflate failed" ; + goto err; + } + +if (MULTIFD_PACKET_SIZE >UINT32_MAX ) { + err_msg ="packet size too large for QAT" ; + goto err; + } + + q->in_len =MULTIFD_PACKET_SIZE ; +/* +* PINNED_MEM is an enum from qatzip headers, which means to use +* kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device +* is not available or software fallback is used, the malloc flag needs to +* be set as COMMON_MEM. +*/ + q->in_buf =qzMalloc (q->in_len,0 ,PINNED_MEM ); +if (!q->in_buf) { + q->in_buf =qzMalloc (q->in_len,0 ,COMMON_MEM ); +if (!q->in_buf) { + err_msg ="qzMalloc failed" ; + goto err; + } + } + + q->out_len =qzMaxCompressedLength (MULTIFD_PACKET_SIZE , &q->sess); + q->out_buf =qzMalloc (q->out_len,0 ,PINNED_MEM ); +if (!q->out_buf) { + q->out_buf =qzMalloc (q->out_len,0 ,COMMON_MEM ); +if (!q->out_buf) { + err_msg ="qzMalloc failed" ; + goto err; + } + } + +return 0 ;
Quoted 9 lines
migration/multifd-qatzip.c line 124
Quoted 5 lines
migration/multifd-qatzip.c new file +5
-0+ +err: +error_setg (errp,"multifd %u: [sender] %s" , p->id, err_msg); +return -1 ; +}* Need to release (
g_freeORqatzip_send_cleanup) allocated memory in the error (err:) path.
The patch was originally written exactly like what you suggest, cleanup in the error path of the same function. However, later I realized in gdb that I was wrong. The qatzip_send_cleanup() function will be called later in another thread in both normal and error paths. So I revised the patch to this behavior, otherwise I will run into double free in the error path.
Quoted 77 lines
Quoted 59 lines
migration/multifd-qatzip.c new file +58
-0+static int qatzip_recv_setup (MultiFDRecvParams *p,Error **errp) +{ +QatzipData *q; +QzSessionParamsDeflate_T params; +const char *err_msg; +int ret; + + q =g_new0 (QatzipData,1 ); + p->compress_data = q; + +/* +* Initialize QAT device with software fallback by default. This allows +* QATzip to use CPU path when QAT hardware reaches maximum throughput. +*/ + ret =qzInit (&q->sess, true); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzInit failed" ; + goto err; + } + + ret =qzGetDefaultsDeflate (¶ms); +if (ret !=QZ_OK ) { + err_msg ="qzGetDefaultsDeflate failed" ; + goto err; + } + + ret =qzSetupSessionDeflate (&q->sess, ¶ms); +if (ret !=QZ_OK && ret !=QZ_DUPLICATE ) { + err_msg ="qzSetupSessionDeflate failed" ; + goto err; + } + +/* +* Reserve extra spaces for the incoming packets. Current implementation +* doesn't send uncompressed pages in case the compression gets too big. +*/ + q->in_len =MULTIFD_PACKET_SIZE *2 ; +/* +* PINNED_MEM is an enum from qatzip headers, which means to use +* kzalloc_node() to allocate memory for QAT DMA purposes. When QAT device +* is not available or software fallback is used, the malloc flag needs to +* be set as COMMON_MEM. +*/ + q->in_buf =qzMalloc (q->in_len,0 ,PINNED_MEM ); +if (!q->in_buf) { + q->in_buf =qzMalloc (q->in_len,0 ,COMMON_MEM ); +if (!q->in_buf) { + err_msg ="qzMalloc failed" ; + goto err; + } + } + + q->out_len =MULTIFD_PACKET_SIZE ; + q->out_buf =qzMalloc (q->out_len,0 ,PINNED_MEM ); +if (!q->out_buf) { + q->out_buf =qzMalloc (q->out_len,0 ,COMMON_MEM ); +if (!q->out_buf) { + err_msg ="qzMalloc failed" ;migration/multifd-qatzip.c line 291
Quoted 10 lines
migration/multifd-qatzip.c new file +10
-0+ goto err; + } + } + +return 0 ; + +err: +error_setg (errp,"multifd %u: [receiver] %s" , p->id, err_msg); +return -1 ; +}* Need to release (
g_freeORqatzip_recv_cleanup) allocated memory in the error (err:) path.Thank you.
---
- Prasad
Prasad Pandit
Reviewed-byRe: [External] Re: [PATCH v8 4/5] migration: Introduce 'qatzip' compression method
Quoted 7 lines · Yichen Wang
The patch was originally written exactly like what you suggest, cleanup in the error path of the same function. However, later I realized in gdb that I was wrong. The
qatzip_send_cleanup()function will be called later in another thread in both normal and error paths. So I revised the patch to this behavior, otherwise I will run into double free in the error path.
* I see, okay, in that case:
Thank you.
---
- Prasad
PATCH v8 5/5 tests/migration: Add integration test for 'qatzip' compression method
In reply to Yichen WangFrom: Bryan Zhang <[email protected]>
Adds an integration test for 'qatzip'.
--- tests/qtest/migration-test.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
tests/qtest/migration-test.c +27 -0
@@ -2992,6 +2992,18 @@ test_migrate_precopy_tcp_multifd_zstd_start(QTestState *from, }#endif /* CONFIG_ZSTD */ +#ifdef CONFIG_QATZIP +static void * +test_migrate_precopy_tcp_multifd_qatzip_start (QTestState *from, +QTestState *to) +{ +migrate_set_parameter_int (from,"multifd-qatzip-level" ,2 ); +migrate_set_parameter_int (to,"multifd-qatzip-level" ,2 ); + +return test_migrate_precopy_tcp_multifd_start_common (from, to,"qatzip" ); +} +#endif +#ifdef CONFIG_QPL static void * test_migrate_precopy_tcp_multifd_qpl_start(QTestState *from,
@@ -3089,6 +3101,17 @@ static void test_multifd_tcp_zstd(void) }#endif +#ifdef CONFIG_QATZIP +static void test_multifd_tcp_qatzip (void ) +{ +MigrateCommon args = { + .listen_uri ="defer" , + .start_hook = test_migrate_precopy_tcp_multifd_qatzip_start, + }; +test_precopy_common (&args); +} +#endif +#ifdef CONFIG_QPL static void test_multifd_tcp_qpl (void ) {
@@ -3992,6 +4015,10 @@ int main(int argc, char **argv)migration_test_add ("/migration/multifd/tcp/plain/zstd" , test_multifd_tcp_zstd);#endif +#ifdef CONFIG_QATZIP +migration_test_add ("/migration/multifd/tcp/plain/qatzip" , + test_multifd_tcp_qatzip); +#endif #ifdef CONFIG_QPL migration_test_add ("/migration/multifd/tcp/plain/qpl" , test_multifd_tcp_qpl);
Signature · 1 line
Yichen Wang