MSG_ZEROCOPY + multifd
v6 cover MSG_ZEROCOPY + multifd
This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make use of it for multifd migration performance improvement, by reducing cpu usage.
Patch #1 creates new callbacks for QIOChannel, allowing the implementation of zero copy writing.
Patch #2 reworks qio_channel_socket_writev() so it accepts flags for that are later passed to sendmsg().
Patch #3 implements writev_zero_copy and flush_zero_copy on QIOChannelSocket, making use of MSG_ZEROCOPY on Linux.
Patch #4 adds a "zero_copy" migration property, only available with CONFIG_LINUX, and compiled-out in any other architectures. This migration property has to be enabled before multifd migration starts.
Patch #5 adds a helper function that allows to see if TLS is going to be used. This helper will be later used in patch #6.
Patch #6 Makes use of QIOChannelSocket zero_copy implementation on nocomp multifd migration.
Results:
In preliminary tests, the resource usage of __sys_sendmsg() reduced 15 times, and the overall migration took 13-22% less time, based in synthetic cpu workload.
In further tests, it was noted that, on multifd migration with 8 channels:
- On idle hosts, migration time reduced in 10% to 21%.
- On hosts busy with heavy cpu stress (1 stress thread per cpu, but not cpu-pinned) migration time reduced in ~25% by enabling zero-copy.
- On hosts with heavy cpu-pinned workloads (1 stress thread per cpu, cpu-pinned), migration time reducted in ~66% by enabling zero-copy.
Above tests setup:
- Sending and Receiving hosts:
- CPU : Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz (448 CPUS)
- Network card: E810-C (100Gbps)
- >1TB RAM
- QEMU: Upstream master branch + This patchset
- Linux: Upstream v5.15
- VM configuration:
- 28 VCPUs
- 512GB RAM
---
Changes since v5:
- flush_zero_copy now returns -1 on fail, 0 on success, and 1 when all processed writes were not able to use zerocopy in kernel.
- qio_channel_socket_poll() removed, using qio_channel_wait() instead
- ENOBUFS is now processed inside qio_channel_socket_writev_flags()
- Most zerocopy parameter validation moved to migrate_params_check(), leaving only feature test to socket_outgoing_migration() callback
- Naming went from *zerocopy to *zero_copy or *zero-copy, due to QAPI/QMP preferences
- Improved docs
Changes since v4:
- 3 patches got splitted in 6
- Flush is used for syncing after each iteration, instead of only at the end
- If zerocopy is not available, fail in connect instead of failing on write
- 'multifd-zerocopy' property renamed to 'zerocopy'
- Fail migrations that don't support zerocopy, if it's enabled.
- Instead of checking for zerocopy at each write, save the flags in MultiFDSendParams->write_flags and use them on write
- Reorganized flag usage in QIOChannelSocket
- A lot of typos fixed
- More doc on buffer restrictions
Changes since v3:
- QIOChannel interface names changed from io_async_{writev,flush} to io_{writev,flush}_zerocopy
- Instead of falling back in case zerocopy is not implemented, return error and abort operation.
- Flush now waits as long as needed, or return error in case anything goes wrong, aborting the operation.
- Zerocopy is now conditional in multifd, being set by parameter multifd-zerocopy
- Moves zerocopy_flush to multifd_send_sync_main() from multifd_save_cleanup so migration can abort if flush goes wrong.
- Several other small improvements
Changes since v2:
- Patch #1: One more fallback
- Patch #2: Fall back to sync if fails to lock buffer memory in MSG_ZEROCOPY send.
Changes since v1:
- Reimplemented the patchset using async_write + async_flush approach.
- Implemented a flush to be able to tell whenever all data was written.
Leonardo Bras (6):
QIOChannel: Add io_writev_zero_copy & io_flush_zero_copy callbacks
QIOChannelSocket: Add flags parameter for writing
QIOChannelSocket: Implement io_writev_zero_copy & io_flush_zero_copy
for CONFIG_LINUX
migration: Add zero-copy parameter for QMP/HMP for Linux
migration: Add migrate_use_tls() helper
multifd: Implement zero copy write in multifd migration
(multifd-zero-copy)
qapi/migration.json | 24 ++++++ include/io/channel-socket.h | 2 + include/io/channel.h | 98 +++++++++++++++++++++--- migration/migration.h | 6 ++ migration/multifd.h | 4 +- io/channel-socket.c | 145 +++++++++++++++++++++++++++++++++--- io/channel.c | 66 +++++++++++++--- migration/channel.c | 6 +- migration/migration.c | 49 ++++++++++++ migration/multifd.c | 45 ++++++++--- migration/ram.c | 29 ++++++-- migration/socket.c | 6 ++ monitor/hmp-cmds.c | 6 ++ 13 files changed, 434 insertions(+), 52 deletions(-)
Signature · 1 line
2.33.1
PATCH v6 1/6 QIOChannel: Add io_writev_zero_copy & io_flush_zero_copy callbacks
Adds io_writev_zero_copy and io_flush_zero_copy as optional callback to QIOChannelClass, allowing the implementation of zero copy writes by subclasses.
How to use them:
- Write data using qio_channel_writev_zero_copy(),
- Wait write completion with qio_channel_flush_zero_copy().
Notes:
As some zero copy implementations work asynchronously, it's recommended to keep the write buffer untouched until the return of qio_channel_flush_zero_copy(), to avoid the risk of sending an updated buffer instead of the one at the write.
As the new callbacks are optional, if a subclass does not implement them, then:
- io_writev_zero_copy will return -1,
- io_flush_zero_copy will return 0 without changing anything.
Also, some functions like qio_channel_writev_full_all() were adapted to receive a flag parameter. That allows shared code between zero copy and non-zero copy writev, and also an easier implementation on new flags.
--- include/io/channel.h | 98 +++++++++++++++++++++++++++++++++++++++----- io/channel.c | 66 +++++++++++++++++++++++------ 2 files changed, 142 insertions(+), 22 deletions(-)
include/io/channel.h +88 -10
@@ -32,12 +32,15 @@ OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass,#define QIO_CHANNEL_ERR_BLOCK -2 +#define QIO_CHANNEL_WRITE_FLAG_ZERO_COPY 0x1 +typedef enum QIOChannelFeature QIOChannelFeature ;enum QIOChannelFeature {QIO_CHANNEL_FEATURE_FD_PASS ,QIO_CHANNEL_FEATURE_SHUTDOWN ,QIO_CHANNEL_FEATURE_LISTEN , +QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY , };
@@ -136,6 +139,12 @@ struct QIOChannelClass { IOHandler *io_read, IOHandler *io_write, void *opaque); +ssize_t (*io_writev_zero_copy)(QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +Error **errp); +int (*io_flush_zero_copy)(QIOChannel *ioc, +Error **errp); };/* General I/O handling functions */
@@ -321,10 +330,11 @@ int qio_channel_readv_all(QIOChannel *ioc,/** -* qio_channel_writev_all: +* qio_channel_writev_all_flags: * @ioc: the channel object * @iov: the array of memory regions to write data from * @niov: the length of the @iov array +* @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*) * @errp: pointer to a NULL-initialized error object * * Write data to the IO channel, reading it from the
@@ -337,12 +347,23 @@ int qio_channel_readv_all(QIOChannel *ioc,* to be written, yielding from the current coroutine * if required. * +* If QIO_CHANNEL_WRITE_FLAG_ZERO_COPY is passed in flags, +* instead of waiting for all requested data to be written, +* this function will wait until it's all queued for writing. +* In this case, if the buffer gets changed between queueing and +* sending, the updated buffer will be sent. If this is not a +* desired behavior, it's suggested to call qio_channel_flush_zero_copy() +* before reusing the buffer. +* * Returns: 0 if all bytes were written, or -1 on error */ -int qio_channel_writev_all (QIOChannel *ioc, -const struct iovec *iov, -size_t niov, -Error **erp); +int qio_channel_writev_all_flags (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +int flags, +Error **errp); +#define qio_channel_writev_all (ioc, iov, niov, errp) \ + qio_channel_writev_all_flags(ioc, iov, niov, 0, errp)/** * qio_channel_readv:
@@ -831,12 +852,13 @@ int qio_channel_readv_full_all(QIOChannel *ioc, Error **errp);/** -* qio_channel_writev_full_all: +* qio_channel_writev_full_all_flags: * @ioc: the channel object * @iov: the array of memory regions to write data from * @niov: the length of the @iov array * @fds: an array of file handles to send * @nfds: number of file handles in @fds +* @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*) * @errp: pointer to a NULL-initialized error object * *
@@ -846,13 +868,69 @@ int qio_channel_readv_full_all(QIOChannel *ioc,* to be written, yielding from the current coroutine * if required. * +* If QIO_CHANNEL_WRITE_FLAG_ZERO_COPY is passed in flags, +* instead of waiting for all requested data to be written, +* this function will wait until it's all queued for writing. +* In this case, if the buffer gets changed between queueing and +* sending, the updated buffer will be sent. If this is not a +* desired behavior, it's suggested to call qio_channel_flush_zero_copy() +* before reusing the buffer. +* * Returns: 0 if all bytes were written, or -1 on error */ -int qio_channel_writev_full_all (QIOChannel *ioc, -const struct iovec *iov, -size_t niov, -int *fds,size_t nfds, +int qio_channel_writev_full_all_flags (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +int *fds,size_t nfds, +int flags,Error **errp); +#define qio_channel_writev_full_all (ioc, iov, niov, fds, nfds, errp) \ + qio_channel_writev_full_all_flags(ioc, iov, niov, fds, nfds, 0, errp) + +/** +* qio_channel_writev_zero_copy: +* @ioc: the channel object +* @iov: the array of memory regions to write data from +* @niov: the length of the @iov array +* @errp: pointer to a NULL-initialized error object +* +* Behaves like qio_channel_writev, but may write +* data asynchronously while avoiding unnecessary data copy. +* This function may return before any data is actually written, +* but will queue every buffer for writing. +* +* Some implementations require the buffer region to be locked, +* so if there is not enough locked memory available to the process +* this function will fail. +* +* If at some point it's necessary to wait for all data to be +* written, use qio_channel_flush_zero_copy(). +* +* If zero copy is not available, returns -1 and set errp. +*/ + +ssize_t qio_channel_writev_zero_copy (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +Error **errp); + +/** +* qio_channel_flush_zero_copy: +* @ioc: the channel object +* @errp: pointer to a NULL-initialized error object +* +* Will block until every packet queued with +* qio_channel_writev_zero_copy() is sent, or return +* in case of any error. +* +* If not implemented, acts as a no-op, and returns 0. +* +* Returns -1 if any error is found, +* 1 if every send failed to use zero copy. +* 0 otherwise. +*/ + +int qio_channel_flush_zero_copy (QIOChannel *ioc,Error **errp);#endif /* QIO_CHANNEL_H */
io/channel.c +54 -12
87 lines not on this page · raw message
Signature · 1 line
2.33.1
Re: [PATCH v6 1/6] QIOChannel: Add io_writev_zero_copy & io_flush_zero_copy callbacks
Quoted 48 lines · Leonardo Bras
Adds
io_writev_zero_copyandio_flush_zero_copyas optional callback to QIOChannelClass, allowing the implementation of zero copy writes by subclasses.How to use them:
- Write data usingqio_channel_writev_zero_copy(),
- Wait write completion withqio_channel_flush_zero_copy().Notes:
As some zero copy implementations work asynchronously, it's recommended to keep the write buffer untouched until the return ofqio_channel_flush_zero_copy(), to avoid the risk of sending an updated buffer instead of the one at the write.As the new callbacks are optional, if a subclass does not implement them, then:
-io_writev_zero_copywill return -1,
-io_flush_zero_copywill return 0 without changing anything.Also, some functions like
qio_channel_writev_full_all()were adapted to receive a flag parameter. That allows shared code between zero copy and non-zero copy writev, and also an easier implementation on new flags.Signed-off-by Leonardo Bras <[email protected]>--- include/io/channel.h | 98 +++++++++++++++++++++++++++++++++++++++----- io/channel.c | 66 +++++++++++++++++++++++------ 2 files changed, 142 insertions(+), 22 deletions(-)include/io/channel.h +3
-0@@ -32,12 +32,15 @@ OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass,#define QIO_CHANNEL_ERR_BLOCK -2 +#define QIO_CHANNEL_WRITE_FLAG_ZERO_COPY 0x1 +typedef enum QIOChannelFeature QIOChannelFeature ;enum QIOChannelFeature {QIO_CHANNEL_FEATURE_FD_PASS ,QIO_CHANNEL_FEATURE_SHUTDOWN ,QIO_CHANNEL_FEATURE_LISTEN , +QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY , };
include/io/channel.h struct QIOChannelClass { · line 148
Quoted 10 lines
include/io/channel.h +6
-0@@ -136,6 +139,12 @@ struct QIOChannelClass { IOHandler *io_read, IOHandler *io_write, void *opaque); +ssize_t (*io_writev_zero_copy)(QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +Error **errp); +int (*io_flush_zero_copy)(QIOChannel *ioc, +Error **errp); };
I've still got the same feedback as previous iterations. It does not make sense to having both separate callbacks / APIs and also add flags to existing methods. It just solves thue same problem twice which si redundant.
I had suggested separate callbacks originally because I thought we would need to have different signature with ability to get completions. We've done completions with a separate API call though.
So the separate zero_copy methods aren't so compelling as an idea, and we could just use flags only in retrospect.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH v6 2/6 QIOChannelSocket: Add flags parameter for writing
In reply to Leonardo BrasChange qio_channel_socket_writev() in order to accept flags, so its possible to selectively make use of sendmsg() flags.
qio_channel_socket_writev() contents were moved to a helper function qio_channel_socket_writev_flags() which accepts an extra argument for flags. (This argument is passed directly to sendmsg().
--- io/channel-socket.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
io/channel-socket.c +19 -7
@@ -520,12 +520,13 @@ static ssize_t qio_channel_socket_readv(QIOChannel *ioc,return ret; } -static ssize_t qio_channel_socket_writev (QIOChannel *ioc, -const struct iovec *iov, -size_t niov, -int *fds, -size_t nfds, -Error **errp) +static ssize_t qio_channel_socket_writev_flags (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +int *fds, +size_t nfds, +int flags, +Error **errp) {QIOChannelSocket *sioc =QIO_CHANNEL_SOCKET (ioc);ssize_t ret;
@@ -558,7 +559,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc, } retry: - ret =sendmsg (sioc->fd, &msg,0 ); + ret =sendmsg (sioc->fd, &msg, flags);if (ret <=0 ) {if (errno ==EAGAIN ) {return QIO_CHANNEL_ERR_BLOCK ;
@@ -572,6 +573,17 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc, }return ret; } + +static ssize_t qio_channel_socket_writev (QIOChannel *ioc, +const struct iovec *iov, +size_t niov, +int *fds, +size_t nfds, +Error **errp) +{ +return qio_channel_socket_writev_flags (ioc, iov, niov, fds, nfds,0 , errp); +} +#else /* WIN32 */ static ssize_t qio_channel_socket_readv(QIOChannel *ioc, const struct iovec *iov,
Signature · 1 line
2.33.1
PATCH v6 3/6 QIOChannelSocket: Implement io_writev_zero_copy & io_flush_zero_copy for CONFIG_LINUX
In reply to Leonardo BrasFor CONFIG_LINUX, implement the new optional callbacks io_write_zero_copy and io_flush_zero_copy on QIOChannelSocket, but enables it only when MSG_ZEROCOPY feature is available in the host kernel, which is checked on qio_channel_socket_connect_sync()
qio_channel_socket_flush_zero_copy() was implemented by counting how many times sendmsg(...,MSG_ZEROCOPY) was successfully called, and then reading the socket's error queue, in order to find how many of them finished sending. Flush will loop until those counters are the same, or until some error occurs.
A new function qio_channel_socket_poll() was also created in order to avoid busy-looping recvmsg() in qio_channel_socket_flush_zero_copy() while waiting for updates in socket's error queue.
Notes on using writev_zero_copy():
1: Buffer
- As MSG_ZEROCOPY tells the kernel to use the same user buffer to avoid copying, some caution is necessary to avoid overwriting any buffer before it's sent. If something like this happen, a newer version of the buffer may be sent instead.
- If this is a problem, it's recommended to call flush_zero_copy() before freeing or re-using the buffer.
2: Locked memory
- When using MSG_ZERCOCOPY, the buffer memory will be locked after queued, and unlocked after it's sent.
- Depending on the size of each buffer, and how often it's sent, it may require a larger amount of locked memory than usually available to non-root user.
- If the required amount of locked memory is not available, writev_zero_copy will return an error, which can abort an operation like migration,
- Because of this, when an user code wants to add zero copy as a feature, it requires a mechanism to disable it, so it can still be accessible to less privileged users.
--- include/io/channel-socket.h | 2 + io/channel-socket.c | 119 +++++++++++++++++++++++++++++++++++- 2 files changed, 118 insertions(+), 3 deletions(-)
include/io/channel-socket.h +2 -0
@@ -47,6 +47,8 @@ struct QIOChannelSocket {socklen_t localAddrLen;struct sockaddr_storage remoteAddr;socklen_t remoteAddrLen; +ssize_t zero_copy_queued; +ssize_t zero_copy_sent; };
io/channel-socket.c +116 -3
158 lines not on this page · raw message
Signature · 1 line
2.33.1
PATCH v6 4/6 migration: Add zero-copy parameter for QMP/HMP for Linux
In reply to Leonardo BrasAdd property that allows zero-copy migration of memory pages, and also includes a helper function migrate_use_zero_copy() to check if it's enabled.
No code is introduced to actually do the migration, but it allow future implementations to enable/disable this feature.
On non-Linux builds this parameter is compiled-out.
--- qapi/migration.json | 24 ++++++++++++++++++++++++ migration/migration.h | 5 +++++ migration/migration.c | 32 ++++++++++++++++++++++++++++++++ migration/socket.c | 5 +++++ monitor/hmp-cmds.c | 6 ++++++ 5 files changed, 72 insertions(+)
qapi/migration.json +24 -0
@@ -730,6 +730,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending memory +# pages, if host supports it. +# Requires that QEMU be permitted to use locked memory for guest +# RAM pages. +# Defaults to false. (Since 7.0) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -769,6 +776,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level' ,'multifd-zstd-level', + { 'name': 'zero-copy', 'if' : 'CONFIG_LINUX'}, 'block-bitmap-mapping' ] } ##
@@ -895,6 +903,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending memory +# pages, if host supports it. +# Requires that QEMU be permitted to use locked memory for guest +# RAM pages. +# Defaults to false. (Since 7.0) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -949,6 +964,7 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', + '*zero-copy': { 'type': 'bool', 'if': 'CONFIG_LINUX' }, '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } ##
@@ -1095,6 +1111,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending memory +# pages, if host supports it. +# Requires that QEMU be permitted to use locked memory for guest +# RAM pages. +# Defaults to false. (Since 7.0) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -1147,6 +1170,7 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', + '*zero-copy': { 'type': 'bool', 'if': 'CONFIG_LINUX' }, '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } ##
migration/migration.h +5 -0
@@ -339,6 +339,11 @@ MultiFDCompression migrate_multifd_compression(void);int migrate_multifd_zlib_level (void );int migrate_multifd_zstd_level (void ); +#ifdef CONFIG_LINUX +bool migrate_use_zero_copy (void ); +#else +#define migrate_use_zero_copy () (false) +#endif int migrate_use_xbzrle (void );uint64_t migrate_xbzrle_cache_size (void );bool migrate_colo_enabled (void );
migration/migration.c +32 -0
@@ -886,6 +886,10 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_zlib_level = s->parameters.multifd_zlib_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; +#ifdef CONFIG_LINUX + params->has_zero_copy = true; + params->zero_copy = s->parameters.zero_copy; +#endif params->has_xbzrle_cache_size = true; params->xbzrle_cache_size = s->parameters.xbzrle_cache_size; params->has_max_postcopy_bandwidth = true;
@@ -1538,6 +1542,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } +#ifdef CONFIG_LINUX +if (params->has_zero_copy) { + dest->zero_copy = params->zero_copy; + } +#endif if (params->has_xbzrle_cache_size) { dest->xbzrle_cache_size = params->xbzrle_cache_size; }
@@ -1650,6 +1659,11 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } +#ifdef CONFIG_LINUX +if (params->has_zero_copy) { + s->parameters.zero_copy = params->zero_copy; + } +#endif if (params->has_xbzrle_cache_size) { s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;xbzrle_cache_resize (params->xbzrle_cache_size, errp);
@@ -2540,6 +2554,17 @@ int migrate_multifd_zstd_level(void)return s->parameters.multifd_zstd_level; } +#ifdef CONFIG_LINUX +bool migrate_use_zero_copy (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->parameters.zero_copy; +} +#endif +int migrate_use_xbzrle (void ) {MigrationState *s;
@@ -4190,6 +4215,10 @@ static Property migration_properties[] = { DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL), +#ifdef CONFIG_LINUX +DEFINE_PROP_BOOL ("zero_copy" , MigrationState, + parameters.zero_copy, false), +#endifDEFINE_PROP_SIZE ("xbzrle-cache-size" , MigrationState, parameters.xbzrle_cache_size, DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE),
@@ -4287,6 +4316,9 @@ static void migration_instance_init(Object *obj) params->has_multifd_compression = true; params->has_multifd_zlib_level = true; params->has_multifd_zstd_level = true; +#ifdef CONFIG_LINUX + params->has_zero_copy = true; +#endif params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true; params->has_max_cpu_throttle = true;
migration/socket.c +5 -0
@@ -77,6 +77,11 @@ static void socket_outgoing_migration(QIOTask *task, }else {trace_migration_socket_outgoing_connected (data->hostname); } + +if (migrate_use_zero_copy ()) { +error_setg (&err,"Zero copy not available in migration" ); + } +migration_channel_connect (data->s, sioc, data->hostname, err);object_unref (OBJECT (sioc)); }
monitor/hmp-cmds.c +6 -0
@@ -1297,6 +1297,12 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);break ; +#ifdef CONFIG_LINUX +case MIGRATION_PARAMETER_ZERO_COPY : + p->has_zero_copy = true; +visit_type_bool (v, param, &p->zero_copy, &err); +break ; +#endif case MIGRATION_PARAMETER_XBZRLE_CACHE_SIZE : p->has_xbzrle_cache_size = true;if (!visit_type_size (v, param, &cache_size, &err)) {
Signature · 1 line
2.33.1
PATCH v6 5/6 migration: Add migrate_use_tls() helper
In reply to Leonardo BrasA lot of places check parameters.tls_creds in order to evaluate if TLS is in use, and sometimes call migrate_get_current() just for that test.
Add new helper function migrate_use_tls() in order to simplify testing for TLS usage.
--- migration/migration.h | 1 + migration/channel.c | 6 +++--- migration/migration.c | 9 +++++++++ migration/multifd.c | 5 +---- 4 files changed, 14 insertions(+), 7 deletions(-)
migration/migration.h +1 -0
@@ -344,6 +344,7 @@ bool migrate_use_zero_copy(void);#else #define migrate_use_zero_copy () (false)#endif +int migrate_use_tls (void );int migrate_use_xbzrle (void );uint64_t migrate_xbzrle_cache_size (void );bool migrate_colo_enabled (void );
migration/channel.c +3 -3
@@ -32,16 +32,16 @@*/ void migration_channel_process_incoming (QIOChannel *ioc) { -MigrationState *s =migrate_get_current ();Error *local_err =NULL ;trace_migration_set_incoming_channel ( ioc,object_get_typename (OBJECT (ioc))); -if (s->parameters.tls_creds && - *s->parameters.tls_creds && +if (migrate_use_tls () && !object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_TLS )) { +MigrationState *s =migrate_get_current (); +migration_tls_channel_process_incoming (s, ioc, &local_err); }else {migration_ioc_register_yank (ioc);
migration/migration.c +9 -0
@@ -2565,6 +2565,15 @@ bool migrate_use_zero_copy(void) }#endif +int migrate_use_tls (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->parameters.tls_creds && *s->parameters.tls_creds; +} +int migrate_use_xbzrle (void ) {MigrationState *s;
migration/multifd.c +1 -4
@@ -794,14 +794,11 @@ static bool multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc, Error *error) { -MigrationState *s =migrate_get_current (); -trace_multifd_set_outgoing_channel ( ioc,object_get_typename (OBJECT (ioc)), p->tls_hostname, error);if (!error) { -if (s->parameters.tls_creds && - *s->parameters.tls_creds && +if (migrate_use_tls () && !object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_TLS )) {multifd_tls_channel_connect (p, ioc, &error);
Signature · 1 line
2.33.1
PATCH v6 6/6 multifd: Implement zero copy write in multifd migration (multifd-zero-copy)
In reply to Leonardo BrasImplement zero copy on nocomp_send_write(), by making use of QIOChannel zero copy interface.
Change multifd_send_sync_main() so it can distinguish each iteration sync from the setup and the completion, so a flush_zero_copy() can be called after each iteration in order to make sure all dirty pages are sent before a new iteration is started.
Also make it return -1 if flush_zero_copy() fails, in order to cancel the migration process, and avoid resuming the guest in the target host without receiving all current RAM.
This will work fine on RAM migration because the RAM pages are not usually freed, and there is no problem on changing the pages content between writev_zero_copy() and the actual sending of the buffer, because this change will dirty the page and cause it to be re-sent on a next iteration anyway.
A lot of locked memory may be needed in order to use multid migration with zero-copy enabled, so disabling the feature should be necessary for low-privileged users trying to perform multifd migrations.
--- migration/multifd.h | 4 +++- migration/migration.c | 11 ++++++++++- migration/multifd.c | 40 +++++++++++++++++++++++++++++++++++----- migration/ram.c | 29 ++++++++++++++++++++++------- migration/socket.c | 5 +++-- 5 files changed, 73 insertions(+), 16 deletions(-)
migration/multifd.h +3 -1
@@ -22,7 +22,7 @@ int multifd_load_cleanup(Error **errp);bool multifd_recv_all_channels_created (void );bool multifd_recv_new_channel (QIOChannel *ioc,Error **errp);void multifd_recv_sync_main (void ); -void multifd_send_sync_main (QEMUFile *f); +int multifd_send_sync_main (QEMUFile *f,bool sync);int multifd_queue_page (QEMUFile *f,RAMBlock *block,ram_addr_t offset);/* Multifd Compression flags */
@@ -97,6 +97,8 @@ typedef struct {uint32_t packet_len;/* pointer to the packet */ MultiFDPacket_t *packet; +/* multifd flags for sending ram */ +int write_flags;/* multifd flags for each packet */ uint32_t flags;/* size of the next packet that contains pages */
migration/migration.c +10 -1
@@ -1468,7 +1468,16 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)error_prepend (errp,"Invalid mapping given for block-bitmap-mapping: " );return false; } - +#ifdef CONFIG_LINUX +if (params->zero_copy && + (!migrate_use_multifd () || + params->multifd_compression !=MULTIFD_COMPRESSION_NONE || + (params->tls_creds && *params->tls_creds))) { +error_setg (errp, +"Zero copy only available for non-compressed non-TLS multifd migration" ); +return false; + } +#endif return true; }
migration/multifd.c +35 -5
@@ -105,7 +105,8 @@ static int nocomp_send_prepare(MultiFDSendParams *p, uint32_t used,*/ static int nocomp_send_write (MultiFDSendParams *p,uint32_t used,Error **errp) { -return qio_channel_writev_all (p->c, p->pages->iov, used, errp); +return qio_channel_writev_all_flags (p->c, p->pages->iov, used, + p->write_flags, errp); }/**
@@ -578,19 +579,28 @@ void multifd_save_cleanup(void) multifd_send_state =NULL ; } -void multifd_send_sync_main (QEMUFile *f) +int multifd_send_sync_main (QEMUFile *f,bool sync) {int i; +bool flush_zero_copy;if (!migrate_use_multifd ()) { -return ; +return 0 ; }if (multifd_send_state->pages->used) {if (multifd_send_pages (f) <0 ) {error_report ("%s: multifd_send_pages fail" , __func__); -return ; +return 0 ; } } + +/* +* When using zero-copy, it's necessary to flush after each iteration to +* make sure pages from earlier iterations don't end up replacing newer +* pages. +*/ + flush_zero_copy = sync &&migrate_use_zero_copy (); +for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDSendParams *p = &multifd_send_state->params[i];
@@ -601,7 +611,7 @@ void multifd_send_sync_main(QEMUFile *f)if (p->quit) {error_report ("%s: channel %d has already quit" , __func__, i);qemu_mutex_unlock (&p->mutex); -return ; +return 0 ; } p->packet_num = multifd_send_state->packet_num++;
@@ -612,6 +622,17 @@ void multifd_send_sync_main(QEMUFile *f) ram_counters.transferred += p->packet_len;qemu_mutex_unlock (&p->mutex);qemu_sem_post (&p->sem); + +if (flush_zero_copy) { +int ret; +Error *err =NULL ; + + ret =qio_channel_flush_zero_copy (p->c, &err); +if (ret <0 ) { +error_report_err (err); +return -1 ; + } + } }for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDSendParams *p = &multifd_send_state->params[i];
@@ -620,6 +641,8 @@ void multifd_send_sync_main(QEMUFile *f)qemu_sem_wait (&p->sem_sync); }trace_multifd_send_sync_main (multifd_send_state->packet_num); + +return 0 ; }static void *multifd_send_thread (void *opaque)
@@ -917,6 +940,13 @@ int multifd_save_setup(Error **errp) p->packet->version =cpu_to_be32 (MULTIFD_VERSION ); p->name =g_strdup_printf ("multifdsend_%d" , i); p->tls_hostname =g_strdup (s->hostname); + +if (migrate_use_zero_copy ()) { + p->write_flags =QIO_CHANNEL_WRITE_FLAG_ZERO_COPY ; + }else { + p->write_flags =0 ; + } +socket_send_channel_create (multifd_new_send_channel_async, p); }
migration/ram.c +22 -7
@@ -2992,6 +2992,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) {RAMState **rsp = opaque;RAMBlock *block; +int ret;if (compress_threads_save_setup ()) {return -1 ;
@@ -3026,7 +3027,11 @@ static int ram_save_setup(QEMUFile *f, void *opaque)ram_control_before_iterate (f,RAM_CONTROL_SETUP );ram_control_after_iterate (f,RAM_CONTROL_SETUP ); -multifd_send_sync_main (f); + ret =multifd_send_sync_main (f, false); +if (ret <0 ) { +return ret; + } +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS );qemu_fflush (f);
@@ -3135,7 +3140,11 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) out:if (ret >=0 &&migration_is_setup_or_active (migrate_get_current ()->state)) { -multifd_send_sync_main (rs->f); + ret =multifd_send_sync_main (rs->f, true); +if (ret <0 ) { +return ret; + } +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS );qemu_fflush (f); ram_counters.transferred +=8 ;
@@ -3193,13 +3202,19 @@ static int ram_save_complete(QEMUFile *f, void *opaque)ram_control_after_iterate (f,RAM_CONTROL_FINISH ); } -if (ret >=0 ) { -multifd_send_sync_main (rs->f); -qemu_put_be64 (f,RAM_SAVE_FLAG_EOS ); -qemu_fflush (f); +if (ret <0 ) { +return ret; } -return ret; + ret =multifd_send_sync_main (rs->f, false); +if (ret <0 ) { +return ret; + } + +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS ); +qemu_fflush (f); + +return 0 ; } static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
migration/socket.c +3 -2
@@ -78,8 +78,9 @@ static void socket_outgoing_migration(QIOTask *task,trace_migration_socket_outgoing_connected (data->hostname); } -if (migrate_use_zero_copy ()) { -error_setg (&err,"Zero copy not available in migration" ); +if (migrate_use_zero_copy () && + !qio_channel_has_feature (sioc,QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY )) { +error_setg (&err,"Zero copy feature not detected in host kernel" ); }migration_channel_connect (data->s, sioc, data->hostname, err);
Signature · 1 line
2.33.1