MSG_ZEROCOPY + multifd
v13 cover MSG_ZEROCOPY + multifd
This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make use of it for multifd migration performance improvement, by reducing cpu usage.
Patch #1 conditionally disables liburing in systems where linux/errqueue.h conflict with liburing/compat.h (__kernel_timespec redefine)
Patch #2 creates new callbacks for QIOChannel, allowing the implementation of zero copy writing.
Patch #3 implements io_writev flags and io_flush() on QIOChannelSocket, making use of MSG_ZEROCOPY on Linux.
Patch #4 adds a "zero_copy_send" migration property, only available with CONFIG_LINUX, and compiled-out in any other architectures. This migration property has to be enabled before multifd migration starts.
Patch #5 adds a helper function that allows to see if TLS is going to be used. This helper will be later used in patch #5.
Patch #6 changes multifd_send_sync_main() so it returns int instead of void. The return value is used to understand if any error happened in the function, allowing migration to possible fail earlier.
Patch #7 implements an workaround: The behavior introduced in d48c3a0445 is hard to deal with in zerocopy, so a workaround is introduced to send the header in a different syscall, without MSG_ZEROCOPY.
Patch #8 Makes use of QIOChannelSocket zero_copy implementation on nocomp multifd migration.
Results:
In preliminary tests, the resource usage of __sys_sendmsg() reduced 15 times, and the overall migration took 13-22% less time, based in synthetic cpu workload.
In further tests, it was noted that, on multifd migration with 8 channels:
- On idle hosts, migration time reduced in 10% to 21%.
- On hosts busy with heavy cpu stress (1 stress thread per cpu, but not cpu-pinned) migration time reduced in ~25% by enabling zero-copy.
- On hosts with heavy cpu-pinned workloads (1 stress thread per cpu, cpu-pinned), migration time reducted in ~66% by enabling zero-copy.
Above tests setup:
- Sending and Receiving hosts:
- CPU : Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz (448 CPUS)
- Network card: E810-C (100Gbps)
- >1TB RAM
- QEMU: Upstream master branch + This patchset
- Linux: Upstream v5.15
- VM configuration:
- 28 VCPUs
- 512GB RAM
---
Changes since v12:
- New patch #1 added to solve an issue with Gitlab CI on alpine
- Removed unnecessary newline in patch #3 (previously #2)
- Removed incorrect commit change in roms/skiboot on patch #4 (previously #3)
Changes since v11:
- Patch #3 now wrap lines around column 75
- Patch #2 now introduce some #ifdefs instead of defining a default value for MSG_ZEROCOPY and SO_ZEROCOPY
Changes since v10:
- Patch #2 was breaking build on systems with glibc < glibc-2.27, and probably non-linux builds.
- Also on Patch #2, replaced bits/socket.h with sys/socket.h, (thanks Peter Xu)
Changes since v9:
- Patch #6 got simplified and improved (thanks Daniel)
- Patch #7 got better comments (thanks Peter Xu)
Changes since v8:
- Inserted two new patches #5 & #6, previous patch #5 is now #7.
- Workaround an optimization introduced in d48c3a0445
- Removed unnecessary assert in qio_channel_writev_full_all
Changes since v7:
- Migration property renamed from zero-copy to zero-copy-send
- A few early tests added to help misconfigurations to fail earlier
- qio_channel_full*_flags() renamed back to qio_channel_full*()
- multifd_send_sync_main() reverted back to not receiving a flag, so it always sync zero-copy when enabled.
- Improve code quality on a few points
Changes since v6:
- Remove io_writev_zero_copy(), and makes use of io_writev() new flags to achieve the same results.
- Rename io_flush_zero_copy() to io_flush()
- Previous patch #2 became too small, so it was squashed in previous patch #3 (now patch #2)
Changes since v5:
- flush_zero_copy now returns -1 on fail, 0 on success, and 1 when all processed writes were not able to use zerocopy in kernel.
- qio_channel_socket_poll() removed, using qio_channel_wait() instead
- ENOBUFS is now processed inside qio_channel_socket_writev_flags()
- Most zerocopy parameter validation moved to migrate_params_check(), leaving only feature test to socket_outgoing_migration() callback
- Naming went from *zerocopy to *zero_copy or *zero-copy, due to QAPI/QMP preferences
- Improved docs
Changes since v4:
- 3 patches got splitted in 6
- Flush is used for syncing after each iteration, instead of only at the end
- If zerocopy is not available, fail in connect instead of failing on write
- 'multifd-zerocopy' property renamed to 'zerocopy'
- Fail migrations that don't support zerocopy, if it's enabled.
- Instead of checking for zerocopy at each write, save the flags in MultiFDSendParams->write_flags and use them on write
- Reorganized flag usage in QIOChannelSocket
- A lot of typos fixed
- More doc on buffer restrictions
Changes since v3:
- QIOChannel interface names changed from io_async_{writev,flush} to io_{writev,flush}_zerocopy
- Instead of falling back in case zerocopy is not implemented, return error and abort operation.
- Flush now waits as long as needed, or return error in case anything goes wrong, aborting the operation.
- Zerocopy is now conditional in multifd, being set by parameter multifd-zerocopy
- Moves zerocopy_flush to multifd_send_sync_main() from multifd_save_cleanup so migration can abort if flush goes wrong.
- Several other small improvements
Changes since v2:
- Patch #1: One more fallback
- Patch #2: Fall back to sync if fails to lock buffer memory in MSG_ZEROCOPY send.
Changes since v1:
- Reimplemented the patchset using async_write + async_flush approach.
- Implemented a flush to be able to tell whenever all data was written.
Leonardo Bras (8):
meson.build: Fix docker-test-build@alpine when including
linux/errqueue.h
QIOChannel: Add flags on io_writev and introduce io_flush callback
QIOChannelSocket: Implement io_writev zero copy flag & io_flush for
CONFIG_LINUX
migration: Add zero-copy-send parameter for QMP/HMP for Linux
migration: Add migrate_use_tls() helper
multifd: multifd_send_sync_main now returns negative on error
multifd: Send header packet without flags if zero-copy-send is enabled
multifd: Implement zero copy write in multifd migration
(multifd-zero-copy)
meson.build | 11 +++ qapi/migration.json | 24 ++++++ include/io/channel-socket.h | 2 + include/io/channel.h | 38 ++++++++- migration/migration.h | 6 ++ migration/multifd.h | 4 +- chardev/char-io.c | 2 +- hw/remote/mpqemu-link.c | 2 +- io/channel-buffer.c | 1 + io/channel-command.c | 1 + io/channel-file.c | 1 + io/channel-socket.c | 118 +++++++++++++++++++++++++++- io/channel-tls.c | 1 + io/channel-websock.c | 1 + io/channel.c | 49 +++++++++--- migration/channel.c | 3 +- migration/migration.c | 52 +++++++++++- migration/multifd.c | 74 ++++++++++++++--- migration/ram.c | 29 +++++-- migration/rdma.c | 1 + migration/socket.c | 12 ++- monitor/hmp-cmds.c | 6 ++ scsi/pr-manager-helper.c | 2 +- tests/unit/test-io-channel-socket.c | 1 + 24 files changed, 397 insertions(+), 44 deletions(-)
Signature · 1 line
2.36.1
PATCH v13 1/8 meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
A build error happens in alpine CI when linux/errqueue.h is included in io/channel-socket.c, due to redefining of 'struct __kernel_timespec':
===
ninja: job failed: [...]
In file included from /usr/include/linux/errqueue.h:6,
from ../io/channel-socket.c:29:
/usr/include/linux/time_types.h:7:8: error: redefinition of 'struct __kernel_timespec'
7 | struct __kernel_timespec {
| ^~~~~~~~~~~~~~~~~
In file included from /usr/include/liburing.h:19,
from /builds/user/qemu/include/block/aio.h:18,
from /builds/user/qemu/include/io/channel.h:26,
from /builds/user/qemu/include/io/channel-socket.h:24,
from ../io/channel-socket.c:24:
/usr/include/liburing/compat.h:9:8: note: originally defined here
9 | struct __kernel_timespec {
| ^~~~~~~~~~~~~~~~~ninja: subcommand failed
===
As above error message suggests, 'struct __kernel_timespec' was already defined by liburing/compat.h.
Fix alpine CI by adding test to disable liburing in configure step if a redefinition happens between linux/errqueue.h and liburing/compat.h.
--- meson.build | 11 +++++++++++ 1 file changed, 11 insertions(+)
meson.build +11 -0
@@ -515,12 +515,23 @@ if not get_option('linux_aio').auto() or have_block required: get_option('linux_aio'), kwargs: static_kwargs) endif + +linux_io_uring_test = ''' + #include <liburing.h> + #include <linux/errqueue.h> + + int main(void) { return 0; }''' + linux_io_uring = not_found if not get_option('linux_io_uring').auto() or have_block linux_io_uring = dependency('liburing', version: '>=0.3', required: get_option('linux_io_uring'), method: 'pkg-config', kwargs: static_kwargs) + if not cc.links(linux_io_uring_test) + linux_io_uring = not_found + endif endif + libnfs = not_found if not get_option('libnfs').auto() or have_block libnfs = dependency('libnfs', version: '>=1.9.3',
Signature · 1 line
2.36.1
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':
OK, looks to be same mechanism as other meson tests.
Quoted 61 lines
===
ninja: job failed: [...]
In file included from/usr/include/linux/errqueue.h:6,from ../io/channel-socket.c:29: /usr/include/linux/time_types.h:7:8: error: redefinition of 'struct __kernel_timespec' 7 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ In file included from /usr/include/liburing.h:19, from /builds/user/qemu/include/block/aio.h:18, from /builds/user/qemu/include/io/channel.h:26, from /builds/user/qemu/include/io/channel-socket.h:24, from ../io/channel-socket.c:24: /usr/include/liburing/compat.h:9:8: note: originally defined here 9 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ninja: subcommand failed
===As above error message suggests, 'struct
__kernel_timespec' was already defined byliburing/compat.h.Fix alpine CI by adding test to disable liburing in configure step if a redefinition happens between
linux/errqueue.handliburing/compat.h.Signed-off-by Leonardo Bras <[email protected]>--- meson.build | 11 +++++++++++ 1 file changed, 11 insertions(+)meson.build +11
-0@@ -515,12 +515,23 @@ if not get_option('linux_aio').auto() or have_block required: get_option('linux_aio'), kwargs: static_kwargs) endif + +linux_io_uring_test = ''' + #include <liburing.h> + #include <linux/errqueue.h> + + int main(void) { return 0; }''' + linux_io_uring = not_found if not get_option('linux_io_uring').auto() or have_block linux_io_uring = dependency('liburing', version: '>=0.3', required: get_option('linux_io_uring'), method: 'pkg-config', kwargs: static_kwargs) + if not cc.links(linux_io_uring_test) + linux_io_uring = not_found + endif endif + libnfs = not_found if not get_option('libnfs').auto() or have_block libnfs = dependency('libnfs', version: '>=1.9.3',Signature · 1 line
2.36.1
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':OK, looks to be same mechanism as other meson tests.
Reviewed-by Dr. David Alan Gilbert <[email protected]>
As of about an hour or so ago, this patch should not be required.
https://gitlab.alpinelinux.org/alpine/aports/-/issues/13813
Quoted 54 lines
Quoted 54 lines
===
ninja: job failed: [...]
In file included from/usr/include/linux/errqueue.h:6,from ../io/channel-socket.c:29: /usr/include/linux/time_types.h:7:8: error: redefinition of 'struct __kernel_timespec' 7 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ In file included from /usr/include/liburing.h:19, from /builds/user/qemu/include/block/aio.h:18, from /builds/user/qemu/include/io/channel.h:26, from /builds/user/qemu/include/io/channel-socket.h:24, from ../io/channel-socket.c:24: /usr/include/liburing/compat.h:9:8: note: originally defined here 9 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ninja: subcommand failed
===As above error message suggests, 'struct
__kernel_timespec' was already defined byliburing/compat.h.Fix alpine CI by adding test to disable liburing in configure step if a redefinition happens between
linux/errqueue.handliburing/compat.h.Signed-off-by Leonardo Bras <[email protected]>--- meson.build | 11 +++++++++++ 1 file changed, 11 insertions(+)meson.build +10
-0@@ -515,12 +515,23 @@ if not get_option('linux_aio').auto() or have_block required: get_option('linux_aio'), kwargs: static_kwargs) endif + +linux_io_uring_test = ''' + #include <liburing.h> + #include <linux/errqueue.h> + + int main(void) { return 0; }''' + linux_io_uring = not_found if not get_option('linux_io_uring').auto() or have_block linux_io_uring = dependency('liburing', version: '>=0.3', required: get_option('linux_io_uring'), method: 'pkg-config', kwargs: static_kwargs) + if not cc.links(linux_io_uring_test) + linux_io_uring = not_found + endif endif
Quoted 11 lines
Quoted 8 lines
meson.build +1
-0+ libnfs = not_found if not get_option('libnfs').auto() or have_block libnfs = dependency('libnfs', version: '>=1.9.3',Signature · 1 line
2.36.1Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 5 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 4 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':
Quoted 8 lines
Quoted 4 lines
OK, looks to be same mechanism as other meson tests.
Reviewed-by Dr. David Alan Gilbert <[email protected]>As of about an hour or so ago, this patch should not be required.
I'll take it anyway as protection against any other broken build envs.
Dave
Quoted 73 lines
Quoted 54 lines
Quoted 54 lines
===
ninja: job failed: [...]
In file included from/usr/include/linux/errqueue.h:6,from ../io/channel-socket.c:29: /usr/include/linux/time_types.h:7:8: error: redefinition of 'struct __kernel_timespec' 7 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ In file included from /usr/include/liburing.h:19, from /builds/user/qemu/include/block/aio.h:18, from /builds/user/qemu/include/io/channel.h:26, from /builds/user/qemu/include/io/channel-socket.h:24, from ../io/channel-socket.c:24: /usr/include/liburing/compat.h:9:8: note: originally defined here 9 | struct __kernel_timespec { | ^~~~~~~~~~~~~~~~~ninja: subcommand failed
===As above error message suggests, 'struct
__kernel_timespec' was already defined byliburing/compat.h.Fix alpine CI by adding test to disable liburing in configure step if a redefinition happens between
linux/errqueue.handliburing/compat.h.Signed-off-by Leonardo Bras <[email protected]>--- meson.build | 11 +++++++++++ 1 file changed, 11 insertions(+)meson.build +10
-0@@ -515,12 +515,23 @@ if not get_option('linux_aio').auto() or have_block required: get_option('linux_aio'), kwargs: static_kwargs) endif + +linux_io_uring_test = ''' + #include <liburing.h> + #include <linux/errqueue.h> + + int main(void) { return 0; }''' + linux_io_uring = not_found if not get_option('linux_io_uring').auto() or have_block linux_io_uring = dependency('liburing', version: '>=0.3', required: get_option('linux_io_uring'), method: 'pkg-config', kwargs: static_kwargs) + if not cc.links(linux_io_uring_test) + linux_io_uring = not_found + endif endifQuoted 11 lines
Quoted 8 lines
meson.build +1
-0+ libnfs = not_found if not get_option('libnfs').auto() or have_block libnfs = dependency('libnfs', version: '>=1.9.3',Signature · 1 line
2.36.1Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UKWith regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 8 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 7 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':OK, looks to be same mechanism as other meson tests.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Reviewed-by Dr. David Alan Gilbert <[email protected]>As of about an hour or so ago, this patch should not be required.
I'll take it anyway as protection against any other broken build envs.
Can you update the commit message at least then.
The root casue trigger for the bug is the OS uses a busybox impl of mkdtemp, which isn't compat with the args liburing configure was previously using. I doubt there are many such OS around to be honest, as most will use coreutils.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 12 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':OK, looks to be same mechanism as other meson tests.
Reviewed-by Dr. David Alan Gilbert <[email protected]>
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
As of about an hour or so ago, this patch should not be required.
I'll take it anyway as protection against any other broken build envs.
Can you update the commit message at least then.
Sure, I've added:
[dgilbert: This has been fixed in Alpine issue 13813 and liburing]
Quoted 1 line
The root casue trigger for the bug is the OS uses a busybox
I guess you mean musl??
Quoted 11 lines
impl of mkdtemp, which isn't compat with the args liburing configure was previously using. I doubt there are many such OS around to be honest, as most will use coreutils.
With regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [PATCH v13 1/8] meson.build: Fix docker-test-build@alpine when including linux/errqueue.h
Quoted 21 lines · Dr. David Alan Gilbert
Quoted 12 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 11 lines · Dr. David Alan Gilbert
Quoted 10 lines · * Daniel P. Berrangé ([email protected]) wrote:
Quoted 8 lines · Dr. David Alan Gilbert
Quoted 2 lines · * Leonardo Bras ([email protected]) wrote:
A build error happens in alpine CI when
linux/errqueue.his included inio/channel-socket.c, due to redefining of 'struct__kernel_timespec':OK, looks to be same mechanism as other meson tests.
Reviewed-by Dr. David Alan Gilbert <[email protected]>Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
As of about an hour or so ago, this patch should not be required.
I'll take it anyway as protection against any other broken build envs.
Can you update the commit message at least then.
Quoted 7 lines
Sure, I've added:
[dgilbert: This has been fixed in Alpine issue 13813 and liburing]
Quoted 1 line
The root casue trigger for the bug is the OS uses a busybox
I guess you mean musl??
I don't think it is musl, its the configure shell script and it is throwing an error from the 'mktemp' command
Quoted 3 lines
Quoted 3 lines
impl of mkdtemp, which isn't compat with the args liburing configure was previously using. I doubt there are many such OS around to be honest, as most will use coreutils.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
PATCH v13 2/8 QIOChannel: Add flags on io_writev and introduce io_flush callback
In reply to Leonardo BrasAdd flags to io_writev and introduce io_flush as optional callback to QIOChannelClass, allowing the implementation of zero copy writes by subclasses.
How to use them:
- Write data using qio_channel_writev*(...,QIO_CHANNEL_WRITE_FLAG_ZERO_COPY),
- Wait write completion with qio_channel_flush().
Notes:
As some zero copy write implementations work asynchronously, it's recommended to keep the write buffer untouched until the return of qio_channel_flush(), to avoid the risk of sending an updated buffer instead of the buffer state during write.
As io_flush callback is optional, if a subclass does not implement it, then:
- io_flush will return 0 without changing anything.
Also, some functions like qio_channel_writev_full_all() were adapted to receive a flag parameter. That allows shared code between zero copy and non-zero copy writev, and also an easier implementation on new flags.
--- include/io/channel.h | 38 +++++++++++++++++++++- chardev/char-io.c | 2 +- hw/remote/mpqemu-link.c | 2 +- io/channel-buffer.c | 1 + io/channel-command.c | 1 + io/channel-file.c | 1 + io/channel-socket.c | 2 ++ io/channel-tls.c | 1 + io/channel-websock.c | 1 + io/channel.c | 49 +++++++++++++++++++++++------ migration/rdma.c | 1 + scsi/pr-manager-helper.c | 2 +- tests/unit/test-io-channel-socket.c | 1 + 13 files changed, 88 insertions(+), 14 deletions(-)
include/io/channel.h +37 -1
@@ -32,12 +32,15 @@ OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass,#define QIO_CHANNEL_ERR_BLOCK -2 +#define QIO_CHANNEL_WRITE_FLAG_ZERO_COPY 0x1 +typedef enum QIOChannelFeature QIOChannelFeature ;enum QIOChannelFeature {QIO_CHANNEL_FEATURE_FD_PASS ,QIO_CHANNEL_FEATURE_SHUTDOWN ,QIO_CHANNEL_FEATURE_LISTEN , +QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY , };
@@ -104,6 +107,7 @@ struct QIOChannelClass {size_t niov, int *fds, size_t nfds, + int flags, Error **errp); ssize_t (*io_readv)(QIOChannel *ioc, const struct iovec *iov,
@@ -136,6 +140,8 @@ struct QIOChannelClass { IOHandler *io_read, IOHandler *io_write, void *opaque); +int (*io_flush)(QIOChannel *ioc, +Error **errp); };/* General I/O handling functions */
@@ -228,6 +234,7 @@ ssize_t qio_channel_readv_full(QIOChannel *ioc,* @niov: the length of the @iov array * @fds: an array of file handles to send * @nfds: number of file handles in @fds +* @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*) * @errp: pointer to a NULL-initialized error object * * Write data to the IO channel, reading it from the
@@ -260,6 +267,7 @@ ssize_t qio_channel_writev_full(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp);/**
@@ -837,6 +845,7 @@ int qio_channel_readv_full_all(QIOChannel *ioc,* @niov: the length of the @iov array * @fds: an array of file handles to send * @nfds: number of file handles in @fds +* @flags: write flags (QIO_CHANNEL_WRITE_FLAG_*) * @errp: pointer to a NULL-initialized error object * *
@@ -846,6 +855,14 @@ int qio_channel_readv_full_all(QIOChannel *ioc,* to be written, yielding from the current coroutine * if required. * +* If QIO_CHANNEL_WRITE_FLAG_ZERO_COPY is passed in flags, +* instead of waiting for all requested data to be written, +* this function will wait until it's all queued for writing. +* In this case, if the buffer gets changed between queueing and +* sending, the updated buffer will be sent. If this is not a +* desired behavior, it's suggested to call qio_channel_flush() +* before reusing the buffer. +* * Returns: 0 if all bytes were written, or -1 on error */
@@ -853,6 +870,25 @@ int qio_channel_writev_full_all(QIOChannel *ioc,const struct iovec *iov, size_t niov, int *fds, size_t nfds, - Error **errp); + int flags, Error **errp); + +/** +* qio_channel_flush: +* @ioc: the channel object +* @errp: pointer to a NULL-initialized error object +* +* Will block until every packet queued with +* qio_channel_writev_full() + QIO_CHANNEL_WRITE_FLAG_ZERO_COPY +* is sent, or return in case of any error. +* +* If not implemented, acts as a no-op, and returns 0. +* +* Returns -1 if any error is found, +* 1 if every send failed to use zero copy. +* 0 otherwise. +*/ + +int qio_channel_flush (QIOChannel *ioc, +Error **errp);#endif /* QIO_CHANNEL_H */
chardev/char-io.c +1 -1
@@ -122,7 +122,7 @@ int io_channel_send_full(QIOChannel *ioc, ret =qio_channel_writev_full ( ioc, &iov,1 , - fds, nfds,NULL ); + fds, nfds,0 ,NULL );if (ret ==QIO_CHANNEL_ERR_BLOCK ) {if (offset) {return offset;
hw/remote/mpqemu-link.c +1 -1
@@ -68,7 +68,7 @@ bool mpqemu_msg_send(MPQemuMsg *msg, QIOChannel *ioc, Error **errp) }if (!qio_channel_writev_full_all (ioc, send,G_N_ELEMENTS (send), - fds, nfds, errp)) { + fds, nfds,0 , errp)) { ret = true; }else {trace_mpqemu_send_io_error (msg->cmd, msg->size, nfds);
io/channel-buffer.c +1 -0
@@ -81,6 +81,7 @@ static ssize_t qio_channel_buffer_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelBuffer *bioc = QIO_CHANNEL_BUFFER(ioc);
io/channel-command.c +1 -0
@@ -276,6 +276,7 @@ static ssize_t qio_channel_command_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelCommand *cioc = QIO_CHANNEL_COMMAND(ioc);
io/channel-file.c +1 -0
@@ -114,6 +114,7 @@ static ssize_t qio_channel_file_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelFile *fioc = QIO_CHANNEL_FILE(ioc);
io/channel-socket.c +2 -0
@@ -524,6 +524,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc);
@@ -619,6 +620,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc);
io/channel-tls.c +1 -0
@@ -301,6 +301,7 @@ static ssize_t qio_channel_tls_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelTLS *tioc = QIO_CHANNEL_TLS(ioc);
io/channel-websock.c +1 -0
@@ -1127,6 +1127,7 @@ static ssize_t qio_channel_websock_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelWebsock *wioc = QIO_CHANNEL_WEBSOCK(ioc);
io/channel.c +39 -10
99 lines not on this page · raw message
migration/rdma.c +1 -0
@@ -2840,6 +2840,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc,size_t niov, int *fds, size_t nfds, + int flags, Error **errp) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc);
scsi/pr-manager-helper.c +1 -1
@@ -77,7 +77,7 @@ static int pr_manager_helper_write(PRManagerHelper *pr_mgr, iov.iov_base = (void *)buf; iov.iov_len = sz; n_written =qio_channel_writev_full (QIO_CHANNEL (pr_mgr->ioc), &iov,1 , - nfds ? &fd :NULL , nfds, errp); + nfds ? &fd :NULL , nfds,0 , errp);if (n_written <=0 ) {assert (n_written !=QIO_CHANNEL_ERR_BLOCK );
tests/unit/test-io-channel-socket.c +1 -0
@@ -444,6 +444,7 @@ static void test_io_channel_unix_fd_pass(void)G_N_ELEMENTS (iosend), fdsend,G_N_ELEMENTS (fdsend), +0 , &error_abort); qio_channel_readv_full(dst,
Signature · 1 line
2.36.1
PATCH v13 3/8 QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
In reply to Leonardo BrasFor CONFIG_LINUX, implement the new zero copy flag and the optional callback io_flush on QIOChannelSocket, but enables it only when MSG_ZEROCOPY feature is available in the host kernel, which is checked on qio_channel_socket_connect_sync()
qio_channel_socket_flush() was implemented by counting how many times sendmsg(...,MSG_ZEROCOPY) was successfully called, and then reading the socket's error queue, in order to find how many of them finished sending. Flush will loop until those counters are the same, or until some error occurs.
Notes on using writev() with QIO_CHANNEL_WRITE_FLAG_ZERO_COPY:
1: Buffer
- As MSG_ZEROCOPY tells the kernel to use the same user buffer to avoid copying, some caution is necessary to avoid overwriting any buffer before it's sent. If something like this happen, a newer version of the buffer may be sent instead.
- If this is a problem, it's recommended to call qio_channel_flush() before freeing or re-using the buffer.
2: Locked memory
- When using MSG_ZERCOCOPY, the buffer memory will be locked after queued, and unlocked after it's sent.
- Depending on the size of each buffer, and how often it's sent, it may require a larger amount of locked memory than usually available to non-root user.
- If the required amount of locked memory is not available, writev_zero_copy will return an error, which can abort an operation like migration,
- Because of this, when an user code wants to add zero copy as a feature, it requires a mechanism to disable it, so it can still be accessible to less privileged users.
--- include/io/channel-socket.h | 2 + io/channel-socket.c | 116 ++++++++++++++++++++++++++++++++++-- 2 files changed, 114 insertions(+), 4 deletions(-)
include/io/channel-socket.h +2 -0
@@ -47,6 +47,8 @@ struct QIOChannelSocket {socklen_t localAddrLen;struct sockaddr_storage remoteAddr;socklen_t remoteAddrLen; +ssize_t zero_copy_queued; +ssize_t zero_copy_sent; };
io/channel-socket.c +112 -4
163 lines not on this page · raw message
Signature · 1 line
2.36.1
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 43 lines · Leonardo Bras
For
CONFIG_LINUX, implement the new zero copy flag and the optional callbackio_flushon QIOChannelSocket, but enables it only whenMSG_ZEROCOPYfeature is available in the host kernel, which is checked onqio_channel_socket_connect_sync()
qio_channel_socket_flush()was implemented by counting how many timessendmsg(...,MSG_ZEROCOPY)was successfully called, and then reading the socket's error queue, in order to find how many of them finished sending. Flush will loop until those counters are the same, or until some error occurs.Notes on using
writev()withQIO_CHANNEL_WRITE_FLAG_ZERO_COPY:
1: Buffer
- AsMSG_ZEROCOPYtells the kernel to use the same user buffer to avoid copying, some caution is necessary to avoid overwriting any buffer before it's sent. If something like this happen, a newer version of the buffer may be sent instead.
- If this is a problem, it's recommended to callqio_channel_flush()before freeing or re-using the buffer.2: Locked memory
- When usingMSG_ZERCOCOPY, the buffer memory will be locked after queued, and unlocked after it's sent.
- Depending on the size of each buffer, and how often it's sent, it may require a larger amount of locked memory than usually available to non-root user.
- If the required amount of locked memory is not available,writev_zero_copywill return an error, which can abort an operation like migration,
- Because of this, when an user code wants to add zero copy as a feature, it requires a mechanism to disable it, so it can still be accessible to less privileged users.Signed-off-by Leonardo Bras <[email protected]>Reviewed-by Peter Xu <[email protected]>Reviewed-by Daniel P. Berrangé <[email protected]>Reviewed-by Juan Quintela <[email protected]>--- include/io/channel-socket.h | 2 + io/channel-socket.c | 116 ++++++++++++++++++++++++++++++++++-- 2 files changed, 114 insertions(+), 4 deletions(-)include/io/channel-socket.h +0
-0
include/io/channel-socket.h struct QIOChannelSocket { · line 53
Quoted 7 lines
include/io/channel-socket.h +2
-0@@ -47,6 +47,8 @@ struct QIOChannelSocket {socklen_t localAddrLen;struct sockaddr_storage remoteAddr;socklen_t remoteAddrLen; +ssize_t zero_copy_queued; +ssize_t zero_copy_sent; };
Hi, Leonardo. I'm also paying attention to the application of MSG_ZEROCOPY in live migration recently. I noticed that you defined a member zero_copy_queued in the struct QIOChannelSocket, but I can't find out where the value of this member has been changed in your patch. Can you answer it for me?
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 40 lines · 徐闯
Quoted 38 lines · Leonardo Bras
For
CONFIG_LINUX, implement the new zero copy flag and the optional callbackio_flushon QIOChannelSocket, but enables it only whenMSG_ZEROCOPYfeature is available in the host kernel, which is checked onqio_channel_socket_connect_sync()
qio_channel_socket_flush()was implemented by counting how many timessendmsg(...,MSG_ZEROCOPY)was successfully called, and then reading the socket's error queue, in order to find how many of them finished sending. Flush will loop until those counters are the same, or until some error occurs.Notes on using
writev()withQIO_CHANNEL_WRITE_FLAG_ZERO_COPY:
1: Buffer
- AsMSG_ZEROCOPYtells the kernel to use the same user buffer to avoid copying, some caution is necessary to avoid overwriting any buffer before it's sent. If something like this happen, a newer version of the buffer may be sent instead.
- If this is a problem, it's recommended to callqio_channel_flush()before freeing or re-using the buffer.2: Locked memory
- When usingMSG_ZERCOCOPY, the buffer memory will be locked after queued, and unlocked after it's sent.
- Depending on the size of each buffer, and how often it's sent, it may require a larger amount of locked memory than usually available to non-root user.
- If the required amount of locked memory is not available,writev_zero_copywill return an error, which can abort an operation like migration,
- Because of this, when an user code wants to add zero copy as a feature, it requires a mechanism to disable it, so it can still be accessible to less privileged users.Signed-off-by Leonardo Bras <[email protected]>Reviewed-by Peter Xu <[email protected]>Reviewed-by Daniel P. Berrangé <[email protected]>Reviewed-by Juan Quintela <[email protected]>--- include/io/channel-socket.h | 2 + io/channel-socket.c | 116 ++++++++++++++++++++++++++++++++++-- 2 files changed, 114 insertions(+), 4 deletions(-)
Quoted 17 lines
include/io/channel-socket.h struct QIOChannelSocket { · line 52
Quoted 11 lines
include/io/channel-socket.h +2
-0@@ -47,6 +47,8 @@ struct QIOChannelSocket {socklen_t localAddrLen;struct sockaddr_storage remoteAddr;socklen_t remoteAddrLen; +ssize_t zero_copy_queued; +ssize_t zero_copy_sent; };Hi, Leonardo. I'm also paying attention to the application of
MSG_ZEROCOPYin live migration recently. I noticed that you defined a memberzero_copy_queuedin the struct QIOChannelSocket, but I can't find out where the value of this member has been changed in your patch. Can you answer it for me?
Good point.. it should probably be increased when queuing the pages. We'd better fix it up or it seems the flush() will be literally an no-op..
Two things in qio_channel_socket_flush() we can do to make sure it'll work as expected, imo:
1) make ret=-1 as initial value, rather than 1 - we only check negative errors in the caller so we could have missed a positive "1"
2) add a tracepoint into the loop of updating zero_copy_sent
Leo, what's your take?
Thanks,
Signature · 1 line
Peter Xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Hello Peter,
Quoted 1 line · Peter Xu
[...]
Quoted 9 lines
Quoted 6 lines
Hi, Leonardo. I'm also paying attention to the application of
MSG_ZEROCOPYin live migration recently. I noticed that you defined a memberzero_copy_queuedin the struct QIOChannelSocket, but I can't find out where the value of this member has been changed in your patch. Can you answer it for me?Good point.. it should probably be increased when queuing the pages. We'd better fix it up or it seems the
flush()will be literally an no-op..
That's correct.
I am working on a fix right now.
The idea is to increment it in qio_channel_socket_writev() if sendmsg succeeds.
Quoted 10 lines
Two things in
qio_channel_socket_flush()we can do to make sure it'll work as expected, imo:1) make ret=-1 as initial value, rather than 1 - we only check negative errors in the caller so we could have missed a positive "1"
2) add a tracepoint into the loop of updating
zero_copy_sentLeo, what's your take?
(1) is not an option, as the interface currently uses ret=1 to make sure MSG_ZEROCOPY is getting used,
I added that so the user of qio_channel can switch off zero-copy if it's not getting used, and save some cpu.
(2) is not a problem, but I fail to see how useful that would be. Is the idea manually keeping track of flush happening?
Best regards,
Leo
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 4 lines · Leonardo Bras Soares Passos
(1) is not an option, as the interface currently uses ret=1 to make sure
MSG_ZEROCOPYis getting used,
I added that so the user ofqio_channelcan switch off zero-copy if it's not getting used, and save some cpu.
Yes (1) is not, but could you explain what do you mean by making sure MSG_ZEROCOPY being used? Why is it relevant to the retval here?
I just figured it's a bit weird to return >0 here in flush().
Quoted 3 lines
(2) is not a problem, but I fail to see how useful that would be. Is the idea manually keeping track of flush happening?
Yes if we can check this up it'll be good enough to me. The trace point could help in some case in the future too to monitor the behavior of kernel MSG_ERRQUEUE but if you don't like it then it's okay.
Signature · 1 line
Peter Xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 9 lines · Peter Xu
Quoted 4 lines · Leonardo Bras Soares Passos
(1) is not an option, as the interface currently uses ret=1 to make sure
MSG_ZEROCOPYis getting used,
I added that so the user ofqio_channelcan switch off zero-copy if it's not getting used, and save some cpu.Yes (1) is not, but could you explain what do you mean by making sure
MSG_ZEROCOPYbeing used? Why is it relevant to the retval here?
If sendmsg() is called with MSG_ZEROCOPY, and everything is configured correctly, the kernel will attempt to send the buffer using zero-copy.
Even with the right configuration on a recent enough kernel, there are factors that can prevent zero-copy from happening, and the kernel will fall back to the copying mechanism.
An example being the net device not supporting 'Scatter-Gather' feature (NETIF_F_SG).
When this happens, there is an overhead for 'trying zero-copy first', instead of just opting for the copying mechanism.
In a previous iteration of the patchset, it was made clear that it's desirable to detect when the kernel falls back to copying mechanism, so the user of 'QIOChannelSocket' can switch to copying and avoid the overhead. This was done by the return value of flush(), which is 1 if that occurs.
Quoted 11 lines
I just figured it's a bit weird to return >0 here in
flush().Quoted 3 lines
(2) is not a problem, but I fail to see how useful that would be. Is the idea manually keeping track of flush happening?
Yes if we can check this up it'll be good enough to me. The trace point could help in some case in the future too to monitor the behavior of kernel
MSG_ERRQUEUEbut if you don't like it then it's okay.
TBH I am not sure how those traces work yet, and I am afraid it can introduce some overhead in flush.
In any way, we can introduce this trace in a separated patch, since fixing zero-copy flush seems more urgent right now.
Best regards,
Leo
Quoted 3 lines
--
Peter Xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 20 lines · Leonardo Bras Soares Passos
Quoted 9 lines · Peter Xu
Quoted 4 lines · Leonardo Bras Soares Passos
(1) is not an option, as the interface currently uses ret=1 to make sure
MSG_ZEROCOPYis getting used,
I added that so the user ofqio_channelcan switch off zero-copy if it's not getting used, and save some cpu.Yes (1) is not, but could you explain what do you mean by making sure
MSG_ZEROCOPYbeing used? Why is it relevant to the retval here?If
sendmsg()is called withMSG_ZEROCOPY, and everything is configured correctly, the kernel will attempt to send the buffer using zero-copy.Even with the right configuration on a recent enough kernel, there are factors that can prevent zero-copy from happening, and the kernel will fall back to the copying mechanism.
An example being the net device not supporting 'Scatter-Gather' feature (NETIF_F_SG).
Quoted 8 lines
When this happens, there is an overhead for 'trying zero-copy first', instead of just opting for the copying mechanism.
In a previous iteration of the patchset, it was made clear that it's desirable to detect when the kernel falls back to copying mechanism, so the user of 'QIOChannelSocket' can switch to copying and avoid the overhead. This was done by the return value of
flush(), which is 1 if that occurs.
Two questions..
1) When that happens, will MSG_ERRQUEUE keeps working just like zerocopy is functional?
If the answer is yes, I don't see how ret=1 will ever be returned.. because we'll also go into the same loop in qio_channel_socket_flush() anyway.
If the answer is no, then since we'll have non-zero zero_copy_queued, will the loop in qio_channel_socket_flush() go into a dead one? How could it return?
2) Even if we have the correct ret=1 returned when that happens, which
caller is detecting that ret==1 and warn the admin?
Thanks,
Signature · 1 line
Peter Xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Hello Peter,
On Wed, Jun 8, 2022 at 5:23 PM Peter Xu <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 5 lines
In a previous iteration of the patchset, it was made clear that it's desirable to detect when the kernel falls back to copying mechanism, so the user of 'QIOChannelSocket' can switch to copying and avoid the overhead. This was done by the return value of
flush(), which is 1 if that occurs.Two questions..
1) When that happens, will
MSG_ERRQUEUEkeeps working just like zerocopy is functional?
I am not sure about what exactly you meant by 'like zerocopy is funcional', but the
idea is that reading from MSG_ERRQUEUE should return a msg for each sendmsg syscall with MSG_ZEROCOPY that previously happened. This does not depend on the outcome (like falling back to the copying mechanism).
btw, most of those messages may be batched to reduce overhead.
At some point, zero-copy may fail, and fall back to copying, so in those messages
an error code SO_EE_CODE_ZEROCOPY_COPIED can be seen. Having only those messages in a flush will trigger the returning of 1 from the flush function.
Quoted 4 lines
If the answer is yes, I don't see how ret=1 will ever be returned.. because we'll also go into the same loop in
qio_channel_socket_flush()anyway.
We set ret to 1 at function entry and then for each message in the MSG_ERRQUEUE, we test if it has error code different than SO_EE_CODE_ZEROCOPY_COPIED. If it ever have a different error code, we set ret=0.
So, in our previous example, if we have a net device not supporting the 'Scatter-Gather'
feature (NETIF_F_SG), every error message will be SO_EE_CODE_ZEROCOPY_COPIED, and it will return 1.
Quoted 4 lines
If the answer is no, then since we'll have non-zero
zero_copy_queued, will the loop inqio_channel_socket_flush()go into a dead one? How could it return?
No, because it will go through all packets sent with MSG_ZEROCOPY, including the ones that fell back to copying, so the counter should be fine. If any code disables
zero-copy, it will both stop sending stuff wil MSG_ZEROCOPY and flushing, so it should be fine.
Quoted 4 lines
2) Even if we have the correct ret=1 returned when that happens, which
caller is detecting that ret==1 and warn the admin?
No caller is using that right now.
It's supposed to be a QIOChannel interface feature, and any user/implementation could use that information to warn if zero-copy is not being used, fall back to copying directly (to avoid overhead of testing zero-copy) or even use it to cancel the
sending if wanted.
It was a suggestion of Daniel on top of [PATCH v5 1/6] IIRC.
Best regards,
Leo
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 22 lines · Leonardo Bras Soares Passos
Hello Peter,
On Wed, Jun 8, 2022 at 5:23 PM Peter Xu <[email protected]> wrote:
[...]Quoted 10 lines
Quoted 5 lines
In a previous iteration of the patchset, it was made clear that it's desirable to detect when the kernel falls back to copying mechanism, so the user of 'QIOChannelSocket' can switch to copying and avoid the overhead. This was done by the return value of
flush(), which is 1 if that occurs.Two questions..
1) When that happens, will
MSG_ERRQUEUEkeeps working just like zerocopy is functional?I am not sure about what exactly you meant by 'like zerocopy is funcional', but the
idea is that reading fromMSG_ERRQUEUEshould return a msg for each sendmsg syscall withMSG_ZEROCOPYthat previously happened. This does not depend on the outcome (like falling back to the copying mechanism).
btw, most of those messages may be batched to reduce overhead.
Quoted 5 lines
At some point, zero-copy may fail, and fall back to copying, so in those messages
an error codeSO_EE_CODE_ZEROCOPY_COPIEDcan be seen. Having only those messages in a flush will trigger the returning of 1 from the flush function.
Ah I think I missed the "reset ret==0 when !SO_EE_CODE_ZEROCOPY_COPIED" path.. Sorry.
Quoted 33 lines
Quoted 5 lines
If the answer is yes, I don't see how ret=1 will ever be returned.. because we'll also go into the same loop in
qio_channel_socket_flush()anyway.We set ret to 1 at function entry and then for each message in the
MSG_ERRQUEUE, we test if it has error code different thanSO_EE_CODE_ZEROCOPY_COPIED. If it ever have a different error code, we set ret=0.So, in our previous example, if we have a net device not supporting the 'Scatter-Gather'
feature (NETIF_F_SG), every error message will beSO_EE_CODE_ZEROCOPY_COPIED, and it will return 1.Quoted 4 lines
If the answer is no, then since we'll have non-zero
zero_copy_queued, will the loop inqio_channel_socket_flush()go into a dead one? How could it return?No, because it will go through all packets sent with
MSG_ZEROCOPY, including the ones that fell back to copying, so the counter should be fine. If any code disables
zero-copy, it will both stop sending stuff wilMSG_ZEROCOPYand flushing, so it should be fine.Quoted 4 lines
2) Even if we have the correct ret=1 returned when that happens, which
caller is detecting that ret==1 and warn the admin?
Quoted 8 lines
No caller is using that right now.
It's supposed to be a QIOChannel interface feature, and any user/implementation could use that information to warn if zero-copy is not being used, fall back to copying directly (to avoid overhead of testing zero-copy) or even use it to cancel the
sending if wanted.It was a suggestion of Daniel on top of [PATCH v5 1/6] IIRC.
OK the detection makes sense, thanks for the details.
Then now I'm wondering whether we should have warned the admin already if zero-copy send is not fully enabled in live migration. Should we add a error_report_once() somewhere for the ret==1 already? After all the user specify zero_copy_send=true explicitly. Did I miss something again?
Thanks,
Signature · 1 line
Peter Xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 73 lines · Peter Xu
Quoted 22 lines · Leonardo Bras Soares Passos
Hello Peter,
On Wed, Jun 8, 2022 at 5:23 PM Peter Xu <[email protected]> wrote:
[...]Quoted 10 lines
Quoted 5 lines
In a previous iteration of the patchset, it was made clear that it's desirable to detect when the kernel falls back to copying mechanism, so the user of 'QIOChannelSocket' can switch to copying and avoid the overhead. This was done by the return value of
flush(), which is 1 if that occurs.Two questions..
1) When that happens, will
MSG_ERRQUEUEkeeps working just like zerocopy is functional?I am not sure about what exactly you meant by 'like zerocopy is funcional', but the
idea is that reading fromMSG_ERRQUEUEshould return a msg for each sendmsg syscall withMSG_ZEROCOPYthat previously happened. This does not depend on the outcome (like falling back to the copying mechanism).
btw, most of those messages may be batched to reduce overhead.Quoted 5 lines
At some point, zero-copy may fail, and fall back to copying, so in those messages
an error codeSO_EE_CODE_ZEROCOPY_COPIEDcan be seen. Having only those messages in a flush will trigger the returning of 1 from the flush function.Ah I think I missed the "reset ret==0 when !SO_EE_CODE_ZEROCOPY_COPIED" path.. Sorry.
Quoted 40 lines
Quoted 5 lines
If the answer is yes, I don't see how ret=1 will ever be returned.. because we'll also go into the same loop in
qio_channel_socket_flush()anyway.We set ret to 1 at function entry and then for each message in the
MSG_ERRQUEUE, we test if it has error code different thanSO_EE_CODE_ZEROCOPY_COPIED. If it ever have a different error code, we set ret=0.So, in our previous example, if we have a net device not supporting the 'Scatter-Gather'
feature (NETIF_F_SG), every error message will beSO_EE_CODE_ZEROCOPY_COPIED, and it will return 1.Quoted 4 lines
If the answer is no, then since we'll have non-zero
zero_copy_queued, will the loop inqio_channel_socket_flush()go into a dead one? How could it return?No, because it will go through all packets sent with
MSG_ZEROCOPY, including the ones that fell back to copying, so the counter should be fine. If any code disables
zero-copy, it will both stop sending stuff wilMSG_ZEROCOPYand flushing, so it should be fine.Quoted 4 lines
2) Even if we have the correct ret=1 returned when that happens, which
caller is detecting that ret==1 and warn the admin?No caller is using that right now.
It's supposed to be a QIOChannel interface feature, and any user/implementation could use that information to warn if zero-copy is not being used, fall back to copying directly (to avoid overhead of testing zero-copy) or even use it to cancel the
sending if wanted.
Quoted 9 lines
Quoted 1 line
It was a suggestion of Daniel on top of [PATCH v5 1/6] IIRC.
OK the detection makes sense, thanks for the details.
Then now I'm wondering whether we should have warned the admin already if zero-copy send is not fully enabled in live migration. Should we add a
error_report_once()somewhere for the ret==1 already? After all the user specify zero_copy_send=true explicitly. Did I miss something again?
You are correct, I think warning the user is the valid thing to have here. At the end of the first iteration, where the first flush happens, I think it's too late to
fail the migration, since a huge lot of the data has already been sent.
Best regards,
Leo
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
In reply to 徐闯Hello 徐闯,
Thanks for reviewing!
On Wed, Jun 1, 2022 at 6:37 AM 徐闯 <[email protected]> wrote:
[...]
Quoted 6 lines
Hi, Leonardo. I'm also paying attention to the application of
MSG_ZEROCOPYin live migration recently. I noticed that you defined a memberzero_copy_queuedin the struct QIOChannelSocket, but I can't find out where the value of this member has been changed in your patch. Can you answer it for me?
You are right.
This was being correctly implemented until v6, and then the increment just vanished.
Since v6 there were a lot of changes both in the patch and in the base repository, so I think I completely missed it in some change iteration.
I will send a fix shortly.
Is that ok if I include a "Reported-by: 徐闯
<[email protected]>" in the patch?
Best regards,
Leo
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 3 lines · Leonardo Bras Soares Passos
I will send a fix shortly.
Is that ok if I include a "Reported-by: 徐闯
<[email protected]>" in the patch?
okay.
Best Regards,
chuang xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
In reply to Leonardo BrasQuoted 26 lines · Leonardo Bras
io/channel-socket.c +14
-4@@ -557,15 +578,31 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,memcpy (CMSG_DATA (cmsg), fds, fdsize); } +#ifdef QEMU_MSG_ZEROCOPY +if (flags &QIO_CHANNEL_WRITE_FLAG_ZERO_COPY ) { + sflags =MSG_ZEROCOPY ; + } +#endif + retry: - ret =sendmsg (sioc->fd, &msg,0 ); + ret =sendmsg (sioc->fd, &msg, sflags); if (ret <= 0) { -if (errno ==EAGAIN ) { + switch (errno) { + case EAGAIN: return QIO_CHANNEL_ERR_BLOCK; - } -if (errno ==EINTR ) { + case EINTR: goto retry; +#ifdef QEMU_MSG_ZEROCOPY + case ENOBUFS: + if (sflags & MSG_ZEROCOPY) { + error_setg_errno(errp, errno,
io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 608
Quoted 10 lines
io/channel-socket.c +6
-0+ "Process can't lock enough memory for using MSG_ZEROCOPY"); + return -1; + } +break ; +#endif } +error_setg_errno (errp, errno,"Unable to write to socket" );return -1 ;
Hi, Leo.
There are some other questions I would like to discuss with you.
I tested the multifd zero_copy migration and found that sometimes even if max locked memory of qemu was set to 16GB(much greater than MULTIFD_PACKET_SIZE), the error "Process can't lock enough memory for using MSG_ZEROCOPY" would still be reported.
I noticed that the doc(https://www.kernel.org/doc/html/v5.12/networking/msg_zerocopy.html) says "A zerocopy failure will return -1 with errno ENOBUFS. This happens if the socket option was not set, _the socket exceeds its optmem limit_ or the user exceeds its ulimit on locked pages."
I also found that the RFC(https://lwn.net/Articles/715279/) says _"__The change to allocate notification skbuffs from optmem requires__ensuring that net.core.optmem is at least a few 100KB."_
On my host, optmem was initially set to 20KB, I tried to change it to 100KB (echo 102400 > /proc/sys/net/core/optmem_max) as the RFC says.Then I tested the multifd zero_copy migration repeatedly,and the error disappeared.
So when sendmsg returns -1 with errno ENOBUFS, should we distinguish between error ''socket exceeds optmem limit" and error "user exceeds ulimit on locked pages"? Or is there any better way to avoid this problem?
Best Regards,
chuang xu
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 53 lines · * chuang xu ([email protected]) wrote:
Quoted 25 lines · Leonardo Bras
io/channel-socket.c +14
-4@@ -557,15 +578,31 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,memcpy (CMSG_DATA (cmsg), fds, fdsize); } +#ifdef QEMU_MSG_ZEROCOPY +if (flags &QIO_CHANNEL_WRITE_FLAG_ZERO_COPY ) { + sflags =MSG_ZEROCOPY ; + } +#endif + retry: - ret =sendmsg (sioc->fd, &msg,0 ); + ret =sendmsg (sioc->fd, &msg, sflags); if (ret <= 0) { -if (errno ==EAGAIN ) { + switch (errno) { + case EAGAIN: return QIO_CHANNEL_ERR_BLOCK; - } -if (errno ==EINTR ) { + case EINTR: goto retry; +#ifdef QEMU_MSG_ZEROCOPY + case ENOBUFS: + if (sflags & MSG_ZEROCOPY) { + error_setg_errno(errp, errno,io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 607
Quoted 10 lines
io/channel-socket.c +6
-0+ "Process can't lock enough memory for using MSG_ZEROCOPY"); + return -1; + } +break ; +#endif } +error_setg_errno (errp, errno,"Unable to write to socket" );return -1 ;Hi, Leo.
There are some other questions I would like to discuss with you.
I tested the multifd
zero_copymigration and found that sometimes even if max locked memory of qemu was set to 16GB(much greater thanMULTIFD_PACKET_SIZE), the error "Process can't lock enough memory for usingMSG_ZEROCOPY" would still be reported.I noticed that the doc(https://www.kernel.org/doc/html/v5.12/networking/msg_zerocopy.html) says "A zerocopy failure will return -1 with errno ENOBUFS. This happens if the socket option was not set,
_thesocket exceeds its optmemlimit_or the user exceeds its ulimit on locked pages."
Quoted 3 lines
I also found that the RFC(https://lwn.net/Articles/715279/) says _"__The change to allocate notification skbuffs from optmem
requires__ensuringthat net.core.optmem is at least a few 100KB."_
Interesting.
Quoted 7 lines
On my host, optmem was initially set to 20KB, I tried to change it to 100KB (echo 102400 >
/proc/sys/net/core/optmem_max) as the RFC says.Then I tested the multifdzero_copymigration repeatedly,and the error disappeared.So when sendmsg returns -1 with errno ENOBUFS, should we distinguish between error ''socket exceeds optmem limit" and error "user exceeds ulimit on locked pages"? Or is there any better way to avoid this problem?
I don't think we can tell which one of them triggered the error; so the only thing I can suggest is that we document the need for optmem_max setting; I wonder how we get a better answer than 'a few 100KB'? I guess it's something like the number of packets inflight * sizeof(cmsghdr) ?
Dave
Quoted 3 lines
Best Regards,
chuang xu
Signature · 1 line
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [External] [PATCH v13 3/8] QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX
Quoted 7 lines · Dr. David Alan Gilbert
I don't think we can tell which one of them triggered the error; so the only thing I can suggest is that we document the need for
optmem_maxsetting; I wonder how we get a better answer than 'a few 100KB'? I guess it's something like the number of packets inflight * sizeof(cmsghdr) ?Dave
Three cases with errno ENOBUFS are described in the official doc(https://www.kernel.org/doc/html/v5.12/networking/msg_zerocopy.html):
1.The socket option was not set
2.The socket exceeds its optmem limit
3.The user exceeds its ulimit on locked pages
For case 1, if the code logic is correct, this possibility can be ignored.
For case 2, I asked a kernel developer about the reason for "a few 100KB". He said that the recommended value should be for the purpose of improving the performance of zero_copy send. If the NICsends data slower than the data generation speed, even if optmem is set to 100KB, there is a probability that sendmsg returns with errno ENOBUFS.
For case 3, If I do not set max locked memory for the qemu, the max locked memory will be unlimited. I set the max locked memory for qemu and found that once the memory usage exceeds the max locked memory, oom will occur. Does this mean that sendmsg cannot return with errno ENOBUFS at all when user exceeds its ulimit on locked pages?
If the above is true, can we take the errno as the case 2?
I modified the code logic to call sendmsg again when the errno is ENOBUFS and set optmem to the initial 20KB(echo 20480 > /proc/sys/net/core/optmem_max), now the multifd zero_copy migration goes well.
Here are the changes I made to the code:
---
io/channel-socket.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
io/channel-socket.c +0 -0
@@ -595,9 +595,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
#ifdef QEMU_MSG_ZEROCOPY
case ENOBUFS:
if (sflags & MSG_ZEROCOPY) {
- error_setg_errno(errp, errno,
- "Process can't lock enough memory for using MSG_ZEROCOPY"); - return -1;
+ goto retry;
}
break;
#endif
Signature · 5 lines
Dave, what's your take? Best Regards, chuang xu
PATCH v13 4/8 migration: Add zero-copy-send parameter for QMP/HMP for Linux
In reply to Leonardo BrasAdd property that allows zero-copy migration of memory pages on the sending side, and also includes a helper function migrate_use_zero_copy_send() to check if it's enabled.
No code is introduced to actually do the migration, but it allow future implementations to enable/disable this feature.
On non-Linux builds this parameter is compiled-out.
--- qapi/migration.json | 24 ++++++++++++++++++++++++ migration/migration.h | 5 +++++ migration/migration.c | 32 ++++++++++++++++++++++++++++++++ migration/socket.c | 11 +++++++++-- monitor/hmp-cmds.c | 6 ++++++ 5 files changed, 76 insertions(+), 2 deletions(-)
qapi/migration.json +24 -0
@@ -741,6 +741,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy-send: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. +# Requires that QEMU be permitted to use locked memory +# for guest RAM pages. +# Defaults to false. (Since 7.1) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -780,6 +787,7 @@ 'xbzrle-cache-size', 'max-postcopy-bandwidth', 'max-cpu-throttle', 'multifd-compression', 'multifd-zlib-level' ,'multifd-zstd-level', + { 'name': 'zero-copy-send', 'if' : 'CONFIG_LINUX'}, 'block-bitmap-mapping' ] } ##
@@ -906,6 +914,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy-send: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. +# Requires that QEMU be permitted to use locked memory +# for guest RAM pages. +# Defaults to false. (Since 7.1) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -960,6 +975,7 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', + '*zero-copy-send': { 'type': 'bool', 'if': 'CONFIG_LINUX' }, '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } ##
@@ -1106,6 +1122,13 @@ # will consume more CPU. # Defaults to 1. (Since 5.0) # +# @zero-copy-send: Controls behavior on sending memory pages on migration. +# When true, enables a zero-copy mechanism for sending +# memory pages, if host supports it. +# Requires that QEMU be permitted to use locked memory +# for guest RAM pages. +# Defaults to false. (Since 7.1) +# # @block-bitmap-mapping: Maps block nodes and bitmaps on them to # aliases for the purpose of dirty bitmap migration. Such # aliases may for example be the corresponding names on the
@@ -1158,6 +1181,7 @@ '*multifd-compression': 'MultiFDCompression', '*multifd-zlib-level': 'uint8', '*multifd-zstd-level': 'uint8', + '*zero-copy-send': { 'type': 'bool', 'if': 'CONFIG_LINUX' }, '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } } ##
migration/migration.h +5 -0
@@ -375,6 +375,11 @@ MultiFDCompression migrate_multifd_compression(void);int migrate_multifd_zlib_level (void );int migrate_multifd_zstd_level (void ); +#ifdef CONFIG_LINUX +bool migrate_use_zero_copy_send (void ); +#else +#define migrate_use_zero_copy_send () (false) +#endif int migrate_use_xbzrle (void );uint64_t migrate_xbzrle_cache_size (void );bool migrate_colo_enabled (void );
migration/migration.c +32 -0
@@ -910,6 +910,10 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) params->multifd_zlib_level = s->parameters.multifd_zlib_level; params->has_multifd_zstd_level = true; params->multifd_zstd_level = s->parameters.multifd_zstd_level; +#ifdef CONFIG_LINUX + params->has_zero_copy_send = true; + params->zero_copy_send = s->parameters.zero_copy_send; +#endif params->has_xbzrle_cache_size = true; params->xbzrle_cache_size = s->parameters.xbzrle_cache_size; params->has_max_postcopy_bandwidth = true;
@@ -1567,6 +1571,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,if (params->has_multifd_compression) { dest->multifd_compression = params->multifd_compression; } +#ifdef CONFIG_LINUX +if (params->has_zero_copy_send) { + dest->zero_copy_send = params->zero_copy_send; + } +#endif if (params->has_xbzrle_cache_size) { dest->xbzrle_cache_size = params->xbzrle_cache_size; }
@@ -1679,6 +1688,11 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)if (params->has_multifd_compression) { s->parameters.multifd_compression = params->multifd_compression; } +#ifdef CONFIG_LINUX +if (params->has_zero_copy_send) { + s->parameters.zero_copy_send = params->zero_copy_send; + } +#endif if (params->has_xbzrle_cache_size) { s->parameters.xbzrle_cache_size = params->xbzrle_cache_size;xbzrle_cache_resize (params->xbzrle_cache_size, errp);
@@ -2563,6 +2577,17 @@ int migrate_multifd_zstd_level(void)return s->parameters.multifd_zstd_level; } +#ifdef CONFIG_LINUX +bool migrate_use_zero_copy_send (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->parameters.zero_copy_send; +} +#endif +int migrate_use_xbzrle (void ) {MigrationState *s;
@@ -4206,6 +4231,10 @@ static Property migration_properties[] = { DEFINE_PROP_UINT8("multifd-zstd-level", MigrationState, parameters.multifd_zstd_level, DEFAULT_MIGRATE_MULTIFD_ZSTD_LEVEL), +#ifdef CONFIG_LINUX +DEFINE_PROP_BOOL ("zero_copy_send" , MigrationState, + parameters.zero_copy_send, false), +#endifDEFINE_PROP_SIZE ("xbzrle-cache-size" , MigrationState, parameters.xbzrle_cache_size, DEFAULT_MIGRATE_XBZRLE_CACHE_SIZE),
@@ -4303,6 +4332,9 @@ static void migration_instance_init(Object *obj) params->has_multifd_compression = true; params->has_multifd_zlib_level = true; params->has_multifd_zstd_level = true; +#ifdef CONFIG_LINUX + params->has_zero_copy_send = true; +#endif params->has_xbzrle_cache_size = true; params->has_max_postcopy_bandwidth = true; params->has_max_cpu_throttle = true;
migration/socket.c +9 -2
@@ -74,9 +74,16 @@ static void socket_outgoing_migration(QIOTask *task,if (qio_task_propagate_error (task, &err)) {trace_migration_socket_outgoing_error (error_get_pretty (err)); - }else { -trace_migration_socket_outgoing_connected (data->hostname); + goto out; } + +trace_migration_socket_outgoing_connected (data->hostname); + +if (migrate_use_zero_copy_send ()) { +error_setg (&err,"Zero copy send not available in migration" ); + } + +out:migration_channel_connect (data->s, sioc, data->hostname, err);object_unref (OBJECT (sioc)); }
monitor/hmp-cmds.c +6 -0
@@ -1309,6 +1309,12 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) p->has_multifd_zstd_level = true;visit_type_uint8 (v, param, &p->multifd_zstd_level, &err);break ; +#ifdef CONFIG_LINUX +case MIGRATION_PARAMETER_ZERO_COPY_SEND : + p->has_zero_copy_send = true; +visit_type_bool (v, param, &p->zero_copy_send, &err); +break ; +#endif case MIGRATION_PARAMETER_XBZRLE_CACHE_SIZE : p->has_xbzrle_cache_size = true;if (!visit_type_size (v, param, &cache_size, &err)) {
Signature · 1 line
2.36.1
PATCH v13 5/8 migration: Add migrate_use_tls() helper
In reply to Leonardo BrasA lot of places check parameters.tls_creds in order to evaluate if TLS is in use, and sometimes call migrate_get_current() just for that test.
Add new helper function migrate_use_tls() in order to simplify testing for TLS usage.
--- migration/migration.h | 1 + migration/channel.c | 3 +-- migration/migration.c | 9 +++++++++ migration/multifd.c | 5 +---- 4 files changed, 12 insertions(+), 6 deletions(-)
migration/migration.h +1 -0
@@ -380,6 +380,7 @@ bool migrate_use_zero_copy_send(void);#else #define migrate_use_zero_copy_send () (false)#endif +int migrate_use_tls (void );int migrate_use_xbzrle (void );uint64_t migrate_xbzrle_cache_size (void );bool migrate_colo_enabled (void );
migration/channel.c +1 -2
@@ -38,8 +38,7 @@ void migration_channel_process_incoming(QIOChannel *ioc)trace_migration_set_incoming_channel ( ioc,object_get_typename (OBJECT (ioc))); -if (s->parameters.tls_creds && - *s->parameters.tls_creds && +if (migrate_use_tls () && !object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_TLS )) {migration_tls_channel_process_incoming (s, ioc, &local_err);
migration/migration.c +9 -0
@@ -2588,6 +2588,15 @@ bool migrate_use_zero_copy_send(void) }#endif +int migrate_use_tls (void ) +{ +MigrationState *s; + + s =migrate_get_current (); + +return s->parameters.tls_creds && *s->parameters.tls_creds; +} +int migrate_use_xbzrle (void ) {MigrationState *s;
migration/multifd.c +1 -4
@@ -782,15 +782,12 @@ static bool multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc, Error *error) { -MigrationState *s =migrate_get_current (); -trace_multifd_set_outgoing_channel ( ioc,object_get_typename (OBJECT (ioc)),migrate_get_current ()->hostname, error);if (!error) { -if (s->parameters.tls_creds && - *s->parameters.tls_creds && +if (migrate_use_tls () && !object_dynamic_cast (OBJECT (ioc),TYPE_QIO_CHANNEL_TLS )) {multifd_tls_channel_connect (p, ioc, &error);
Signature · 1 line
2.36.1
PATCH v13 6/8 multifd: multifd_send_sync_main now returns negative on error
In reply to Leonardo BrasEven though multifd_send_sync_main() currently emits error_reports, it's callers don't really check it before continuing.
Change multifd_send_sync_main() to return -1 on error and 0 on success. Also change all it's callers to make use of this change and possibly fail earlier.
(This change is important to next patch on multifd zero copy implementation, to make it sure an error in zero-copy flush does not go unnoticed.
--- migration/multifd.h | 2 +- migration/multifd.c | 10 ++++++---- migration/ram.c | 29 ++++++++++++++++++++++------- 3 files changed, 29 insertions(+), 12 deletions(-)
migration/multifd.h +1 -1
@@ -20,7 +20,7 @@ int multifd_load_cleanup(Error **errp);bool multifd_recv_all_channels_created (void );bool multifd_recv_new_channel (QIOChannel *ioc,Error **errp);void multifd_recv_sync_main (void ); -void multifd_send_sync_main (QEMUFile *f); +int multifd_send_sync_main (QEMUFile *f);int multifd_queue_page (QEMUFile *f,RAMBlock *block,ram_addr_t offset);/* Multifd Compression flags */
migration/multifd.c +6 -4
@@ -566,17 +566,17 @@ void multifd_save_cleanup(void) multifd_send_state =NULL ; } -void multifd_send_sync_main (QEMUFile *f) +int multifd_send_sync_main (QEMUFile *f) {int i;if (!migrate_use_multifd ()) { -return ; +return 0 ; }if (multifd_send_state->pages->num) {if (multifd_send_pages (f) <0 ) {error_report ("%s: multifd_send_pages fail" , __func__); -return ; +return -1 ; } }for (i =0 ; i <migrate_multifd_channels (); i++) {
@@ -589,7 +589,7 @@ void multifd_send_sync_main(QEMUFile *f)if (p->quit) {error_report ("%s: channel %d has already quit" , __func__, i);qemu_mutex_unlock (&p->mutex); -return ; +return -1 ; } p->packet_num = multifd_send_state->packet_num++;
@@ -608,6 +608,8 @@ void multifd_send_sync_main(QEMUFile *f)qemu_sem_wait (&p->sem_sync); }trace_multifd_send_sync_main (multifd_send_state->packet_num); + +return 0 ; }static void *multifd_send_thread (void *opaque)
migration/ram.c +22 -7
@@ -2909,6 +2909,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) {RAMState **rsp = opaque;RAMBlock *block; +int ret;if (compress_threads_save_setup ()) {return -1 ;
@@ -2943,7 +2944,11 @@ static int ram_save_setup(QEMUFile *f, void *opaque)ram_control_before_iterate (f,RAM_CONTROL_SETUP );ram_control_after_iterate (f,RAM_CONTROL_SETUP ); -multifd_send_sync_main (f); + ret =multifd_send_sync_main (f); +if (ret <0 ) { +return ret; + } +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS );qemu_fflush (f);
@@ -3052,7 +3057,11 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) out:if (ret >=0 &&migration_is_setup_or_active (migrate_get_current ()->state)) { -multifd_send_sync_main (rs->f); + ret =multifd_send_sync_main (rs->f); +if (ret <0 ) { +return ret; + } +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS );qemu_fflush (f);ram_transferred_add (8 );
@@ -3112,13 +3121,19 @@ static int ram_save_complete(QEMUFile *f, void *opaque)ram_control_after_iterate (f,RAM_CONTROL_FINISH ); } -if (ret >=0 ) { -multifd_send_sync_main (rs->f); -qemu_put_be64 (f,RAM_SAVE_FLAG_EOS ); -qemu_fflush (f); +if (ret <0 ) { +return ret; } -return ret; + ret =multifd_send_sync_main (rs->f); +if (ret <0 ) { +return ret; + } + +qemu_put_be64 (f,RAM_SAVE_FLAG_EOS ); +qemu_fflush (f); + +return 0 ; } static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
Signature · 1 line
2.36.1
PATCH v13 7/8 multifd: Send header packet without flags if zero-copy-send is enabled
In reply to Leonardo BrasSince d48c3a0445 ("multifd: Use a single writev on the send side"), sending the header packet and the memory pages happens in the same writev, which can potentially make the migration faster.
Using channel-socket as example, this works well with the default copying mechanism of sendmsg(), but with zero-copy-send=true, it will cause the migration to often break.
This happens because the header packet buffer gets reused quite often, and there is a high chance that by the time the MSG_ZEROCOPY mechanism get to send the buffer, it has already changed, sending the wrong data and causing the migration to abort.
It means that, as it is, the buffer for the header packet is not suitable for sending with MSG_ZEROCOPY.
In order to enable zero copy for multifd, send the header packet on an individual write(), without any flags, and the remanining pages with a writev(), as it was happening before. This only changes how a migration with zero-copy-send=true works, not changing any current behavior for migrations with zero-copy-send=false.
--- migration/multifd.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
migration/multifd.c +19 -3
@@ -617,6 +617,7 @@ static void *multifd_send_thread(void *opaque)MultiFDSendParams *p = opaque;Error *local_err =NULL ;int ret =0 ; +bool use_zero_copy_send =migrate_use_zero_copy_send ();trace_multifd_send_thread_start (p->id);rcu_register_thread ();
@@ -639,9 +640,14 @@ static void *multifd_send_thread(void *opaque)if (p->pending_job) {uint64_t packet_num = p->packet_num;uint32_t flags = p->flags; - p->iovs_num =1 ; p->normal_num =0 ; +if (use_zero_copy_send) { + p->iovs_num =0 ; + }else { + p->iovs_num =1 ; + } +for (int i =0 ; i < p->pages->num; i++) { p->normal[p->normal_num] = p->pages->offset[i]; p->normal_num++;
@@ -665,8 +671,18 @@ static void *multifd_send_thread(void *opaque)trace_multifd_send (p->id, packet_num, p->normal_num, flags, p->next_packet_size); - p->iov[0 ].iov_len = p->packet_len; - p->iov[0 ].iov_base = p->packet; +if (use_zero_copy_send) { +/* Send header first, without zerocopy */ + ret =qio_channel_write_all (p->c, (void *)p->packet, + p->packet_len, &local_err); +if (ret !=0 ) { +break ; + } + }else { +/* Send header using the same writev call */ + p->iov[0 ].iov_len = p->packet_len; + p->iov[0 ].iov_base = p->packet; + } ret =qio_channel_writev_all (p->c, p->iov, p->iovs_num, &local_err);
Signature · 1 line
2.36.1
PATCH v13 8/8 multifd: Implement zero copy write in multifd migration (multifd-zero-copy)
In reply to Leonardo BrasImplement zero copy send on nocomp_send_write(), by making use of QIOChannel writev + flags & flush interface.
Change multifd_send_sync_main() so flush_zero_copy() can be called after each iteration in order to make sure all dirty pages are sent before a new iteration is started. It will also flush at the beginning and at the end of migration.
Also make it return -1 if flush_zero_copy() fails, in order to cancel the migration process, and avoid resuming the guest in the target host without receiving all current RAM.
This will work fine on RAM migration because the RAM pages are not usually freed, and there is no problem on changing the pages content between writev_zero_copy() and the actual sending of the buffer, because this change will dirty the page and cause it to be re-sent on a next iteration anyway.
A lot of locked memory may be needed in order to use multifd migration with zero-copy enabled, so disabling the feature should be necessary for low-privileged users trying to perform multifd migrations.
--- migration/multifd.h | 2 ++ migration/migration.c | 11 ++++++++++- migration/multifd.c | 37 +++++++++++++++++++++++++++++++++++-- migration/socket.c | 5 +++-- 4 files changed, 50 insertions(+), 5 deletions(-)
migration/multifd.h +2 -0
@@ -92,6 +92,8 @@ typedef struct {uint32_t packet_len;/* pointer to the packet */ MultiFDPacket_t *packet; +/* multifd flags for sending ram */ +int write_flags;/* multifd flags for each packet */ uint32_t flags;/* size of the next packet that contains pages */
migration/migration.c +10 -1
@@ -1497,7 +1497,16 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)error_prepend (errp,"Invalid mapping given for block-bitmap-mapping: " );return false; } - +#ifdef CONFIG_LINUX +if (params->zero_copy_send && + (!migrate_use_multifd () || + params->multifd_compression !=MULTIFD_COMPRESSION_NONE || + (params->tls_creds && *params->tls_creds))) { +error_setg (errp, +"Zero copy only available for non-compressed non-TLS multifd migration" ); +return false; + } +#endif return true; }
migration/multifd.c +35 -2
@@ -569,6 +569,7 @@ void multifd_save_cleanup(void)int multifd_send_sync_main (QEMUFile *f) {int i; +bool flush_zero_copy;if (!migrate_use_multifd ()) {return 0 ;
@@ -579,6 +580,20 @@ int multifd_send_sync_main(QEMUFile *f)return -1 ; } } + +/* +* When using zero-copy, it's necessary to flush the pages before any of +* the pages can be sent again, so we'll make sure the new version of the +* pages will always arrive _later_ than the old pages. +* +* Currently we achieve this by flushing the zero-page requested writes +* per ram iteration, but in the future we could potentially optimize it +* to be less frequent, e.g. only after we finished one whole scanning of +* all the dirty bitmaps. +*/ + + flush_zero_copy =migrate_use_zero_copy_send (); +for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDSendParams *p = &multifd_send_state->params[i];
@@ -600,6 +615,17 @@ int multifd_send_sync_main(QEMUFile *f) ram_counters.transferred += p->packet_len;qemu_mutex_unlock (&p->mutex);qemu_sem_post (&p->sem); + +if (flush_zero_copy && p->c) { +int ret; +Error *err =NULL ; + + ret =qio_channel_flush (p->c, &err); +if (ret <0 ) { +error_report_err (err); +return -1 ; + } + } }for (i =0 ; i <migrate_multifd_channels (); i++) {MultiFDSendParams *p = &multifd_send_state->params[i];
@@ -684,8 +710,8 @@ static void *multifd_send_thread(void *opaque) p->iov[0 ].iov_base = p->packet; } - ret =qio_channel_writev_all (p->c, p->iov, p->iovs_num, - &local_err); + ret =qio_channel_writev_full_all (p->c, p->iov, p->iovs_num,NULL , +0 , p->write_flags, &local_err);if (ret !=0 ) {break ; }
@@ -913,6 +939,13 @@ int multifd_save_setup(Error **errp)/* We need one extra place for the packet header */ p->iov = g_new0(struct iovec, page_count + 1); p->normal =g_new0 (ram_addr_t, page_count); + +if (migrate_use_zero_copy_send ()) { + p->write_flags =QIO_CHANNEL_WRITE_FLAG_ZERO_COPY ; + }else { + p->write_flags =0 ; + } +socket_send_channel_create (multifd_new_send_channel_async, p); }
migration/socket.c +3 -2
@@ -79,8 +79,9 @@ static void socket_outgoing_migration(QIOTask *task,trace_migration_socket_outgoing_connected (data->hostname); -if (migrate_use_zero_copy_send ()) { -error_setg (&err,"Zero copy send not available in migration" ); +if (migrate_use_zero_copy_send () && + !qio_channel_has_feature (sioc,QIO_CHANNEL_FEATURE_WRITE_ZERO_COPY )) { +error_setg (&err,"Zero copy send feature not detected in host kernel" ); } out:
Signature · 1 line
2.36.1