Zero copy improvements (QIOChannel + multifd)
v3 cover Zero copy improvements (QIOChannel + multifd)
The first patch avoid spuriously returning 1 [*] when zero-copy flush is attempted before any buffer was sent using MSG_ZEROCOPY.
[*] zero-copy not being used, even though it's enabled and supported by kernel
The second patch introduces a new migration stat (dirty-sync-missed-zero-copy) that will be used to keep track of [*].
The third patch keeps track of how many zero-copy flushes retured 1 [*]
Changes since v2:
- Documentation release number changed from 7.2 to 7.1
- migration stat renamed from zero-copy-copied to dirty-sync-missed-zero-copy
- Updated documentation to make it more user-friendly
Changes since v1:
- Idea of using a warning replaced by using a migration stat counter
Leonardo Bras (3):
QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing
sent
Add dirty-sync-missed-zero-copy migration stat migration/multifd: Warn user when zerocopy not working
qapi/migration.json | 7 ++++++- migration/ram.h | 2 ++ io/channel-socket.c | 8 +++++++- migration/migration.c | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ monitor/hmp-cmds.c | 4 ++++ 7 files changed, 28 insertions(+), 2 deletions(-)
Signature · 1 line
2.36.1
PATCH v3 1/3 QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().
Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.
--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
io/channel-socket.c +7 -1
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) { +return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) {
Signature · 1 line
2.36.1
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 4 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().
Quoted 8 lines
Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
And if this merges via migration maintainers' tree
Quoted 28 lines
io/channel-socket.c +7
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) { +return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) {Signature · 1 line
2.36.1
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Juan Quintela
Reviewed-byRe: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 14 lines · Daniel P. Berrangé
Quoted 4 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().Quoted 8 lines
Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 5 lines
Reviewed-by Daniel P. Berrangé <[email protected]>And if this merges via migration maintainers' tree
I will add this on the next PULLL request.
Thanks.
Quoted 31 lines
Quoted 28 lines
io/channel-socket.c +7
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) { +return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) {Signature · 1 line
2.36.1With regards,
Daniel
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
In reply to Leonardo BrasHi, Leo,
Quoted 18 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)io/channel-socket.c +0
-0
io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721
Quoted 7 lines
io/channel-socket.c +3
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) {
I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?
Quoted 15 lines
io/channel-socket.c +4
-0+return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) {Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Hello Peter,
Quoted 17 lines · Peter Xu
Hi, Leo,
Quoted 13 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 15 lines
io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721
Quoted 12 lines
io/channel-socket.c +3
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) {I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?
This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.
Without this, the function could return 1 if nothing was sent with zero-copy, and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.
Best regards,
Leo
Quoted 20 lines
Quoted 14 lines
io/channel-socket.c +4
-1+return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) { --Quoted 2 lines
2.36.1
--
Peter Xu
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 36 lines · Leonardo Bras Soares Passos
Hello Peter,
Quoted 17 lines · Peter Xu
Hi, Leo,
Quoted 13 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>--- io/channel-socket.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)Quoted 15 lines
io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721
Quoted 12 lines
io/channel-socket.c +3
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,struct cmsghdr *cm;char control[CMSG_SPACE (sizeof (*serr))];int received; -int ret =1 ; +int ret; + +if (!sioc->zero_copy_queued) {I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?
Quoted 6 lines
This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.
Without this, the function could return 1 if nothing was sent with zero-copy, and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.
I know it won't happen in practise, but.. what if we call flush() without sending anything zero-copy-wise at all (so zero_copy_sent > 0, zero_copy_queued > 0, meanwhile zero_copy_sent == zero_copy_queued)? Then IIUC we'll return 1 even if we didn't do any fallback, or am I wrong?
Quoted 25 lines
Best regards,
LeoQuoted 20 lines
Quoted 14 lines
io/channel-socket.c +4
-1+return 0 ; + } msg.msg_control = control; msg.msg_controllen =sizeof (control);memset (control,0 ,sizeof (control)); + ret =1 ; +while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received =recvmsg (sioc->fd, &msg,MSG_ERRQUEUE );if (received <0 ) { --Quoted 2 lines
2.36.1
--
Peter Xu
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 43 lines · Peter Xu
Quoted 42 lines · Leonardo Bras Soares Passos
Hello Peter,
Quoted 19 lines · Peter Xu
Hi, Leo,
Quoted 15 lines · Leonardo Bras
If flush is called when no buffer was sent with
MSG_ZEROCOPY, it currently
returns 1. This return code should be used only when Linux fails to useMSG_ZEROCOPYon a lot ofsendmsg().Fix this by returning early from flush if no
sendmsg(...,MSG_ZEROCOPY)was attempted.Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copyflag &
io_flushforCONFIG_LINUX")Signed-off-by Leonardo Bras <[email protected]>---
io/channel-socket.c | 8 +++++++-1 file changed, 7 insertions(+), 1 deletion(-)Quoted 16 lines
io/channel-socket.c static int qio_channel_socket_flush(QIOChannel · line 722
Quoted 13 lines
io/channel-socket.c +3
-1@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc, struct cmsghdr *cm; char control[CMSG_SPACE(sizeof(*serr))]; int received; - int ret = 1; + int ret; + + if (!sioc->zero_copy_queued) {I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?
This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.
Quoted 8 lines
Quoted 4 lines
Without this, the function could return 1 if nothing was sent with zero- copy,
and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.I know it won't happen in practise, but.. what if we call
flush()without sending anything zero-copy-wise at all (sozero_copy_sent> 0,zero_copy_queued> 0,
On a no-bug scenario:
- If we don't send anything zero-copy wise, zero_copy_queued will never get incremented.
- If we don't get inside the while loop in flush, zero_copy_sent will never be incremented.
But sure, suppose it does get incremented by bug / mistake:
- zero_copy_queued incremented, zero_copy_sent untouched :
On (queued == 0) it will go past the 'if', and get stuck 'forever' in the while
loop, since sent will 'never' get incremented.
On (queued == sent), same.
On (queued <= sent), same.
- zero_copy_queued untouched, zero_copy_sent incremented :
On 'if (queued == 0)' it will not go past the 'if'
On 'if (queued == sent)', will go past the 'if', but will exit on the 'while',
falsely returning 1.
On 'if (queued <= sent)', it will not go past the 'if'.
- zero_copy_queued incremented, zero_copy_sent incremented :
On 'if (queued == 0)', infinite loop as above.
On 'if (queued == sent)',
if sent < queued : infinite loop ,
if sent == queued : won't go past 'if' ,
if sent > queued : will go past the 'if', but will exit on the
'while', falsely returning 1.
On (queued <= sent), infinite loop if sent < queued, not past if otherwise
BTW, I consider it 'infinite loop', but it could also end up returning -1 on any error.
Quoted 2 lines
meanwhile
zero_copy_sent== zero_copy_queued)? Then
IIUC we'll return 1 even if we didn't do any fallback, or am I wrong?
Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.
Best regards,
Leo
Quoted 27 lines
Quoted 26 lines
Best regards,
LeoQuoted 20 lines
Quoted 12 lines
io/channel-socket.c +4
-0+ return 0; + } msg.msg_control = control; msg.msg_controllen = sizeof(control); memset(control, 0, sizeof(control)); + ret = 1; + while (sioc->zero_copy_sent < sioc->zero_copy_queued) { received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);Quoted 4 lines
if (received < 0) {
--
2.36.1--
Peter Xu
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 2 lines · Leonardo Brás
Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.
I'm not sure I strictly follow here..
Imagine the case we do flush() twice without sending anything, then in the 1st flush we'll see queued>sent, we'll finish flush() until queued==sent. Then in the 2nd (continuous) flush() we'll see queued==sent immediately.
IIUC with the current patch we'll return 1 which I think is wrong because fallback didn't happen, and if with the change to "if (queued==sent) return 0" it'll fix it?
Signature · 1 line
Peter Xu
Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent
Quoted 7 lines · Peter Xu
Quoted 2 lines · Leonardo Brás
Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.
I'm not sure I strictly follow here..
Sorry, I was thinking of a different scenario.
Quoted 7 lines
Imagine the case we do
flush()twice without sending anything, then in the 1st flush we'll see queued>sent, we'll finishflush()until queued==sent. Then in the 2nd (continuous)flush()we'll see queued==sent immediately.IIUC with the current patch we'll return 1 which I think is wrong because fallback didn't happen, and if with the change to "if (queued==sent) return 0" it'll fix it?
Yes, you are correct.
It's a possible scenario to have a flush happen just after another without any sending in between.
I will fix it as suggested.
Best regards,
Leo
Quoted 4 lines
--
Peter Xu
PATCH v3 2/3 Add dirty-sync-missed-zero-copy migration stat
In reply to Leonardo Bras--- qapi/migration.json | 7 ++++++- migration/migration.c | 2 ++ monitor/hmp-cmds.c | 4 ++++ 3 files changed, 12 insertions(+), 1 deletion(-)
qapi/migration.json +6 -1
@@ -55,6 +55,10 @@ # @postcopy-bytes: The number of bytes sent during the post-copy phase # (since 7.0). # +# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could +# not avoid copying zero pages. This is between 0 +# and @dirty-sync-count * @multifd-channels. +# (since 7.1) # Since: 0.14 ## { 'struct': 'MigrationStats',
@@ -65,7 +69,8 @@ 'postcopy-requests' : 'int', 'page-size' : 'int', 'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64', 'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64', - 'postcopy-bytes' : 'uint64' } } + 'postcopy-bytes' : 'uint64', + 'dirty-sync-missed-zero-copy' : 'uint64' } } ## # @XBZRLECacheStats:
migration/migration.c +2 -0
@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal_bytes = ram_counters.normal * page_size; info->ram->mbps = s->mbps; info->ram->dirty_sync_count = ram_counters.dirty_sync_count; + info->ram->dirty_sync_missed_zero_copy = + ram_counters.dirty_sync_missed_zero_copy; info->ram->postcopy_requests = ram_counters.postcopy_requests; info->ram->page_size = page_size; info->ram->multifd_bytes = ram_counters.multifd_bytes;
monitor/hmp-cmds.c +4 -0
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)monitor_printf (mon,"postcopy ram: %" PRIu64" kbytes\n" , info->ram->postcopy_bytes >>10 ); } +if (info->ram->dirty_sync_missed_zero_copy) { +monitor_printf (mon,"missed zero-copy on: %" PRIu64" iterations\n" , + info->ram->dirty_sync_missed_zero_copy); + } }if (info->has_disk) {
Signature · 1 line
2.36.1
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
Quoted 1 line · Leonardo Bras
Signed-off-by Leonardo Bras <[email protected]>
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
In reply to Leonardo BrasQuoted 6 lines · Leonardo Bras
Signed-off-by Leonardo Bras <[email protected]>--- qapi/migration.json | 7 ++++++- migration/migration.c | 2 ++ monitor/hmp-cmds.c | 4 ++++ 3 files changed, 12 insertions(+), 1 deletion(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
In reply to Leonardo BrasQuoted 12 lines · Leonardo Bras
Signed-off-by Leonardo Bras <[email protected]>--- qapi/migration.json | 7 ++++++- migration/migration.c | 2 ++ monitor/hmp-cmds.c | 4 ++++ 3 files changed, 12 insertions(+), 1 deletion(-)qapi/migration.json +0
-0
qapi/migration.json line 59
Quoted 5 lines
qapi/migration.json +2
-0@@ -55,6 +55,10 @@ # @postcopy-bytes: The number of bytes sent during the post-copy phase # (since 7.0). # +# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could +# not avoid copying zero pages. This is between 0
Avoid copying zero pages? Isn't this for counting MSG_ZEROCOPY fallbacks?
qapi/migration.json line 60
Quoted 1 line
qapi/migration.json +1
-0+# and @dirty-sync-count * @multifd-channels.
I'd not name it as "dirty-sync-*" because fundamentally the accounting is not doing like that (more in latter patch). I also think we should squash patch 2/3 as patch 3 only started to provide meaningful values.
Quoted 32 lines
qapi/migration.json +3
-1+# (since 7.1) # Since: 0.14 ## { 'struct': 'MigrationStats',@@ -65,7 +69,8 @@ 'postcopy-requests' : 'int', 'page-size' : 'int', 'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64', 'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64', - 'postcopy-bytes' : 'uint64' } } + 'postcopy-bytes' : 'uint64', + 'dirty-sync-missed-zero-copy' : 'uint64' } } ## # @XBZRLECacheStats:migration/migration.c +2
-0@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal_bytes = ram_counters.normal * page_size; info->ram->mbps = s->mbps; info->ram->dirty_sync_count = ram_counters.dirty_sync_count; + info->ram->dirty_sync_missed_zero_copy = + ram_counters.dirty_sync_missed_zero_copy; info->ram->postcopy_requests = ram_counters.postcopy_requests; info->ram->page_size = page_size; info->ram->multifd_bytes = ram_counters.multifd_bytes;monitor/hmp-cmds.c +0
-0
monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312
Quoted 6 lines
monitor/hmp-cmds.c +3
-0@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)monitor_printf (mon,"postcopy ram: %" PRIu64" kbytes\n" , info->ram->postcopy_bytes >>10 ); } +if (info->ram->dirty_sync_missed_zero_copy) { +monitor_printf (mon,"missed zero-copy on: %" PRIu64" iterations\n" , + info->ram->dirty_sync_missed_zero_copy);
I suggest we don't call it "iterations" because it's not the generic mean of iterations.
Quoted 7 lines
monitor/hmp-cmds.c +1
-0+ } }if (info->has_disk) {Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
Hello Peter,
Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras
Signed-off-by Leonardo Bras <[email protected]>--- qapi/migration.json | 7 ++++++- migration/migration.c | 2 ++ monitor/hmp-cmds.c | 4 ++++ 3 files changed, 12 insertions(+), 1 deletion(-)
Quoted 12 lines
qapi/migration.json line 59
Quoted 10 lines
qapi/migration.json +2
-0@@ -55,6 +55,10 @@ # @postcopy-bytes: The number of bytes sent during the post-copy phase # (since 7.0). # +# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could +# not avoid copying zero pages. This is between 0Avoid copying zero pages? Isn't this for counting
MSG_ZEROCOPYfallbacks?
Yes, sorry, I think I got confused at some point between some cuts & pastes. It should be "not avoid copying dirty pages." I will fix that on a V4.
Quoted 5 lines
qapi/migration.json line 60
Quoted 2 lines
qapi/migration.json +1
-0+# and @dirty-sync-count * @multifd-channels.I'd not name it as "dirty-sync-*" because fundamentally the accounting is not doing like that (more in latter patch).
Ok, I will take a look & answer there.
Quoted 2 lines
I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.
IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review. So I thought it would be helpful now.
Quoted 34 lines
Quoted 34 lines
qapi/migration.json +3
-1+# (since 7.1) # Since: 0.14 ## { 'struct': 'MigrationStats',@@ -65,7 +69,8 @@ 'postcopy-requests' : 'int', 'page-size' : 'int', 'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64', 'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64', - 'postcopy-bytes' : 'uint64' } } + 'postcopy-bytes' : 'uint64', + 'dirty-sync-missed-zero-copy' : 'uint64' } } ## # @XBZRLECacheStats:migration/migration.c +2
-0@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->ram->normal_bytes = ram_counters.normal * page_size; info->ram->mbps = s->mbps; info->ram->dirty_sync_count = ram_counters.dirty_sync_count; + info->ram->dirty_sync_missed_zero_copy = + ram_counters.dirty_sync_missed_zero_copy; info->ram->postcopy_requests = ram_counters.postcopy_requests; info->ram->page_size = page_size; info->ram->multifd_bytes = ram_counters.multifd_bytes;monitor/hmp-cmds.c +0
-0@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
Quoted 8 lines
monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312
Quoted 5 lines
monitor/hmp-cmds.c +3
-0info->ram->postcopy_bytes >> 10); } +if (info->ram->dirty_sync_missed_zero_copy) { +monitor_printf (mon,"missed zero-copy on: %" PRIu64" iterations\n" , + info->ram->dirty_sync_missed_zero_copy);I suggest we don't call it "iterations" because it's not the generic mean of iterations.
Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?
Best regards,
Leo
Quoted 12 lines
Quoted 8 lines
monitor/hmp-cmds.c +1
-1+ } }if (info->has_disk) { --2.36.1
--
Peter Xu
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
Quoted 6 lines · Leonardo Bras Soares Passos
Quoted 2 lines
I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review. So I thought it would be helpful now.
Ah, that's fine then.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
monitor/hmp-cmds.c +0
-0@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",Quoted 8 lines
monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312
Quoted 5 lines
monitor/hmp-cmds.c +3
-0info->ram->postcopy_bytes >> 10); } +if (info->ram->dirty_sync_missed_zero_copy) { +monitor_printf (mon,"missed zero-copy on: %" PRIu64" iterations\n" , + info->ram->dirty_sync_missed_zero_copy);I suggest we don't call it "iterations" because it's not the generic mean of iterations.
Quoted 2 lines
Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?
"Zero-copy-send fallbacks happened: xxx times\n"?
Signature · 1 line
Peter Xu
Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat
Quoted 28 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 2 lines
I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review.
So I thought it would be helpful now.Ah, that's fine then.
Quoted 17 lines
Quoted 16 lines
monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict · line 314
Quoted 13 lines
monitor/hmp-cmds.c +3
-0@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n", info->ram->postcopy_bytes >> 10); } + if (info->ram->dirty_sync_missed_zero_copy) { + monitor_printf(mon, "missed zero-copy on: %" PRIu64 " iterations\n", + info->ram->dirty_sync_missed_zero_copy);I suggest we don't call it "iterations" because it's not the generic mean of iterations.
Quoted 4 lines
Quoted 2 lines
Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?
"Zero-copy-send fallbacks happened: xxx times\n"?
Oh, yeah, that will work.
I was thinking on keeping the pattern and ended up thinking what was the correct unit. But this is much simpler and work better.
Best regards,
Leo
Quoted 1 line
PATCH v3 3/3 migration/multifd: Warn user when zerocopy not working
In reply to Leonardo BrasSome errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.
After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.
--- migration/ram.h | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);int ram_write_tracking_start (void );void ram_write_tracking_stop (void ); +void dirty_sync_missed_zero_copy (void ); +#endif
migration/multifd.c +2 -0
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)if (ret <0 ) {error_report_err (err);return -1 ; + }else if (ret ==1 ) { +dirty_sync_missed_zero_copy (); } } }
migration/ram.c +5 -0
@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes) ram_counters.transferred += bytes; } +void dirty_sync_missed_zero_copy (void ) +{ + ram_counters.dirty_sync_missed_zero_copy++; +} +/* used by the search for pages to send */ struct PageSearchStatus {/* Current block being searched */
Signature · 1 line
2.36.1
Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working
Quoted 10 lines · Leonardo Bras
Some errors, like the lack of Scatter-Gather support by the network
interface(NETIF_F_SG)may causesendmsg(...,MSG_ZEROCOPY)to fail on using zero-copy, which causes it to fall back to the default copying mechanism.After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to
sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then incrementdirty_sync_missed_zero_copymigration stat to let the user know about it.
Quoted 6 lines
Signed-off-by Leonardo Bras <[email protected]>--- migration/ram.h | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ 3 files changed, 9 insertions(+)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working
In reply to Leonardo BrasQuoted 33 lines · Leonardo Bras
Some errors, like the lack of Scatter-Gather support by the network
interface(NETIF_F_SG)may causesendmsg(...,MSG_ZEROCOPY)to fail on using zero-copy, which causes it to fall back to the default copying mechanism.After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to
sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then incrementdirty_sync_missed_zero_copymigration stat to let the user know about it.Signed-off-by Leonardo Bras <[email protected]>--- migration/ram.h | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ 3 files changed, 9 insertions(+)migration/ram.h +2
-0@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);int ram_write_tracking_start (void );void ram_write_tracking_stop (void ); +void dirty_sync_missed_zero_copy (void ); +#endif migration/multifd.c +0
-0
migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631
Quoted 8 lines
migration/multifd.c +2
-0@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)if (ret <0 ) {error_report_err (err);return -1 ; + }else if (ret ==1 ) { +dirty_sync_missed_zero_copy (); } } }
I know that Juan is working on some patch to only do multifd_send_sync_main() for each dirty sync, but that's not landed, right?
Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?
The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.
Thanks,
Quoted 19 lines
migration/ram.c +5
-0@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes) ram_counters.transferred += bytes; } +void dirty_sync_missed_zero_copy (void ) +{ + ram_counters.dirty_sync_missed_zero_copy++; +} +/* used by the search for pages to send */ struct PageSearchStatus {/* Current block being searched */ Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working
Hello Peter,
Quoted 38 lines · Peter Xu
Quoted 36 lines · Leonardo Bras
Some errors, like the lack of Scatter-Gather support by the network
interface(NETIF_F_SG)may causesendmsg(...,MSG_ZEROCOPY)to fail on using zero-copy, which causes it to fall back to the default copying mechanism.After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to
sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then incrementdirty_sync_missed_zero_copymigration stat to let the user know about it.Signed-off-by Leonardo Bras <[email protected]>--- migration/ram.h | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ 3 files changed, 9 insertions(+)migration/ram.h +2
-0@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);int ram_write_tracking_start (void );void ram_write_tracking_stop (void ); +void dirty_sync_missed_zero_copy (void ); +#endif migration/multifd.c +0
-0@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)if (ret <0 ) {error_report_err (err);return -1 ;
Quoted 8 lines
migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631
Quoted 5 lines
migration/multifd.c +2
-0+ }else if (ret ==1 ) { +dirty_sync_missed_zero_copy (); } } }I know that Juan is working on some patch to only do
multifd_send_sync_main()for each dirty sync, but that's not landed, right?
That's correct, but I am hoping it should land before the release, so the numbers will match.
Quoted 3 lines
Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?
It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.
Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.
Quoted 3 lines
The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.
Ok, Warning sounds misleading here.
What do you think about 'report' instead?
Best regards,
Leo
Quoted 26 lines
Thanks,
Quoted 17 lines
migration/ram.c +5
-1@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes) ram_counters.transferred += bytes; } +void dirty_sync_missed_zero_copy (void ) +{ + ram_counters.dirty_sync_missed_zero_copy++; +} +/* used by the search for pages to send */ struct PageSearchStatus {/* Current block being searched */ --Quoted 2 lines
2.36.1
--
Peter Xu
Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working
Quoted 58 lines · Leonardo Bras Soares Passos
Hello Peter,
Quoted 38 lines · Peter Xu
Quoted 36 lines · Leonardo Bras
Some errors, like the lack of Scatter-Gather support by the network
interface(NETIF_F_SG)may causesendmsg(...,MSG_ZEROCOPY)to fail on using zero-copy, which causes it to fall back to the default copying mechanism.After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to
sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then incrementdirty_sync_missed_zero_copymigration stat to let the user know about it.Signed-off-by Leonardo Bras <[email protected]>--- migration/ram.h | 2 ++ migration/multifd.c | 2 ++ migration/ram.c | 5 +++++ 3 files changed, 9 insertions(+)migration/ram.h +2
-0@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);int ram_write_tracking_start (void );void ram_write_tracking_stop (void ); +void dirty_sync_missed_zero_copy (void ); +#endif migration/multifd.c +0
-0@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)if (ret <0 ) {error_report_err (err);return -1 ;Quoted 8 lines
migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631
Quoted 5 lines
migration/multifd.c +2
-0+ }else if (ret ==1 ) { +dirty_sync_missed_zero_copy (); } } }I know that Juan is working on some patch to only do
multifd_send_sync_main()for each dirty sync, but that's not landed, right?That's correct, but I am hoping it should land before the release, so the numbers will match.
Quoted 3 lines
Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?
Quoted 8 lines
It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.
Ah okay.
But then I suggest we make sure it lands only after Juan's.. or it won't really match. Also when Juan's patch ready, we'd also double check it will be exactly called once per iteration, or we can get confusing numbers. I assume Juan will take care of that then.
Quoted 7 lines
Quoted 4 lines
The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.
Ok, Warning sounds misleading here.
What do you think about 'report' instead?
Looks good. Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working
Quoted 81 lines · Peter Xu
Quoted 62 lines · Leonardo Bras Soares Passos
Hello Peter,
Quoted 42 lines · Peter Xu
Quoted 40 lines · Leonardo Bras
Some errors, like the lack of Scatter-Gather support by the network
interface(NETIF_F_SG)may causesendmsg(...,MSG_ZEROCOPY)to fail on using
zero-copy, which causes it to fall back to the default copying mechanism.After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to
sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then incrementdirty_sync_missed_zero_copymigration stat to let the user know
about it.Signed-off-by Leonardo Bras <[email protected]>---
migration/ram.h | 2 ++
migration/multifd.c | 2 ++
migration/ram.c | 5 +++++3 files changed, 9 insertions(+)migration/ram.h +2
-0@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void); int ram_write_tracking_start(void); void ram_write_tracking_stop(void); +void dirty_sync_missed_zero_copy (void ); + #endifmigration/multifd.c +1
-0@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f) if (ret < 0) { error_report_err(err); return -1; + } else if (ret == 1) {Quoted 8 lines
migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631
Quoted 4 lines
migration/multifd.c +1
-0+ dirty_sync_missed_zero_copy(); } } }I know that Juan is working on some patch to only do
multifd_send_sync_main()for each dirty sync, but that's not landed, right?That's correct, but I am hoping it should land before the release, so the numbers will match.
Quoted 3 lines
Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?
Quoted 8 lines
It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.
Ah okay.
But then I suggest we make sure it lands only after Juan's.. or it won't really match. Also when Juan's patch ready, we'd also double check it will be exactly called once per iteration, or we can get confusing numbers. I assume Juan will take care of that then.
Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.
Ok, Warning sounds misleading here.
What do you think about 'report' instead?Looks good. Thanks,
Thank you for reviewing, Peter!
Leo