Skip to contentSkip to conversation

Zero copy improvements (QIOChannel + multifd)

PATCH Superseded Leonardo Bras 5 participants 23 messages Jul 4, 2022 – Jul 11, 2022

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover Zero copy improvements (QIOChannel + multifd)

The first patch avoid spuriously returning 1 [*] when zero-copy flush is attempted before any buffer was sent using MSG_ZEROCOPY.

[*] zero-copy not being used, even though it's enabled and supported by kernel

The second patch introduces a new migration stat (dirty-sync-missed-zero-copy) that will be used to keep track of [*].

The third patch keeps track of how many zero-copy flushes retured 1 [*]

Changes since v2:
- Documentation release number changed from 7.2 to 7.1
- migration stat renamed from zero-copy-copied to dirty-sync-missed-zero-copy
- Updated documentation to make it more user-friendly

Changes since v1:
- Idea of using a warning replaced by using a migration stat counter

Leonardo Bras (3):
QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing
sent
Add dirty-sync-missed-zero-copy migration stat migration/multifd: Warn user when zerocopy not working

 qapi/migration.json   | 7 ++++++-
 migration/ram.h       | 2 ++
 io/channel-socket.c   | 8 +++++++-
 migration/migration.c | 2 ++
 migration/multifd.c   | 2 ++
 migration/ram.c       | 5 +++++
 monitor/hmp-cmds.c    | 4 ++++
 7 files changed, 28 insertions(+), 2 deletions(-)
Signature · 1 line
2.36.1
#1 lore raw

PATCH v3 1/3 QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
io/channel-socket.c +7 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
Signature · 1 line
2.36.1
Reviewed-by Acked-by on v3 1/3
#2 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 4 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Quoted 8 lines

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

And if this merges via migration maintainers' tree

Quoted 28 lines
io/channel-socket.c +7 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
Signature · 1 line
2.36.1

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Reviewed-by
#3 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 14 lines · Daniel P. Berrangé
Quoted 4 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Quoted 8 lines

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 5 lines

And if this merges via migration maintainers' tree

I will add this on the next PULLL request.

Thanks.

Quoted 31 lines
Quoted 28 lines
io/channel-socket.c +7 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
Signature · 1 line
2.36.1

With regards,
Daniel

#4 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

In reply to Leonardo Bras

Hi, Leo,

Quoted 18 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
io/channel-socket.c +0 -0
index 4466bb1cd4..698c086b70 100644

io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721

Quoted 7 lines
io/channel-socket.c +3 -1
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {

I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?

Quoted 15 lines
io/channel-socket.c +4 -0
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
#5 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Hello Peter,

Quoted 17 lines · Peter Xu

Hi, Leo,

Quoted 13 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 15 lines

io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721

Quoted 12 lines
io/channel-socket.c +3 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {

I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?

This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.

Without this, the function could return 1 if nothing was sent with zero-copy, and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.

Best regards,
Leo

Quoted 20 lines
Quoted 14 lines
io/channel-socket.c +4 -1
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
--
Quoted 2 lines

2.36.1

--
Peter Xu

#6 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 36 lines · Leonardo Bras Soares Passos

Hello Peter,

Quoted 17 lines · Peter Xu

Hi, Leo,

Quoted 13 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 15 lines

io/channel-socket.c static int qio_channel_socket_flush(QIOChannel *ioc, · line 721

Quoted 12 lines
io/channel-socket.c +3 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel *ioc,
     struct cmsghdr *cm;
     char control[CMSG_SPACE(sizeof(*serr))];
     int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {

I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?

Quoted 6 lines

This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.

Without this, the function could return 1 if nothing was sent with zero-copy, and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.

I know it won't happen in practise, but.. what if we call flush() without sending anything zero-copy-wise at all (so zero_copy_sent > 0, zero_copy_queued > 0, meanwhile zero_copy_sent == zero_copy_queued)? Then IIUC we'll return 1 even if we didn't do any fallback, or am I wrong?

Quoted 25 lines

Best regards,
Leo

Quoted 20 lines
Quoted 14 lines
io/channel-socket.c +4 -1
+        return 0;
+    }
 
     msg.msg_control = control;
     msg.msg_controllen = sizeof(control);
     memset(control, 0, sizeof(control));
 
+    ret = 1;
+
     while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
         received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
         if (received < 0) {
--
Quoted 2 lines

2.36.1

--
Peter Xu

Signature · 1 line
Peter Xu
#7 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 43 lines · Peter Xu
Quoted 42 lines · Leonardo Bras Soares Passos

Hello Peter,

Quoted 19 lines · Peter Xu

Hi, Leo,

Quoted 15 lines · Leonardo Bras

If flush is called when no buffer was sent with MSG_ZEROCOPY, it currently
returns 1. This return code should be used only when Linux fails to use MSG_ZEROCOPY on a lot of sendmsg().

Fix this by returning early from flush if no sendmsg(...,MSG_ZEROCOPY) was attempted.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy

flag & io_flush for CONFIG_LINUX")

---
 io/channel-socket.c | 8 +++++++-

 1 file changed, 7 insertions(+), 1 deletion(-)
Quoted 16 lines

io/channel-socket.c static int qio_channel_socket_flush(QIOChannel · line 722

Quoted 13 lines
io/channel-socket.c +3 -1
index 4466bb1cd4..698c086b70 100644
@@ -716,12 +716,18 @@ static int qio_channel_socket_flush(QIOChannel
 *ioc,
      struct cmsghdr *cm;
      char control[CMSG_SPACE(sizeof(*serr))];
      int received;
-    int ret = 1;
+    int ret;
+
+    if (!sioc->zero_copy_queued) {

I think I asked this in the downstream review but didn't get a response.. shouldn't this check be "queued == sent"?

This is just supposed to skip flush if nothing was queued for sending. queued == sent is tested bellow in the while part.

Quoted 8 lines
Quoted 4 lines

Without this, the function could return 1 if nothing was sent with zero- copy,
and it would be confusing, because the QIOChannel API says 1 should be returned only if all zero-copy sends fell back to copying.

I know it won't happen in practise, but.. what if we call flush() without sending anything zero-copy-wise at all (so zero_copy_sent > 0, zero_copy_queued > 0,

On a no-bug scenario:
- If we don't send anything zero-copy wise, zero_copy_queued will never get incremented.
- If we don't get inside the while loop in flush, zero_copy_sent will never be incremented.

But sure, suppose it does get incremented by bug / mistake:
- zero_copy_queued incremented, zero_copy_sent untouched :
On (queued == 0) it will go past the 'if', and get stuck 'forever' in the while
loop, since sent will 'never' get incremented.
On (queued == sent), same.
On (queued <= sent), same.

- zero_copy_queued untouched, zero_copy_sent incremented :
On 'if (queued == 0)' it will not go past the 'if'
On 'if (queued == sent)', will go past the 'if', but will exit on the 'while',
falsely returning 1.
On 'if (queued <= sent)', it will not go past the 'if'.

- zero_copy_queued incremented, zero_copy_sent incremented :
On 'if (queued == 0)', infinite loop as above.
On 'if (queued == sent)',
if sent < queued : infinite loop ,
if sent == queued : won't go past 'if' ,
if sent > queued : will go past the 'if', but will exit on the
'while', falsely returning 1.
On (queued <= sent), infinite loop if sent < queued, not past if otherwise

BTW, I consider it 'infinite loop', but it could also end up returning -1 on any error.

Quoted 2 lines

meanwhile zero_copy_sent == zero_copy_queued)?  Then
IIUC we'll return 1 even if we didn't do any fallback, or am I wrong?

Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.

Best regards,
Leo

Quoted 27 lines
Quoted 26 lines

Best regards,
Leo

Quoted 20 lines
Quoted 12 lines
io/channel-socket.c +4 -0
+        return 0;
+    }
 
      msg.msg_control = control;
      msg.msg_controllen = sizeof(control);
      memset(control, 0, sizeof(control));
 
+    ret = 1;
+
      while (sioc->zero_copy_sent < sioc->zero_copy_queued) {
          received = recvmsg(sioc->fd, &msg, MSG_ERRQUEUE);
Quoted 4 lines

         if (received < 0) {
--
2.36.1

--
Peter Xu

#8 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 2 lines · Leonardo Brás

Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.

I'm not sure I strictly follow here..

Imagine the case we do flush() twice without sending anything, then in the 1st flush we'll see queued>sent, we'll finish flush() until queued==sent. Then in the 2nd (continuous) flush() we'll see queued==sent immediately.

IIUC with the current patch we'll return 1 which I think is wrong because fallback didn't happen, and if with the change to "if (queued==sent) return 0" it'll fix it?

Signature · 1 line
Peter Xu
#9 lore raw

Re: [PATCH v3 1/3] QIOChannelSocket: Fix zero-copy flush returning code 1 when nothing sent

Quoted 7 lines · Peter Xu
Quoted 2 lines · Leonardo Brás

Having 'if(queued == sent)' will cause us to falsely return '1' in two buggy cases, while 'if queued == 0) will either skip early or go into 'infinite' loop.

I'm not sure I strictly follow here..

Sorry, I was thinking of a different scenario.

Quoted 7 lines

Imagine the case we do flush() twice without sending anything, then in the 1st flush we'll see queued>sent, we'll finish flush() until queued==sent. Then in the 2nd (continuous) flush() we'll see queued==sent immediately.

IIUC with the current patch we'll return 1 which I think is wrong because fallback didn't happen, and if with the change to "if (queued==sent) return 0" it'll fix it?

Yes, you are correct.
It's a possible scenario to have a flush happen just after another without any sending in between.

I will fix it as suggested.

Best regards,
Leo

Quoted 4 lines

--
Peter Xu

#10 lore raw

PATCH v3 2/3 Add dirty-sync-missed-zero-copy migration stat

In reply to Leonardo Bras
---
 qapi/migration.json   | 7 ++++++-
 migration/migration.c | 2 ++
 monitor/hmp-cmds.c    | 4 ++++
 3 files changed, 12 insertions(+), 1 deletion(-)
qapi/migration.json +6 -1
index 7102e474a6..fed08b9b88 100644
@@ -55,6 +55,10 @@
 # @postcopy-bytes: The number of bytes sent during the post-copy phase
 #                  (since 7.0).
 #
+# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could
+#                               not avoid copying zero pages.  This is between 0
+#                               and @dirty-sync-count * @multifd-channels.
+#                               (since 7.1)
 # Since: 0.14
 ##
 { 'struct': 'MigrationStats',
@@ -65,7 +69,8 @@
            'postcopy-requests' : 'int', 'page-size' : 'int',
            'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64',
            'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64',
-           'postcopy-bytes' : 'uint64' } }
+           'postcopy-bytes' : 'uint64',
+           'dirty-sync-missed-zero-copy' : 'uint64' } }
 
 ##
 # @XBZRLECacheStats:
migration/migration.c +2 -0
index 78f5057373..048f7f8bdb 100644
@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal_bytes = ram_counters.normal * page_size;
     info->ram->mbps = s->mbps;
     info->ram->dirty_sync_count = ram_counters.dirty_sync_count;
+    info->ram->dirty_sync_missed_zero_copy =
+            ram_counters.dirty_sync_missed_zero_copy;
     info->ram->postcopy_requests = ram_counters.postcopy_requests;
     info->ram->page_size = page_size;
     info->ram->multifd_bytes = ram_counters.multifd_bytes;
monitor/hmp-cmds.c +4 -0
index ca98df0495..5f3be9e405 100644
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
             monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
                            info->ram->postcopy_bytes >> 10);
         }
+        if (info->ram->dirty_sync_missed_zero_copy) {
+            monitor_printf(mon, "missed zero-copy on: %" PRIu64 " iterations\n",
+                           info->ram->dirty_sync_missed_zero_copy);
+        }
     }
 
     if (info->has_disk) {
Signature · 1 line
2.36.1
Reviewed-by on v3 2/3
#12 lore raw

Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat

In reply to Leonardo Bras
Quoted 6 lines · Leonardo Bras
---
 qapi/migration.json   | 7 ++++++-
 migration/migration.c | 2 ++
 monitor/hmp-cmds.c    | 4 ++++
 3 files changed, 12 insertions(+), 1 deletion(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#13 lore raw

Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat

In reply to Leonardo Bras
Quoted 12 lines · Leonardo Bras
---
 qapi/migration.json   | 7 ++++++-
 migration/migration.c | 2 ++
 monitor/hmp-cmds.c    | 4 ++++
 3 files changed, 12 insertions(+), 1 deletion(-)
qapi/migration.json +0 -0
index 7102e474a6..fed08b9b88 100644

qapi/migration.json line 59

Quoted 5 lines
qapi/migration.json +2 -0
@@ -55,6 +55,10 @@
 # @postcopy-bytes: The number of bytes sent during the post-copy phase
 #                  (since 7.0).
 #
+# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could
+#                               not avoid copying zero pages.  This is between 0

Avoid copying zero pages? Isn't this for counting MSG_ZEROCOPY fallbacks?

qapi/migration.json line 60

Quoted 1 line
qapi/migration.json +1 -0
+#                               and @dirty-sync-count * @multifd-channels.

I'd not name it as "dirty-sync-*" because fundamentally the accounting is not doing like that (more in latter patch). I also think we should squash patch 2/3 as patch 3 only started to provide meaningful values.

Quoted 32 lines
qapi/migration.json +3 -1
+#                               (since 7.1)
 # Since: 0.14
 ##
 { 'struct': 'MigrationStats',
@@ -65,7 +69,8 @@
            'postcopy-requests' : 'int', 'page-size' : 'int',
            'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64',
            'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64',
-           'postcopy-bytes' : 'uint64' } }
+           'postcopy-bytes' : 'uint64',
+           'dirty-sync-missed-zero-copy' : 'uint64' } }
 
 ##
 # @XBZRLECacheStats:
migration/migration.c +2 -0
index 78f5057373..048f7f8bdb 100644
@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal_bytes = ram_counters.normal * page_size;
     info->ram->mbps = s->mbps;
     info->ram->dirty_sync_count = ram_counters.dirty_sync_count;
+    info->ram->dirty_sync_missed_zero_copy =
+            ram_counters.dirty_sync_missed_zero_copy;
     info->ram->postcopy_requests = ram_counters.postcopy_requests;
     info->ram->page_size = page_size;
     info->ram->multifd_bytes = ram_counters.multifd_bytes;
monitor/hmp-cmds.c +0 -0
index ca98df0495..5f3be9e405 100644

monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312

Quoted 6 lines
monitor/hmp-cmds.c +3 -0
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
             monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
                            info->ram->postcopy_bytes >> 10);
         }
+        if (info->ram->dirty_sync_missed_zero_copy) {
+            monitor_printf(mon, "missed zero-copy on: %" PRIu64 " iterations\n",
+                           info->ram->dirty_sync_missed_zero_copy);

I suggest we don't call it "iterations" because it's not the generic mean of iterations.

Quoted 7 lines
monitor/hmp-cmds.c +1 -0
+        }
     }
 
     if (info->has_disk) {
Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
#14 lore raw

Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat

Hello Peter,

Quoted 9 lines · Peter Xu
Quoted 7 lines · Leonardo Bras
---
 qapi/migration.json   | 7 ++++++-
 migration/migration.c | 2 ++
 monitor/hmp-cmds.c    | 4 ++++
 3 files changed, 12 insertions(+), 1 deletion(-)
Quoted 12 lines

qapi/migration.json line 59

Quoted 10 lines
qapi/migration.json +2 -0
index 7102e474a6..fed08b9b88 100644
@@ -55,6 +55,10 @@
 # @postcopy-bytes: The number of bytes sent during the post-copy phase
 #                  (since 7.0).
 #
+# @dirty-sync-missed-zero-copy: Number of times dirty RAM synchronization could
+#                               not avoid copying zero pages.  This is between 0

Avoid copying zero pages? Isn't this for counting MSG_ZEROCOPY fallbacks?

Yes, sorry, I think I got confused at some point between some cuts & pastes. It should be "not avoid copying dirty pages." I will fix that on a V4.

Quoted 5 lines

qapi/migration.json line 60

Quoted 2 lines
qapi/migration.json +1 -0
+#                               and @dirty-sync-count * @multifd-channels.

I'd not name it as "dirty-sync-*" because fundamentally the accounting is not doing like that (more in latter patch).

Ok, I will take a look & answer there.

Quoted 2 lines

I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.

IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review. So I thought it would be helpful now.

Quoted 34 lines
Quoted 34 lines
qapi/migration.json +3 -1
+#                               (since 7.1)
 # Since: 0.14
 ##
 { 'struct': 'MigrationStats',
@@ -65,7 +69,8 @@
            'postcopy-requests' : 'int', 'page-size' : 'int',
            'multifd-bytes' : 'uint64', 'pages-per-second' : 'uint64',
            'precopy-bytes' : 'uint64', 'downtime-bytes' : 'uint64',
-           'postcopy-bytes' : 'uint64' } }
+           'postcopy-bytes' : 'uint64',
+           'dirty-sync-missed-zero-copy' : 'uint64' } }
 
 ##
 # @XBZRLECacheStats:
migration/migration.c +2 -0
index 78f5057373..048f7f8bdb 100644
@@ -1027,6 +1027,8 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     info->ram->normal_bytes = ram_counters.normal * page_size;
     info->ram->mbps = s->mbps;
     info->ram->dirty_sync_count = ram_counters.dirty_sync_count;
+    info->ram->dirty_sync_missed_zero_copy =
+            ram_counters.dirty_sync_missed_zero_copy;
     info->ram->postcopy_requests = ram_counters.postcopy_requests;
     info->ram->page_size = page_size;
     info->ram->multifd_bytes = ram_counters.multifd_bytes;
monitor/hmp-cmds.c +0 -0
index ca98df0495..5f3be9e405 100644
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
             monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
Quoted 8 lines

monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312

Quoted 5 lines
monitor/hmp-cmds.c +3 -0
                            info->ram->postcopy_bytes >> 10);
         }
+        if (info->ram->dirty_sync_missed_zero_copy) {
+            monitor_printf(mon, "missed zero-copy on: %" PRIu64 " iterations\n",
+                           info->ram->dirty_sync_missed_zero_copy);

I suggest we don't call it "iterations" because it's not the generic mean of iterations.

Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?

Best regards,
Leo

Quoted 12 lines
Quoted 8 lines
monitor/hmp-cmds.c +1 -1
+        }
     }
 
     if (info->has_disk) {
--

2.36.1

--
Peter Xu

#15 lore raw

Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat

Quoted 6 lines · Leonardo Bras Soares Passos
Quoted 2 lines

I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.

IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review. So I thought it would be helpful now.

Ah, that's fine then.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
monitor/hmp-cmds.c +0 -0
index ca98df0495..5f3be9e405 100644
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict)
             monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
Quoted 8 lines

monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict *qdict) · line 312

Quoted 5 lines
monitor/hmp-cmds.c +3 -0
                            info->ram->postcopy_bytes >> 10);
         }
+        if (info->ram->dirty_sync_missed_zero_copy) {
+            monitor_printf(mon, "missed zero-copy on: %" PRIu64 " iterations\n",
+                           info->ram->dirty_sync_missed_zero_copy);

I suggest we don't call it "iterations" because it's not the generic mean of iterations.

Quoted 2 lines

Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?

"Zero-copy-send fallbacks happened: xxx times\n"?

Signature · 1 line
Peter Xu
#16 lore raw

Re: [PATCH v3 2/3] Add dirty-sync-missed-zero-copy migration stat

Quoted 28 lines · Peter Xu
Quoted 7 lines · Leonardo Bras Soares Passos
Quoted 2 lines

I also think we should squash
patch 2/3 as patch 3 only started to provide meaningful values.

IIRC Previously in zero-copy-send implementation, I was asked to keep the property/capability in a separated patch in order to make it easier to review.
So I thought it would be helpful now.

Ah, that's fine then.

Quoted 17 lines
Quoted 16 lines

monitor/hmp-cmds.c void hmp_info_migrate(Monitor *mon, const QDict · line 314

Quoted 13 lines
monitor/hmp-cmds.c +3 -0
index ca98df0495..5f3be9e405 100644
@@ -307,6 +307,10 @@ void hmp_info_migrate(Monitor *mon, const QDict
 *qdict)
              monitor_printf(mon, "postcopy ram: %" PRIu64 " kbytes\n",
                             info->ram->postcopy_bytes >> 10);
          }
+        if (info->ram->dirty_sync_missed_zero_copy) {
+            monitor_printf(mon, "missed zero-copy on: %" PRIu64 "
 iterations\n",
+                           info->ram->dirty_sync_missed_zero_copy);

I suggest we don't call it "iterations" because it's not the generic mean of iterations.

Quoted 4 lines
Quoted 2 lines

Yeah, I thought that too, but I could not think on anything better. What do you suggest instead?

"Zero-copy-send fallbacks happened: xxx times\n"?

Oh, yeah, that will work.
I was thinking on keeping the pattern and ended up thinking what was the correct unit. But this is much simpler and work better.

Best regards,
Leo

Quoted 1 line
#17 lore raw

PATCH v3 3/3 migration/multifd: Warn user when zerocopy not working

In reply to Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.

---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++
 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
index ded0a3a086..d3c7eb96f5 100644
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);
 int ram_write_tracking_start(void);
 void ram_write_tracking_stop(void);
 
+void dirty_sync_missed_zero_copy(void);
+
 #endif
migration/multifd.c +2 -0
index 684c014c86..3909b34967 100644
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)
             if (ret < 0) {
                 error_report_err(err);
                 return -1;
+            } else if (ret == 1) {
+                dirty_sync_missed_zero_copy();
             }
         }
     }
migration/ram.c +5 -0
index 01f9cc1d72..db948c4787 100644
@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes)
     ram_counters.transferred += bytes;
 }
 
+void dirty_sync_missed_zero_copy(void)
+{
+    ram_counters.dirty_sync_missed_zero_copy++;
+}
+
 /* used by the search for pages to send */
 struct PageSearchStatus {
     /* Current block being searched */
Signature · 1 line
2.36.1
Reviewed-by on v3 3/3
#18 lore raw

Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working

Quoted 10 lines · Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.

Quoted 6 lines
---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++
 3 files changed, 9 insertions(+)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#19 lore raw

Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working

In reply to Leonardo Bras
Quoted 33 lines · Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.

---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++
 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
index ded0a3a086..d3c7eb96f5 100644
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);
 int ram_write_tracking_start(void);
 void ram_write_tracking_stop(void);
 
+void dirty_sync_missed_zero_copy(void);
+
 #endif
migration/multifd.c +0 -0
index 684c014c86..3909b34967 100644

migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631

Quoted 8 lines
migration/multifd.c +2 -0
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)
             if (ret < 0) {
                 error_report_err(err);
                 return -1;
+            } else if (ret == 1) {
+                dirty_sync_missed_zero_copy();
             }
         }
     }

I know that Juan is working on some patch to only do multifd_send_sync_main() for each dirty sync, but that's not landed, right?

Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?

The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.

Thanks,

Quoted 19 lines
migration/ram.c +5 -0
index 01f9cc1d72..db948c4787 100644
@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes)
     ram_counters.transferred += bytes;
 }
 
+void dirty_sync_missed_zero_copy(void)
+{
+    ram_counters.dirty_sync_missed_zero_copy++;
+}
+
 /* used by the search for pages to send */
 struct PageSearchStatus {
     /* Current block being searched */
Signature · 1 line
2.36.1
Signature · 1 line
Peter Xu
#20 lore raw

Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working

Hello Peter,

Quoted 38 lines · Peter Xu
Quoted 36 lines · Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.

---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++
 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
index ded0a3a086..d3c7eb96f5 100644
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);
 int ram_write_tracking_start(void);
 void ram_write_tracking_stop(void);
 
+void dirty_sync_missed_zero_copy(void);
+
 #endif
migration/multifd.c +0 -0
index 684c014c86..3909b34967 100644
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)
             if (ret < 0) {
                 error_report_err(err);
                 return -1;
Quoted 8 lines

migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631

Quoted 5 lines
migration/multifd.c +2 -0
+            } else if (ret == 1) {
+                dirty_sync_missed_zero_copy();
             }
         }
     }

I know that Juan is working on some patch to only do multifd_send_sync_main() for each dirty sync, but that's not landed, right?

That's correct, but I am hoping it should land before the release, so the numbers will match.

Quoted 3 lines

Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?

It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.

Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.

Quoted 3 lines

The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.

Ok, Warning sounds misleading here.
What do you think about 'report' instead?

Best regards,
Leo

Quoted 26 lines

Thanks,

Quoted 17 lines
migration/ram.c +5 -1
index 01f9cc1d72..db948c4787 100644
@@ -407,6 +407,11 @@ static void ram_transferred_add(uint64_t bytes)
     ram_counters.transferred += bytes;
 }
 
+void dirty_sync_missed_zero_copy(void)
+{
+    ram_counters.dirty_sync_missed_zero_copy++;
+}
+
 /* used by the search for pages to send */
 struct PageSearchStatus {
     /* Current block being searched */
--
Quoted 2 lines

2.36.1

--
Peter Xu

#21 lore raw

Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working

Quoted 58 lines · Leonardo Bras Soares Passos

Hello Peter,

Quoted 38 lines · Peter Xu
Quoted 36 lines · Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know about it.

---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++
 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
index ded0a3a086..d3c7eb96f5 100644
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);
 int ram_write_tracking_start(void);
 void ram_write_tracking_stop(void);
 
+void dirty_sync_missed_zero_copy(void);
+
 #endif
migration/multifd.c +0 -0
index 684c014c86..3909b34967 100644
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)
             if (ret < 0) {
                 error_report_err(err);
                 return -1;
Quoted 8 lines

migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631

Quoted 5 lines
migration/multifd.c +2 -0
+            } else if (ret == 1) {
+                dirty_sync_missed_zero_copy();
             }
         }
     }

I know that Juan is working on some patch to only do multifd_send_sync_main() for each dirty sync, but that's not landed, right?

That's correct, but I am hoping it should land before the release, so the numbers will match.

Quoted 3 lines

Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)? Something like "zero-copy-send-fallbacks"?

Quoted 8 lines

It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.

Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.

Ah okay.

But then I suggest we make sure it lands only after Juan's.. or it won't really match. Also when Juan's patch ready, we'd also double check it will be exactly called once per iteration, or we can get confusing numbers. I assume Juan will take care of that then.

Quoted 7 lines
Quoted 4 lines

The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.

Ok, Warning sounds misleading here.
What do you think about 'report' instead?

Looks good. Thanks,

Signature · 1 line
Peter Xu
#22 lore raw

Re: [PATCH v3 3/3] migration/multifd: Warn user when zerocopy not working

Quoted 81 lines · Peter Xu
Quoted 62 lines · Leonardo Bras Soares Passos

Hello Peter,

Quoted 42 lines · Peter Xu
Quoted 40 lines · Leonardo Bras

Some errors, like the lack of Scatter-Gather support by the network interface(NETIF_F_SG) may cause sendmsg(...,MSG_ZEROCOPY) to fail on using
zero-copy, which causes it to fall back to the default copying mechanism.

After each full dirty-bitmap scan there should be a zero-copy flush happening, which checks for errors each of the previous calls to sendmsg(...,MSG_ZEROCOPY). If all of them failed to use zero-copy, then increment dirty_sync_missed_zero_copy migration stat to let the user know
about it.

---
 migration/ram.h     | 2 ++
 migration/multifd.c | 2 ++
 migration/ram.c     | 5 +++++

 3 files changed, 9 insertions(+)
migration/ram.h +2 -0
index ded0a3a086..d3c7eb96f5 100644
@@ -87,4 +87,6 @@ void ram_write_tracking_prepare(void);
  int ram_write_tracking_start(void);
  void ram_write_tracking_stop(void);
 
+void dirty_sync_missed_zero_copy(void);
+
  #endif
migration/multifd.c +1 -0
index 684c014c86..3909b34967 100644
@@ -624,6 +624,8 @@ int multifd_send_sync_main(QEMUFile *f)
              if (ret < 0) {
                  error_report_err(err);
                  return -1;
+            } else if (ret == 1) {
Quoted 8 lines

migration/multifd.c int multifd_send_sync_main(QEMUFile *f) · line 631

Quoted 4 lines
migration/multifd.c +1 -0
+                dirty_sync_missed_zero_copy();
              }
          }
      }

I know that Juan is working on some patch to only do multifd_send_sync_main() for each dirty sync, but that's not landed, right?

That's correct, but I am hoping it should land before the release, so the numbers will match.

Quoted 3 lines

Can we name it without "dirty-sync" at all (so it'll work before/after Juan's patch will be applied)?  Something like "zero-copy-send-fallbacks"?

Quoted 8 lines

It initially was something like that, but on the v2 thread there was some discussion on
the topic, and it was suggested the number would not mean much to the user, unless
it was connected to something else.

Markus suggested the connection to @dirty-sync-count right in the name, and Daniel suggested the above name, which sounds fine to me.

Ah okay.

But then I suggest we make sure it lands only after Juan's.. or it won't really match.  Also when Juan's patch ready, we'd also double check it will be exactly called once per iteration, or we can get confusing numbers.  I assume Juan will take care of that then.

Quoted 2 lines
Quoted 2 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

The other thing is the subject may need to be touched up as right now with the field we don't warn the user anymore on zero-copy-send fallbacks.

Ok, Warning sounds misleading here.
What do you think about 'report' instead?

Looks good.  Thanks,

Thank you for reviewing, Peter!

Leo

Quoted 1 line