Skip to contentSkip to conversation

QIOChannelSocket: Fix zero-copy send so socket flush works

PATCH In review Leonardo Bras 3 participants 7 messages Jun 8, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 QIOChannelSocket: Fix zero-copy send so socket flush works

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
io/channel-socket.c +8 -3
index dc9c165de1..ca4cae930f 100644
@@ -554,6 +554,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
     size_t fdsize = sizeof(int) * nfds;
     struct cmsghdr *cmsg;
     int sflags = 0;
+    bool zero_copy_enabled = false;
 
     memset(control, 0, CMSG_SPACE(sizeof(int) * SOCKET_MAX_FDS));
 
@@ -581,6 +582,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
 #ifdef QEMU_MSG_ZEROCOPY
     if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
         sflags = MSG_ZEROCOPY;
+        zero_copy_enabled = true;
     }
 #endif
 
@@ -592,21 +594,24 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
             return QIO_CHANNEL_ERR_BLOCK;
         case EINTR:
             goto retry;
-#ifdef QEMU_MSG_ZEROCOPY
         case ENOBUFS:
-            if (sflags & MSG_ZEROCOPY) {
+            if (zero_copy_enabled) {
                 error_setg_errno(errp, errno,
                                  "Process can't lock enough memory for using MSG_ZEROCOPY");
                 return -1;
             }
             break;
-#endif
         }
 
         error_setg_errno(errp, errno,
                          "Unable to write to socket");
         return -1;
     }
+
+    if (zero_copy_enabled) {
+        sioc->zero_copy_queued++;
+    }
+
     return ret;
 }
 #else /* WIN32 */
Signature · 1 line
2.36.1
#1 lore raw
Quoted 36 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
io/channel-socket.c +2 -0
index dc9c165de1..ca4cae930f 100644
@@ -554,6 +554,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
     size_t fdsize = sizeof(int) * nfds;
     struct cmsghdr *cmsg;
     int sflags = 0;
+    bool zero_copy_enabled = false;
 
     memset(control, 0, CMSG_SPACE(sizeof(int) * SOCKET_MAX_FDS));
 
@@ -581,6 +582,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
 #ifdef QEMU_MSG_ZEROCOPY
     if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
         sflags = MSG_ZEROCOPY;
+        zero_copy_enabled = true;
     }
 #endif
 

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 595

Quoted 4 lines
io/channel-socket.c +0 -1
@@ -592,21 +594,24 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
             return QIO_CHANNEL_ERR_BLOCK;
         case EINTR:
             goto retry;
-#ifdef QEMU_MSG_ZEROCOPY

Removing this ifdef appears incidental to the change. If this is redundant just remove it in its own patch.

Quoted 7 lines
io/channel-socket.c +1 -1
         case ENOBUFS:
-            if (sflags & MSG_ZEROCOPY) {
+            if (zero_copy_enabled) {
                 error_setg_errno(errp, errno,
                                  "Process can't lock enough memory for using MSG_ZEROCOPY");
                 return -1;
             }

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 611

Quoted 10 lines
io/channel-socket.c +2 -1
             break;
-#endif
         }
 
         error_setg_errno(errp, errno,
                          "Unable to write to socket");
         return -1;
     }
+
+    if (zero_copy_enabled) {

What's wrong with

   if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
        sioc->zero_copy_queued++;
    }

Introducing another local variable doesn't really add value IMHO.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#2 lore raw
Quoted 65 lines · Daniel P. Berrangé
Quoted 36 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
io/channel-socket.c +2 -0
index dc9c165de1..ca4cae930f 100644
@@ -554,6 +554,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
     size_t fdsize = sizeof(int) * nfds;
     struct cmsghdr *cmsg;
     int sflags = 0;
+    bool zero_copy_enabled = false;
 
     memset(control, 0, CMSG_SPACE(sizeof(int) * SOCKET_MAX_FDS));
 
@@ -581,6 +582,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
 #ifdef QEMU_MSG_ZEROCOPY
     if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
         sflags = MSG_ZEROCOPY;
+        zero_copy_enabled = true;
     }
 #endif
 

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 595

Quoted 4 lines
io/channel-socket.c +0 -1
@@ -592,21 +594,24 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
             return QIO_CHANNEL_ERR_BLOCK;
         case EINTR:
             goto retry;
-#ifdef QEMU_MSG_ZEROCOPY

Removing this ifdef appears incidental to the change. If this is redundant just remove it in its own patch.

Quoted 7 lines
io/channel-socket.c +1 -1
         case ENOBUFS:
-            if (sflags & MSG_ZEROCOPY) {
+            if (zero_copy_enabled) {
                 error_setg_errno(errp, errno,
                                  "Process can't lock enough memory for using MSG_ZEROCOPY");
                 return -1;
             }

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 611

Quoted 10 lines
io/channel-socket.c +2 -1
             break;
-#endif
         }
 
         error_setg_errno(errp, errno,
                          "Unable to write to socket");
         return -1;
     }
+
+    if (zero_copy_enabled) {

What's wrong with

Quoted 6 lines
   if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
        sioc->zero_copy_queued++;
    }

Introducing another local variable doesn't really add value IMHO.

One benefit of having that variable is we setup zero_copy_enabled once in the #ifdef and the rest code can avoid wrapping with the macro. From that pov the patch looks okay to me. Thanks,

Signature · 1 line
Peter Xu
#3 lore raw
In reply to Daniel P. Berrangé

Hello Daniel,

Quoted 37 lines · Daniel P. Berrangé
Quoted 35 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")
---
 io/channel-socket.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
io/channel-socket.c +2 -0
index dc9c165de1..ca4cae930f 100644
@@ -554,6 +554,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
     size_t fdsize = sizeof(int) * nfds;
     struct cmsghdr *cmsg;
     int sflags = 0;
+    bool zero_copy_enabled = false;
 
     memset(control, 0, CMSG_SPACE(sizeof(int) * SOCKET_MAX_FDS));
 
@@ -581,6 +582,7 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
 #ifdef QEMU_MSG_ZEROCOPY
     if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
         sflags = MSG_ZEROCOPY;
+        zero_copy_enabled = true;
     }
 #endif
 
Quoted 8 lines

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 595

Quoted 5 lines
io/channel-socket.c +0 -1
@@ -592,21 +594,24 @@ static ssize_t qio_channel_socket_writev(QIOChannel *ioc,
             return QIO_CHANNEL_ERR_BLOCK;
         case EINTR:
             goto retry;
-#ifdef QEMU_MSG_ZEROCOPY

Removing this ifdef appears incidental to the change. If this is redundant just remove it in its own patch.

The idea is to reduce the amount of #ifdefs as Peter suggested, because adding another ifdef here
would introduce extra noise. But sure, I see no problem adding this change as a previous patch.

Quoted 21 lines
Quoted 8 lines
io/channel-socket.c +1 -1
         case ENOBUFS:
-            if (sflags & MSG_ZEROCOPY) {
+            if (zero_copy_enabled) {
                 error_setg_errno(errp, errno,
                                  "Process can't lock enough memory for using MSG_ZEROCOPY");
                 return -1;
             }

io/channel-socket.c static ssize_t qio_channel_socket_writev(QIOChannel *ioc, · line 611

Quoted 10 lines
io/channel-socket.c +2 -1
             break;
-#endif
         }
 
         error_setg_errno(errp, errno,
                          "Unable to write to socket");
         return -1;
     }
+
+    if (zero_copy_enabled) {

What's wrong with

Quoted 3 lines
   if (flags & QIO_CHANNEL_WRITE_FLAG_ZERO_COPY) {
        sioc->zero_copy_queued++;
    }

There is nothing wrong with it, but using zero_copy_enabled as presented here will
compile-out this 'if()' block if the user does not support MSG_ZEROCOPY.

Best regards,
Leo

Quoted 11 lines

Introducing another local variable doesn't really add value IMHO.

With regards,
Daniel
--

|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
In reply to Leonardo Bras
Quoted 11 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")

Copy Dave/Juan; Leo please remember to do so in the next posts, or no one will be picking this up. :)

Signature · 1 line
Peter Xu
#5 lore raw
Quoted 7 lines · Peter Xu
Quoted 6 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

Quoted 8 lines
Quoted 5 lines

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")

Copy Dave/Juan; Leo please remember to do so in the next posts, or no one will be picking this up. :)

My fault, it's an io channel patch. But still good to copy relevant developers..

Signature · 1 line
Peter Xu
#6 lore raw
Quoted 9 lines · Peter Xu
Quoted 7 lines · Peter Xu
Quoted 6 lines · Leonardo Bras

Somewhere between v6 and v7 the of the zero-copy-send patchset a crucial part of the flushing mechanism got missing: incrementing zero_copy_queued.

Without that, the flushing interface becomes a no-op, and there is no garantee the buffer is really sent.

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

This can go as bad as causing a corruption in RAM during migration.

Fixes 2bc58ffc2926 ("QIOChannelSocket: Implement io_writev zero copy flag & io_flush for CONFIG_LINUX")

Copy Dave/Juan; Leo please remember to do so in the next posts, or no one will be picking this up. :)

Thanks for letting me know.

Quoted 2 lines

My fault, it's an io channel patch. But still good to copy relevant developers..

Np. Sure, I will keep in mind to add them in the next version.

Oh, BTW: I will be sending a v2 shortly.

Quoted 4 lines

--
Peter Xu