Skip to contentSkip to conversation

nbd: enable keepalive

PATCH Superseded Vladimir Sementsov-Ogievskiy 4 participants 16 messages Jun 5, 2019 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover nbd: enable keepalive

Hi all!

Here is a suggestion to enable keepalive option to track server availablity.

Vladimir Sementsov-Ogievskiy (2):
io/channel: add qio_channel_set_keepalive
nbd-client: enable TCP keepalive

 include/io/channel.h | 13 +++++++++++++
 block/nbd-client.c   |  1 +
 io/channel-socket.c  | 19 +++++++++++++++++++
 io/channel.c         | 14 ++++++++++++++
 4 files changed, 47 insertions(+)
Signature · 1 line
2.18.0
#1 lore raw

PATCH 1/2 io/channel: add qio_channel_set_keepalive

---
 include/io/channel.h | 13 +++++++++++++
 io/channel-socket.c  | 19 +++++++++++++++++++
 io/channel.c         | 14 ++++++++++++++
 3 files changed, 46 insertions(+)
include/io/channel.h +13 -0
index 59460cb1ec..34d871a414 100644
@@ -124,6 +124,9 @@ struct QIOChannelClass {
     int (*io_set_blocking)(QIOChannel *ioc,
                            bool enabled,
                            Error **errp);
+    int (*io_set_keepalive)(QIOChannel *ioc,
+                            bool enabled,
+                            Error **errp);
 
     /* Optional callbacks */
     int (*io_shutdown)(QIOChannel *ioc,
@@ -490,6 +493,16 @@ int qio_channel_set_blocking(QIOChannel *ioc,
                              bool enabled,
                              Error **errp);
 
+/*
+ * qio_channel_set_keepalive:
+ * @ioc: the channel object
+ * @enabled: the keepalive flag state
+ * @errp: pointer to a NULL-initialized error object
+ */
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp);
+
 /**
  * qio_channel_close:
  * @ioc: the channel object
io/channel-socket.c +19 -0
index bc5f80e780..5c1ea08660 100644
@@ -656,6 +656,24 @@ qio_channel_socket_set_blocking(QIOChannel *ioc,
 }
 
 
+static int
+qio_channel_socket_set_keepalive(QIOChannel *ioc,
+                                 bool enabled,
+                                 Error **errp)
+{
+    QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc);
+    int val = enabled;
+    int ret = qemu_setsockopt(sioc->fd, SOL_SOCKET, SO_KEEPALIVE,
+                              &val, sizeof(val));
+
+    if (ret < 0) {
+        error_setg_errno(errp, errno, "Unable to set KEEPALIVE");
+    }
+
+    return ret;
+}
+
+
 static void
 qio_channel_socket_set_delay(QIOChannel *ioc,
                              bool enabled)
@@ -762,6 +780,7 @@ static void qio_channel_socket_class_init(ObjectClass *klass,
     ioc_klass->io_writev = qio_channel_socket_writev;
     ioc_klass->io_readv = qio_channel_socket_readv;
     ioc_klass->io_set_blocking = qio_channel_socket_set_blocking;
+    ioc_klass->io_set_keepalive = qio_channel_socket_set_keepalive;
     ioc_klass->io_close = qio_channel_socket_close;
     ioc_klass->io_shutdown = qio_channel_socket_shutdown;
     ioc_klass->io_set_cork = qio_channel_socket_set_cork;
io/channel.c +14 -0
index 2a26c2a2c0..0f0b2b7b65 100644
@@ -265,6 +265,20 @@ int qio_channel_set_blocking(QIOChannel *ioc,
     return klass->io_set_blocking(ioc, enabled, errp);
 }
 
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp)
+{
+    QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc);
+
+    if (!klass->io_set_keepalive) {
+        error_setg(errp, "KEEPALIVE is not supported by IO channel");
+        return -ENOTSUP;
+    }
+
+    return klass->io_set_keepalive(ioc, enabled, errp);
+}
+
 
 int qio_channel_close(QIOChannel *ioc,
                       Error **errp)
Signature · 1 line
2.18.0
Reviewed-by on v1 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH 1/2] io/channel: add qio_channel_set_keepalive

Quoted 6 lines · Vladimir Sementsov-Ogievskiy
---
 include/io/channel.h | 13 +++++++++++++
 io/channel-socket.c  | 19 +++++++++++++++++++
 io/channel.c         | 14 ++++++++++++++
 3 files changed, 46 insertions(+)

Dan, if you'd like, I can take this through my NBD tree.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

Re: [Qemu-devel] [PATCH 1/2] io/channel: add qio_channel_set_keepalive

Quoted 11 lines · Eric Blake
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
---
 include/io/channel.h | 13 +++++++++++++
 io/channel-socket.c  | 19 +++++++++++++++++++
 io/channel.c         | 14 ++++++++++++++
 3 files changed, 46 insertions(+)

Dan, if you'd like, I can take this through my NBD tree.

There's a couple of small tweaks needed, but if you apply those then consider it to have my Acked-by: Daniel P. Berrangé <[email protected]> for you to merge directly.

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw

Re: [Qemu-devel] [PATCH 1/2] io/channel: add qio_channel_set_keepalive

In reply to Vladimir Sementsov-Ogievskiy
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
---
 include/io/channel.h | 13 +++++++++++++
 io/channel-socket.c  | 19 +++++++++++++++++++
 io/channel.c         | 14 ++++++++++++++
 3 files changed, 46 insertions(+)
include/io/channel.h +3 -0
index 59460cb1ec..34d871a414 100644
@@ -124,6 +124,9 @@ struct QIOChannelClass {
     int (*io_set_blocking)(QIOChannel *ioc,
                            bool enabled,
                            Error **errp);
+    int (*io_set_keepalive)(QIOChannel *ioc,
+                            bool enabled,
+                            Error **errp);
 
     /* Optional callbacks */
     int (*io_shutdown)(QIOChannel *ioc,

include/io/channel.h int qio_channel_set_blocking(QIOChannel *ioc, · line 501

Quoted 9 lines
include/io/channel.h +6 -0
@@ -490,6 +493,16 @@ int qio_channel_set_blocking(QIOChannel *ioc,
                              bool enabled,
                              Error **errp);
 
+/*
+ * qio_channel_set_keepalive:
+ * @ioc: the channel object
+ * @enabled: the keepalive flag state
+ * @errp: pointer to a NULL-initialized error object
+ */

Missing docs for the return value. SHould be

"Returns 0 on success, -1 on error."

note we do *not* return "-errno" values in QIOChannel APIs

Quoted 15 lines
include/io/channel.h +4 -0
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp);
+
 
io/channel-socket.c +2 -0
index bc5f80e780..5c1ea08660 100644
@@ -656,6 +656,24 @@ qio_channel_socket_set_blocking(QIOChannel *ioc,
 }
 
 
+static int
+qio_channel_socket_set_keepalive(QIOChannel *ioc,

io/channel-socket.c qio_channel_socket_set_blocking(QIOChannel *ioc, · line 670

Quoted 10 lines
io/channel-socket.c +10 -0
+                                 bool enabled,
+                                 Error **errp)
+{
+    QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc);
+    int val = enabled;
+    int ret = qemu_setsockopt(sioc->fd, SOL_SOCKET, SO_KEEPALIVE,
+                              &val, sizeof(val));
+
+    if (ret < 0) {
+        error_setg_errno(errp, errno, "Unable to set KEEPALIVE");

Add 'return -1' there to be explicit, avoiding need to read up on whether qemu_setsockopt returns -1 or -errno. (It returns -1)

Quoted 24 lines
io/channel-socket.c +7 -0
+    }
+
+    return ret;
+}
+
+
 static void
 qio_channel_socket_set_delay(QIOChannel *ioc,
                              bool enabled)
@@ -762,6 +780,7 @@ static void qio_channel_socket_class_init(ObjectClass *klass,
     ioc_klass->io_writev = qio_channel_socket_writev;
     ioc_klass->io_readv = qio_channel_socket_readv;
     ioc_klass->io_set_blocking = qio_channel_socket_set_blocking;
+    ioc_klass->io_set_keepalive = qio_channel_socket_set_keepalive;
     ioc_klass->io_close = qio_channel_socket_close;
     ioc_klass->io_shutdown = qio_channel_socket_shutdown;
     ioc_klass->io_set_cork = qio_channel_socket_set_cork;
io/channel.c +0 -0
index 2a26c2a2c0..0f0b2b7b65 100644
@@ -265,6 +265,20 @@ int qio_channel_set_blocking(QIOChannel *ioc,
     return klass->io_set_blocking(ioc, enabled, errp);
 }

io/channel.c int qio_channel_set_blocking(QIOChannel *ioc, · line 276

Quoted 10 lines
io/channel.c +9 -0
 
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp)
+{
+    QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc);
+
+    if (!klass->io_set_keepalive) {
+        error_setg(errp, "KEEPALIVE is not supported by IO channel");
+        return -ENOTSUP;

return -1;

io/channel.c int qio_channel_set_blocking(QIOChannel *ioc, · line 284

Quoted 8 lines
io/channel.c +5 -0
+    }
+
+    return klass->io_set_keepalive(ioc, enabled, errp);
+}
+
 
 int qio_channel_close(QIOChannel *ioc,
                       Error **errp)

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw

Re: [Qemu-devel] [PATCH 1/2] io/channel: add qio_channel_set_keepalive

Quoted 111 lines · Daniel P. Berrangé
Quoted 22 lines · Vladimir Sementsov-Ogievskiy
---
  include/io/channel.h | 13 +++++++++++++
  io/channel-socket.c  | 19 +++++++++++++++++++
  io/channel.c         | 14 ++++++++++++++
  3 files changed, 46 insertions(+)
include/io/channel.h +3 -0
index 59460cb1ec..34d871a414 100644
@@ -124,6 +124,9 @@ struct QIOChannelClass {
      int (*io_set_blocking)(QIOChannel *ioc,
                             bool enabled,
                             Error **errp);
+    int (*io_set_keepalive)(QIOChannel *ioc,
+                            bool enabled,
+                            Error **errp);
 
      /* Optional callbacks */
      int (*io_shutdown)(QIOChannel *ioc,

include/io/channel.h int qio_channel_set_blocking(QIOChannel *ioc, · line 501

Quoted 9 lines
include/io/channel.h +6 -0
@@ -490,6 +493,16 @@ int qio_channel_set_blocking(QIOChannel *ioc,
                               bool enabled,
                               Error **errp);
 
+/*
+ * qio_channel_set_keepalive:
+ * @ioc: the channel object
+ * @enabled: the keepalive flag state
+ * @errp: pointer to a NULL-initialized error object
+ */

Missing docs for the return value. SHould be

"Returns 0 on success, -1 on error."

note we do *not* return "-errno" values in QIOChannel APIs

Quoted 15 lines
include/io/channel.h +4 -0
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp);
+
 
io/channel-socket.c +2 -0
index bc5f80e780..5c1ea08660 100644
@@ -656,6 +656,24 @@ qio_channel_socket_set_blocking(QIOChannel *ioc,
  }
 
 
+static int
+qio_channel_socket_set_keepalive(QIOChannel *ioc,

io/channel-socket.c qio_channel_socket_set_blocking(QIOChannel *ioc, · line 670

Quoted 10 lines
io/channel-socket.c +10 -0
+                                 bool enabled,
+                                 Error **errp)
+{
+    QIOChannelSocket *sioc = QIO_CHANNEL_SOCKET(ioc);
+    int val = enabled;
+    int ret = qemu_setsockopt(sioc->fd, SOL_SOCKET, SO_KEEPALIVE,
+                              &val, sizeof(val));
+
+    if (ret < 0) {
+        error_setg_errno(errp, errno, "Unable to set KEEPALIVE");

Add 'return -1' there to be explicit, avoiding need to read up on whether qemu_setsockopt returns -1 or -errno. (It returns -1)

Quoted 24 lines
io/channel-socket.c +7 -0
+    }
+
+    return ret;
+}
+
+
  static void
  qio_channel_socket_set_delay(QIOChannel *ioc,
                               bool enabled)
@@ -762,6 +780,7 @@ static void qio_channel_socket_class_init(ObjectClass *klass,
      ioc_klass->io_writev = qio_channel_socket_writev;
      ioc_klass->io_readv = qio_channel_socket_readv;
      ioc_klass->io_set_blocking = qio_channel_socket_set_blocking;
+    ioc_klass->io_set_keepalive = qio_channel_socket_set_keepalive;
      ioc_klass->io_close = qio_channel_socket_close;
      ioc_klass->io_shutdown = qio_channel_socket_shutdown;
      ioc_klass->io_set_cork = qio_channel_socket_set_cork;
io/channel.c +0 -0
index 2a26c2a2c0..0f0b2b7b65 100644
@@ -265,6 +265,20 @@ int qio_channel_set_blocking(QIOChannel *ioc,
      return klass->io_set_blocking(ioc, enabled, errp);
  }

io/channel.c int qio_channel_set_blocking(QIOChannel *ioc, · line 276

Quoted 10 lines
io/channel.c +9 -0
 
+int qio_channel_set_keepalive(QIOChannel *ioc,
+                              bool enabled,
+                              Error **errp)
+{
+    QIOChannelClass *klass = QIO_CHANNEL_GET_CLASS(ioc);
+
+    if (!klass->io_set_keepalive) {
+        error_setg(errp, "KEEPALIVE is not supported by IO channel");
+        return -ENOTSUP;

return -1;

Quoted 6 lines
io/channel.c +5 -0
+    }
+
+    return klass->io_set_keepalive(ioc, enabled, errp);
+}
+
 
Quoted 6 lines

io/channel.c int qio_channel_set_blocking(QIOChannel *ioc, · line 284

Quoted 2 lines
io/channel.c +0 -0
  int qio_channel_close(QIOChannel *ioc,
                        Error **errp)

Regards,
Daniel

Thank you, I'll resend

Signature · 2 lines
Best regards,
Vladimir
#6 lore raw

PATCH 2/2 nbd-client: enable TCP keepalive

In reply to Vladimir Sementsov-Ogievskiy

Enable keepalive option to track server availablity.

---
 block/nbd-client.c | 1 +
 1 file changed, 1 insertion(+)
block/nbd-client.c +1 -0
index 790ecc1ee1..b57cea8482 100644
@@ -1137,6 +1137,7 @@ static int nbd_client_connect(BlockDriverState *bs,
 
     /* NBD handshake */
     logout("session init %s\n", export);
+    qio_channel_set_keepalive(QIO_CHANNEL(sioc), true, NULL);
     qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL);
 
     client->info.request_sizes = true;
Signature · 1 line
2.18.0
#7 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 1 line · Vladimir Sementsov-Ogievskiy

Enable keepalive option to track server availablity.

s/availablity/availability/

Do we want this unconditionally, or should it be an option (and hence exposed over QMP)?

Quoted 20 lines
---
 block/nbd-client.c | 1 +
 1 file changed, 1 insertion(+)
block/nbd-client.c +1 -0
index 790ecc1ee1..b57cea8482 100644
@@ -1137,6 +1137,7 @@ static int nbd_client_connect(BlockDriverState *bs,
 
     /* NBD handshake */
     logout("session init %s\n", export);
+    qio_channel_set_keepalive(QIO_CHANNEL(sioc), true, NULL);
     qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL);
 
     client->info.request_sizes = true;
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#8 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 6 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy

Enable keepalive option to track server availablity.

s/availablity/availability/

Do we want this unconditionally, or should it be an option (and hence exposed over QMP)?

That is good question, if we would expose it, we should specify timeout duration as an option. Though IMHO it would be safe to get this unconditional.

Quoted 19 lines
Quoted 19 lines
---
 block/nbd-client.c | 1 +
 1 file changed, 1 insertion(+)
block/nbd-client.c +1 -0
index 790ecc1ee1..b57cea8482 100644
@@ -1137,6 +1137,7 @@ static int nbd_client_connect(BlockDriverState *bs,
 
     /* NBD handshake */
     logout("session init %s\n", export);
+    qio_channel_set_keepalive(QIO_CHANNEL(sioc), true, NULL);
     qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL);
 
     client->info.request_sizes = true;
#9 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

In reply to Eric Blake
Quoted 7 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy

Enable keepalive option to track server availablity.

s/availablity/availability/

Do we want this unconditionally, or should it be an option (and hence exposed over QMP)?

I guess this is really a question about what our intended connection reliability policy should be.

By enabling TCP keepalives we are explicitly making the connection less reliable by forcing it to be terminated when keepalive threshold triggers, instead of waiting longer for TCP to recover.

The rationale s that once a connection has been in a hung state for so long that keepalive triggers, its (hopefully) not useful to the mgmt app to carry on waiting anyway.

If the connection is terminated by keepalive & the mgmt app then spawns a new client to carry on with the work, what are the risks involved ? eg Could packets from the stuck, terminated, connection suddenly arrive later and trigger I/O with outdated data payload ?

I guess this is no different a situation from an app explicitly killing the QEMU NBD client process instead & spawning a new one.

I'm still feeling a little uneasy about enabling it unconditionally though, since pretty much everything I know which supports keepalives has a way to turn them on/off at least, even if you can't tune the individual timer settings.

Quoted 12 lines
Quoted 12 lines
---
 block/nbd-client.c | 1 +
 1 file changed, 1 insertion(+)
block/nbd-client.c +0 -0
index 790ecc1ee1..b57cea8482 100644
@@ -1137,6 +1137,7 @@ static int nbd_client_connect(BlockDriverState *bs,
 
Quoted 13 lines
Quoted 7 lines
block/nbd-client.c +1 -0
     /* NBD handshake */
     logout("session init %s\n", export);
+    qio_channel_set_keepalive(QIO_CHANNEL(sioc), true, NULL);
     qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL);
 
     client->info.request_sizes = true;
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#10 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 16 lines · Daniel P. Berrangé
Quoted 7 lines · Eric Blake
Quoted 1 line · Vladimir Sementsov-Ogievskiy

Enable keepalive option to track server availablity.

s/availablity/availability/

Do we want this unconditionally, or should it be an option (and hence exposed over QMP)?

I guess this is really a question about what our intended connection reliability policy should be.

By enabling TCP keepalives we are explicitly making the connection less reliable by forcing it to be terminated when keepalive threshold triggers, instead of waiting longer for TCP to recover.

Quoted 8 lines

The rationale s that once a connection has been in a hung state for so long that keepalive triggers, its (hopefully) not useful to the mgmt app to carry on waiting anyway.

If the connection is terminated by keepalive & the mgmt app then spawns a new client to carry on with the work, what are the risks involved ? eg Could packets from the stuck, terminated, connection suddenly arrive later and trigger I/O with outdated data payload ?

Hmm, I believe that tcp guarantees isolation between different connections

Quoted 8 lines

I guess this is no different a situation from an app explicitly killing the QEMU NBD client process instead & spawning a new one.

I'm still feeling a little uneasy about enabling it unconditionally though, since pretty much everything I know which supports keepalives has a way to turn them on/off at least, even if you can't tune the individual timer settings.

Hm. So, I can add bool keepalive parameter for nbd format with default to true. And if needed, it may be later extended to be qapi 'alternate' of bool or struct with three numeric parameters, corresponding to TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL .

Opinions?

Quoted 33 lines
Quoted 13 lines
Quoted 13 lines
---
  block/nbd-client.c | 1 +
  1 file changed, 1 insertion(+)
block/nbd-client.c +0 -0
index 790ecc1ee1..b57cea8482 100644
@@ -1137,6 +1137,7 @@ static int nbd_client_connect(BlockDriverState *bs,
 
Quoted 13 lines
Quoted 7 lines
block/nbd-client.c +1 -0
      /* NBD handshake */
      logout("session init %s\n", export);
+    qio_channel_set_keepalive(QIO_CHANNEL(sioc), true, NULL);
      qio_channel_set_blocking(QIO_CHANNEL(sioc), true, NULL);
 
      client->info.request_sizes = true;
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Regards,
Daniel

Signature · 2 lines
Best regards,
Vladimir
#11 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines

By enabling TCP keepalives we are explicitly making the connection less reliable by forcing it to be terminated when keepalive threshold triggers, instead of waiting longer for TCP to recover.

Quoted 8 lines

The rationale s that once a connection has been in a hung state for so long that keepalive triggers, its (hopefully) not useful to the mgmt app to carry on waiting anyway.

If the connection is terminated by keepalive & the mgmt app then spawns a new client to carry on with the work, what are the risks involved ? eg Could packets from the stuck, terminated, connection suddenly arrive later and trigger I/O with outdated data payload ?

Hmm, I believe that tcp guarantees isolation between different connections

Quoted 8 lines

I guess this is no different a situation from an app explicitly killing the QEMU NBD client process instead & spawning a new one.

I'm still feeling a little uneasy about enabling it unconditionally though, since pretty much everything I know which supports keepalives has a way to turn them on/off at least, even if you can't tune the individual timer settings.

Quoted 5 lines

Hm. So, I can add bool keepalive parameter for nbd format with default to true. And if needed, it may be later extended to be qapi 'alternate' of bool or struct with three numeric parameters, corresponding to TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL .

Opinions?

Adding a bool that could later turn into a qapi 'alternate' for fine-tuning seems reasonable. Defaulting the bool to true is not backwards-compatible; better would be defaulting it to false and letting users opt-in; introspection will also work to let you know whether the feature is present.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#12 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 30 lines · Eric Blake
Quoted 28 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines

By enabling TCP keepalives we are explicitly making the connection less reliable by forcing it to be terminated when keepalive threshold triggers, instead of waiting longer for TCP to recover.

Quoted 8 lines

The rationale s that once a connection has been in a hung state for so long that keepalive triggers, its (hopefully) not useful to the mgmt app to carry on waiting anyway.

If the connection is terminated by keepalive & the mgmt app then spawns a new client to carry on with the work, what are the risks involved ? eg Could packets from the stuck, terminated, connection suddenly arrive later and trigger I/O with outdated data payload ?

Hmm, I believe that tcp guarantees isolation between different connections

Quoted 8 lines

I guess this is no different a situation from an app explicitly killing the QEMU NBD client process instead & spawning a new one.

I'm still feeling a little uneasy about enabling it unconditionally though, since pretty much everything I know which supports keepalives has a way to turn them on/off at least, even if you can't tune the individual timer settings.

Hm. So, I can add bool keepalive parameter for nbd format with default to true. And if needed, it may be later extended to be qapi 'alternate' of bool or struct with three numeric parameters, corresponding to TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL .

Quoted 8 lines
Quoted 1 line

Opinions?

Adding a bool that could later turn into a qapi 'alternate' for fine-tuning seems reasonable. Defaulting the bool to true is not backwards-compatible; better would be defaulting it to false and letting users opt-in; introspection will also work to let you know whether the feature is present.

Ok.

One more thing to discuss then. Should I add keepalive directly to BlockdevOptionsNbd?

Seems more useful to put it into SocketAddress, to be reused by other socket users.. But "SocketAddress" sounds like address, not like address+connection-options. On the other hand, structure names are not part of API. So, finally, is InetSocketAddress a good place for such thing?

Signature · 2 lines
Best regards,
Vladimir
#13 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 30 lines · Eric Blake
Quoted 28 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines

By enabling TCP keepalives we are explicitly making the connection less reliable by forcing it to be terminated when keepalive threshold triggers, instead of waiting longer for TCP to recover.

Quoted 8 lines

The rationale s that once a connection has been in a hung state for so long that keepalive triggers, its (hopefully) not useful to the mgmt app to carry on waiting anyway.

If the connection is terminated by keepalive & the mgmt app then spawns a new client to carry on with the work, what are the risks involved ? eg Could packets from the stuck, terminated, connection suddenly arrive later and trigger I/O with outdated data payload ?

Hmm, I believe that tcp guarantees isolation between different connections

Quoted 8 lines

I guess this is no different a situation from an app explicitly killing the QEMU NBD client process instead & spawning a new one.

I'm still feeling a little uneasy about enabling it unconditionally though, since pretty much everything I know which supports keepalives has a way to turn them on/off at least, even if you can't tune the individual timer settings.

Hm. So, I can add bool keepalive parameter for nbd format with default to true. And if needed, it may be later extended to be qapi 'alternate' of bool or struct with three numeric parameters, corresponding to TCP_KEEPCNT, TCP_KEEPIDLE and TCP_KEEPINTVL .

Quoted 8 lines
Quoted 1 line

Opinions?

Adding a bool that could later turn into a qapi 'alternate' for fine-tuning seems reasonable. Defaulting the bool to true is not backwards-compatible; better would be defaulting it to false and letting users opt-in; introspection will also work to let you know whether the feature is present.

Quoted 8 lines

Ok.

One more thing to discuss then. Should I add keepalive directly to BlockdevOptionsNbd?

Seems more useful to put it into SocketAddress, to be reused by other socket users.. But "SocketAddress" sounds like address, not like address+connection-options. On the other hand, structure names are not part of API. So, finally, is InetSocketAddress a good place for such thing?

That's an interesting idea. Using InetSocketAddress would mean that we could get support for this enabled "for free" everywhere in QEMU that uses an InetSocketAddress as its master config format.

Of course there's plenty of places not using InetSocketAddress that would still require some glue to wire up the code which converts the custom format into InetSocketAddress

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [Qemu-devel] [PATCH 2/2] nbd-client: enable TCP keepalive

Quoted 5 lines · Daniel P. Berrangé
Quoted 5 lines

Ok.

One more thing to discuss then. Should I add keepalive directly to BlockdevOptionsNbd?

Quoted 8 lines
Quoted 4 lines

Seems more useful to put it into SocketAddress, to be reused by other socket users.. But "SocketAddress" sounds like address, not like address+connection-options. On the other hand, structure names are not part of API. So, finally, is InetSocketAddress a good place for such thing?

That's an interesting idea. Using InetSocketAddress would mean that we could get support for this enabled "for free" everywhere in QEMU that uses an InetSocketAddress as its master config format.

I like the idea as well.

Quoted 4 lines

Of course there's plenty of places not using InetSocketAddress that would still require some glue to wire up the code which converts the custom format into InetSocketAddress

Hmm - how many places are we using InetSocketAddress (which allows an optional 'to' port value) when we really meant InetSocketAddressBase? There may be some interesting hierarchy decisions to consider on where we stick a keepalive option.

This also made me wonder if we should start a deprecation clock to improve the nbd-server-start command to use SocketAddress instead of SocketAddressLegacy. If we revive Max's work on implementing a default branch for a union discriminator
(https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01682.html),
we could have something like:

{ 'enum': 'NbdSocketAddressHack',
'data': [ 'legacy', 'inet', 'unix' ] }
{ 'struct': 'NbdServerAddrLegacy',
'data': { 'addr', 'SocketAddressLegacy' } }
{ 'union': 'NbdServerAddr',
'base': { 'type': 'NbdSocketAddressHack',

            '*tls-creds': 'str',
            '*tls-authz': 'str' },
  'discriminator': 'type',
  'default-variant': 'legacy',
  'data': { 'legacy': 'NbdServerAddrLegacy',
            'inet', 'InetSocketAddress',
            'unix', 'UnixSocketAddress' } }

{ 'command', 'nbd-server-start', 'data': 'NbdServerAddr' }

which should be backwards compatible with the existing:

{ "execute": "nbd-server-start", "arguments": {
    "tls-authz": "authz0",
    "addr": { "type": "inet", "data": {
      "host": "localhost", "port": "10809" } } } }

by relying on the discriminator's default expansion to:

{ "execute": "nbd-server-start", "arguments": {
    "tls-authz": "authz0",
    "type": "legacy",
    "addr": { "type": "inet", "data": {
      "host": "localhost", "port": "10809" } } } }

but also permit the flatter:

{ "execute": "nbd-server-start", "arguments": {
    "tls-authz": "authz0",
    "type": "inet", "host": "localhost", "port": "10809" } }

and let us start a deprecation clock to get rid of the "legacy" branch (especially if coupled with Kevin's work on adding introspectable deprecation annotations in QAPI).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#15 lore raw

[Qemu-devel] nbd-server-add [was: [PATCH 2/2] nbd-client: enable TCP keepalive]

Quoted 6 lines · Eric Blake

This also made me wonder if we should start a deprecation clock to improve the nbd-server-start command to use SocketAddress instead of SocketAddressLegacy. If we revive Max's work on implementing a default branch for a union discriminator
(https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg01682.html),
we could have something like:

Re-reading that thread, I see that Markus was arguing for a slightly different QAPI syntax than Max's proposal, basically:

Quoted 7 lines

{ 'enum': 'NbdSocketAddressHack',
'data': [ 'legacy', 'inet', 'unix' ] }
{ 'struct': 'NbdServerAddrLegacy',
'data': { 'addr', 'SocketAddressLegacy' } }
{ 'union': 'NbdServerAddr',
'base': { 'type': 'NbdSocketAddressHack',

Quoted 8 lines
            '*tls-creds': 'str',
            '*tls-authz': 'str' },
  'discriminator': 'type',
  'default-variant': 'legacy',
  'data': { 'legacy': 'NbdServerAddrLegacy',
            'inet', 'InetSocketAddress',
            'unix', 'UnixSocketAddress' } }

{ 'command', 'nbd-server-start', 'data': 'NbdServerAddr' }

{ 'union': 'NbdServerAddr',
'base': { '*type': { 'type': 'NbdSocketAddressHack',

                       'default': 'legacy' },
            '*tls-creds', 'str', '*tls-authz', 'str' },
  'discriminator': 'type',
  'data': { 'legacy': 'NbdServerAddrLegacy',
            'inet', 'InetSocketAddress',
            'unix', 'UnixSocketAddress' } }
Quoted 8 lines

which should be backwards compatible with the existing:

{ "execute": "nbd-server-start", "arguments": {
    "tls-authz": "authz0",
    "addr": { "type": "inet", "data": {
      "host": "localhost", "port": "10809" } } } }
Quoted 7 lines

by relying on the discriminator's default expansion to:

{ "execute": "nbd-server-start", "arguments": {
    "tls-authz": "authz0",
    "type": "legacy",
    "addr": { "type": "inet", "data": {
      "host": "localhost", "port": "10809" } } } }

But this part remains true - if a flat union has an optional discriminator, then the discriminator must include a default value, where omitting the discriminator then results in sane expansion, and where a careful choice of discriminator default allows legacy syntax to co-exist with new preferred syntax.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org