Skip to contentSkip to conversation

Revival of patches to implement NBD client multi-conn

RFC PATCH In review Eric Blake 6 participants 22 messages Apr 28, 2025 – May 28, 2025 Block layer

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v2 cover Revival of patches to implement NBD client multi-conn

This is a rebase of patches that Rich Jones first posted in 2023:
https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03320.html

I'm still leaving the series in an RFC state while I'm trying to improve the code to better play with all of the multi-queue changes in the block layer in the meantime. In my own testing, the patches as presented here are not making any noticeable difference in qemu-img convert to a local destination file (multi-conn=1 was not much different than multi-conn=4, although I did validate that multiple sockets were in use and the round robin code was working). Other benchmarks did show improvements, such as qemu convert targeting an nbdkit server on a curl backend getting a 20% boost when I ramped multi-conn from 1 to 4.

I have more patches on top of these to post to the list once I can get benchmark numbers that make more sense.

Richard W.M. Jones (4):
nbd: Add multi-conn option
nbd: Split out block device state from underlying NBD connections
nbd: Open multiple NBD connections if multi-conn is set
nbd: Enable multi-conn using round-robin

 qapi/block-core.json |   8 +-
 block/coroutines.h   |   5 +-
 block/nbd.c          | 796 +++++++++++++++++++++++++------------------
 3 files changed, 479 insertions(+), 330 deletions(-)
Signature · 1 line
2.49.0
author
#1 lore raw

PATCH v2 1/4 nbd: Add multi-conn option

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +7 -1
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +24 -0
index d5a2b21c6d1..5eb00e360af 100644
@@ -48,6 +48,7 @@
 
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16
 
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",
+        },
         { /* end of list */ }
     },
 };
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }
 
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }
+
     return 0;
 
 fail:
Signature · 1 line
2.49.0
#2 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 23 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +0 -0
index 7f70ec6d3cb..5c10824f35b 100644

qapi/block-core.json line 4552

Quoted 8 lines
qapi/block-core.json +5 -0
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#

So we silently ignore @multi-conn when its value is (nonsensical) zero, and when the server doesn't let us honor the value. Hmm. Silently ignoring the user's wishes can result in confusion. Should we reject instead?

Quoted 18 lines
qapi/block-core.json +2 -1
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +0 -0
index d5a2b21c6d1..5eb00e360af 100644

block/nbd.c line 51

Quoted 4 lines
block/nbd.c +1 -0
@@ -48,6 +48,7 @@
 
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16

Out of curiosity: where does this value come from?

Quoted 13 lines
block/nbd.c +1 -0
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "

block/nbd.c static QemuOptsList nbd_runtime_opts = { · line 1852

Quoted 10 lines
block/nbd.c +8 -0
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",

This text implies the requested value is silently limited to the value provided by the server, unlike the doc comment above. Although the "must" in "the sever must" could also be understood as "error when it doesn't".

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1912

Quoted 12 lines
block/nbd.c +5 -0
+        },
         { /* end of list */ }
     },
 };
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }

We silently cap the user's requested number to 16. Not clear from QAPI schema doc comment; the "up to 16" there suggests more is an error. Should we error out instead?

Quoted 19 lines
block/nbd.c +9 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }
+
     return 0;
 
 fail:
#3 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 30 lines · Markus Armbruster
Quoted 28 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +2 -0
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
Quoted 8 lines

qapi/block-core.json line 4552

Quoted 3 lines
qapi/block-core.json +3 -0
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#

So we silently ignore @multi-conn when its value is (nonsensical) zero, and when the server doesn't let us honor the value. Hmm. Silently ignoring the user's wishes can result in confusion. Should we reject instead?

We could certainly reject 0. It's also possible to reject the case where multi-conn is not supported by the server, but is requested by the client, but I feel that's a bit user-unfriendly. After all, multi-conn isn't essential for it to work, it's needed if you want best performance. (Maybe issue a warning in the code - below - where we set multi-conn back to 1? I don't know what qemu thinks about warnings.)

Quoted 19 lines
Quoted 19 lines
qapi/block-core.json +2 -1
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +0 -0
index d5a2b21c6d1..5eb00e360af 100644
@@ -48,6 +48,7 @@
 
Quoted 5 lines

block/nbd.c line 51

Quoted 3 lines
block/nbd.c +1 -0
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16

Out of curiosity: where does this value come from?

So I should note first this is a maximum, not a recommendation.

nbdcopy defaults to 4, which was derived from testing on a high end (for 2024) AMD machine. Above 4 performance doesn't increase any further on that machine. It's going to very much depend on how many cores you have spare, how many TCP connections you want to open, and how effectively the client and server handle parallelism.

And imponderables like one we hit in virt-v2v: If accessing a VMware server, the VMware server actually slows down as you add more connections, even though it should theoretically support multi-conn. We ended up forcing multi-conn to 1 in this case. You can't know this in advance from the client side.

Quoted 20 lines
Quoted 20 lines
block/nbd.c +6 -0
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
Quoted 8 lines

block/nbd.c static QemuOptsList nbd_runtime_opts = { · line 1852

Quoted 3 lines
block/nbd.c +3 -0
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",

This text implies the requested value is silently limited to the value provided by the server, unlike the doc comment above. Although the "must" in "the sever must" could also be understood as "error when it doesn't".

I'll just note that multi-conn is a boolean flag advertised by the server. Servers don't advertise any preferred number of connections. I don't know how to improve the text.

Quoted 35 lines

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1912

Quoted 12 lines
block/nbd.c +5 -0
+        },
         { /* end of list */ }
     },
 };
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }

We silently cap the user's requested number to 16. Not clear from QAPI schema doc comment; the "up to 16" there suggests more is an error. Should we error out instead?

Quoted 18 lines
block/nbd.c +9 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }
+
     return 0;
 
Quoted 1 line

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1978

Quoted 1 line
block/nbd.c +0 -0
 fail:

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
#4 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 40 lines · Richard W.M. Jones
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +2 -0
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
Quoted 8 lines

qapi/block-core.json line 4552

Quoted 3 lines
qapi/block-core.json +3 -0
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#

So we silently ignore @multi-conn when its value is (nonsensical) zero, and when the server doesn't let us honor the value. Hmm. Silently ignoring the user's wishes can result in confusion. Should we reject instead?

Quoted 7 lines

We could certainly reject 0. It's also possible to reject the case where multi-conn is not supported by the server, but is requested by the client, but I feel that's a bit user-unfriendly. After all, multi-conn isn't essential for it to work, it's needed if you want best performance. (Maybe issue a warning in the code - below - where we set multi-conn back to 1? I don't know what qemu thinks about warnings.)

QMP doesn't support warnings, so they go to stderr instead, where they may or may not be seen.

When I instruct a program to do X, I prefer it to do exactly X, and fail when that's not possible. Correcting X behind my back may be friendly, until the day I spent quality time figuring out WTF is going on.

Perhaps this one is a matter of documentation. As is, @multi-conn feels like "set the number of connections" to me, until I read the fine print, which contradicts it. We could perhaps phrase it as a limit instead:
enable multiple connections and simultaneously limit their number.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
qapi/block-core.json +2 -1
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +0 -0
index d5a2b21c6d1..5eb00e360af 100644
@@ -48,6 +48,7 @@
 
Quoted 7 lines
Quoted 5 lines

block/nbd.c line 51

Quoted 3 lines
block/nbd.c +1 -0
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16

Out of curiosity: where does this value come from?

So I should note first this is a maximum, not a recommendation.

Is it the arbitrarily chosen maximum for QEMU, or is it the well-known maximum for NBD, or is it something else?

Quoted 41 lines

nbdcopy defaults to 4, which was derived from testing on a high end (for 2024) AMD machine. Above 4 performance doesn't increase any further on that machine. It's going to very much depend on how many cores you have spare, how many TCP connections you want to open, and how effectively the client and server handle parallelism.

And imponderables like one we hit in virt-v2v: If accessing a VMware server, the VMware server actually slows down as you add more connections, even though it should theoretically support multi-conn. We ended up forcing multi-conn to 1 in this case. You can't know this in advance from the client side.

Quoted 20 lines
Quoted 20 lines
block/nbd.c +6 -0
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
Quoted 8 lines

block/nbd.c static QemuOptsList nbd_runtime_opts = { · line 1852

Quoted 3 lines
block/nbd.c +3 -0
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",

This text implies the requested value is silently limited to the value provided by the server, unlike the doc comment above. Although the "must" in "the sever must" could also be understood as "error when it doesn't".

Quoted 3 lines

I'll just note that multi-conn is a boolean flag advertised by the server. Servers don't advertise any preferred number of connections. I don't know how to improve the text.

Let's get the QAPI schema doc comment right before we worry about this one.

Quoted 38 lines
Quoted 35 lines

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1912

Quoted 12 lines
block/nbd.c +5 -0
+        },
         { /* end of list */ }
     },
 };
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }

We silently cap the user's requested number to 16. Not clear from QAPI schema doc comment; the "up to 16" there suggests more is an error. Should we error out instead?

Quoted 18 lines
block/nbd.c +9 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }
+
     return 0;
 
Quoted 1 line

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1978

Quoted 1 line
block/nbd.c +0 -0
 fail:

Rich.

#5 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 53 lines · Markus Armbruster
Quoted 40 lines · Richard W.M. Jones
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +2 -0
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
Quoted 8 lines

qapi/block-core.json line 4552

Quoted 3 lines
qapi/block-core.json +3 -0
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#

So we silently ignore @multi-conn when its value is (nonsensical) zero, and when the server doesn't let us honor the value. Hmm. Silently ignoring the user's wishes can result in confusion. Should we reject instead?

Quoted 7 lines

We could certainly reject 0. It's also possible to reject the case where multi-conn is not supported by the server, but is requested by the client, but I feel that's a bit user-unfriendly. After all, multi-conn isn't essential for it to work, it's needed if you want best performance. (Maybe issue a warning in the code - below - where we set multi-conn back to 1? I don't know what qemu thinks about warnings.)

QMP doesn't support warnings, so they go to stderr instead, where they may or may not be seen.

Quoted 8 lines

When I instruct a program to do X, I prefer it to do exactly X, and fail when that's not possible. Correcting X behind my back may be friendly, until the day I spent quality time figuring out WTF is going on.

Perhaps this one is a matter of documentation. As is, @multi-conn feels like "set the number of connections" to me, until I read the fine print, which contradicts it. We could perhaps phrase it as a limit instead:
enable multiple connections and simultaneously limit their number.

It is tricky. In nbdcopy we've preferred to go with "you suggest some numbers and we'll pick something that works":

https://gitlab.com/nbdkit/libnbd/-/blob/master/copy/main.c?ref_type=heads#L446-L493

but also we do provide a way for you to find out what was selected:

https://gitlab.com/nbdkit/libnbd/-/blob/master/copy/main.c?ref_type=heads#L521

(I'm not claiming this is the best approach or suitable for everyone.)

In the context of qemu that might suggest having separate multi_conn_requested and multi_conn fields, where the latter can be queried over QMP to find out what is actually going on. Could even add multi_conn_max to allow MAX_MULTI_CONN constant to be read out.

Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 18 lines
qapi/block-core.json +2 -1
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +0 -0
index d5a2b21c6d1..5eb00e360af 100644

block/nbd.c line 51

Quoted 4 lines
block/nbd.c +1 -0
@@ -48,6 +48,7 @@
 
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16

Out of curiosity: where does this value come from?

Quoted 4 lines
Quoted 1 line

So I should note first this is a maximum, not a recommendation.

Is it the arbitrarily chosen maximum for QEMU, or is it the well-known maximum for NBD, or is it something else?

I don't recall exactly, but it was probably an ass-pulled number.

Quoted 41 lines
Quoted 41 lines

nbdcopy defaults to 4, which was derived from testing on a high end (for 2024) AMD machine. Above 4 performance doesn't increase any further on that machine. It's going to very much depend on how many cores you have spare, how many TCP connections you want to open, and how effectively the client and server handle parallelism.

And imponderables like one we hit in virt-v2v: If accessing a VMware server, the VMware server actually slows down as you add more connections, even though it should theoretically support multi-conn. We ended up forcing multi-conn to 1 in this case. You can't know this in advance from the client side.

Quoted 28 lines
Quoted 13 lines
block/nbd.c +1 -0
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "

block/nbd.c static QemuOptsList nbd_runtime_opts = { · line 1852

Quoted 10 lines
block/nbd.c +8 -0
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",

This text implies the requested value is silently limited to the value provided by the server, unlike the doc comment above. Although the "must" in "the sever must" could also be understood as "error when it doesn't".

Quoted 45 lines
Quoted 3 lines

I'll just note that multi-conn is a boolean flag advertised by the server. Servers don't advertise any preferred number of connections. I don't know how to improve the text.

Let's get the QAPI schema doc comment right before we worry about this one.

Quoted 38 lines
Quoted 35 lines

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1912

Quoted 12 lines
block/nbd.c +5 -0
+        },
         { /* end of list */ }
     },
 };
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }

We silently cap the user's requested number to 16. Not clear from QAPI schema doc comment; the "up to 16" there suggests more is an error. Should we error out instead?

Quoted 18 lines
block/nbd.c +9 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }
+
     return 0;
 
Quoted 1 line

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1978

Quoted 1 line
block/nbd.c +0 -0
 fail:

Rich.

Rich.

Signature · 4 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
#6 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 70 lines · Richard W.M. Jones
Quoted 53 lines · Markus Armbruster
Quoted 40 lines · Richard W.M. Jones
Quoted 30 lines · Markus Armbruster
Quoted 28 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
qapi/block-core.json +2 -0
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
Quoted 8 lines

qapi/block-core.json line 4552

Quoted 3 lines
qapi/block-core.json +3 -0
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#

So we silently ignore @multi-conn when its value is (nonsensical) zero, and when the server doesn't let us honor the value. Hmm. Silently ignoring the user's wishes can result in confusion. Should we reject instead?

Quoted 7 lines

We could certainly reject 0. It's also possible to reject the case where multi-conn is not supported by the server, but is requested by the client, but I feel that's a bit user-unfriendly. After all, multi-conn isn't essential for it to work, it's needed if you want best performance. (Maybe issue a warning in the code - below - where we set multi-conn back to 1? I don't know what qemu thinks about warnings.)

QMP doesn't support warnings, so they go to stderr instead, where they may or may not be seen.

Quoted 8 lines

When I instruct a program to do X, I prefer it to do exactly X, and fail when that's not possible. Correcting X behind my back may be friendly, until the day I spent quality time figuring out WTF is going on.

Perhaps this one is a matter of documentation. As is, @multi-conn feels like "set the number of connections" to me, until I read the fine print, which contradicts it. We could perhaps phrase it as a limit instead:
enable multiple connections and simultaneously limit their number.

It is tricky. In nbdcopy we've preferred to go with "you suggest some numbers and we'll pick something that works":

https://gitlab.com/nbdkit/libnbd/-/blob/master/copy/main.c?ref_type=heads#L446-L493

but also we do provide a way for you to find out what was selected:

Quoted 8 lines

https://gitlab.com/nbdkit/libnbd/-/blob/master/copy/main.c?ref_type=heads#L521

(I'm not claiming this is the best approach or suitable for everyone.)

In the context of qemu that might suggest having separate multi_conn_requested and multi_conn fields, where the latter can be queried over QMP to find out what is actually going on. Could even add multi_conn_max to allow MAX_MULTI_CONN constant to be read out.

You decide what to do with my feedback :)

[...]

author
#7 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 6 lines · Markus Armbruster
Quoted 4 lines

In the context of qemu that might suggest having separate multi_conn_requested and multi_conn fields, where the latter can be queried over QMP to find out what is actually going on. Could even add multi_conn_max to allow MAX_MULTI_CONN constant to be read out.

You decide what to do with my feedback :)

I've got a local patch that adds the ability for query-named-block-nodes (and query-block) to output image-specific information for NBD that includes how many connections are actually in use. But now I've got a QMP question:

My patch, as written, makes the output look like this:

"format-specific": {"mode": "extended", "type": "nbd", "connections": 1}},

by changing block-core.json like this (partial patch shown):

@@ -208,10 +223,12 @@
#
# @file: Since 8.0
#
+# @nbd: Since 10.1
+#
# Since: 1.7
##
{ 'enum': 'ImageInfoSpecificKind',
- 'data': [ 'qcow2', 'vmdk', 'luks', 'rbd', 'file' ] }
+ 'data': [ 'qcow2', 'vmdk', 'luks', 'rbd', 'file', 'nbd' ] }

##
# @ImageInfoSpecificQCow2Wrapper:
@@ -284,7 +301,8 @@
'luks': 'ImageInfoSpecificLUKSWrapper',
'rbd': 'ImageInfoSpecificRbdWrapper',

-      'file': 'ImageInfoSpecificFileWrapper'
+      'file': 'ImageInfoSpecificFileWrapper',
+      'nbd': 'ImageInfoSpecificNbd'

} }

But that is different from all of the other image modes, where the output looks more like:

"format-specific": {"type": "rbd", "data": {"encryption-format":"..."}}},

note the extra layer of nesting, due to historical reasons of being added in a time when the QMP generator was not as nice on supporting flatter union-style coding.

Must I create an ImageInfoSpecificNbdWrapper type, with the sole purpose of having the same (pointless, IMO) "data":{} wrapper as all the other branches of the union type, or am I okay with my addition using the flatter style?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
#8 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 44 lines · Eric Blake
Quoted 6 lines · Markus Armbruster
Quoted 4 lines

In the context of qemu that might suggest having separate multi_conn_requested and multi_conn fields, where the latter can be queried over QMP to find out what is actually going on. Could even add multi_conn_max to allow MAX_MULTI_CONN constant to be read out.

You decide what to do with my feedback :)

I've got a local patch that adds the ability for query-named-block-nodes (and query-block) to output image-specific information for NBD that includes how many connections are actually in use. But now I've got a QMP question:

My patch, as written, makes the output look like this:

"format-specific": {"mode": "extended", "type": "nbd", "connections": 1}},

by changing block-core.json like this (partial patch shown):

(quoted diff) +5 -2
@@ -208,10 +223,12 @@
 #
 # @file: Since 8.0
 #
+# @nbd: Since 10.1
+#
 # Since: 1.7
 ##
 { 'enum': 'ImageInfoSpecificKind',
-  'data': [ 'qcow2', 'vmdk', 'luks', 'rbd', 'file' ] }
+  'data': [ 'qcow2', 'vmdk', 'luks', 'rbd', 'file', 'nbd' ] }
 
 ##
 # @ImageInfoSpecificQCow2Wrapper:
@@ -284,7 +301,8 @@
       'luks': 'ImageInfoSpecificLUKSWrapper',
       'rbd': 'ImageInfoSpecificRbdWrapper',
-      'file': 'ImageInfoSpecificFileWrapper'
+      'file': 'ImageInfoSpecificFileWrapper',
+      'nbd': 'ImageInfoSpecificNbd'
   } }
 

But that is different from all of the other image modes, where the output looks more like:

Quoted 5 lines

"format-specific": {"type": "rbd", "data": {"encryption-format":"..."}}},

note the extra layer of nesting, due to historical reasons of being added in a time when the QMP generator was not as nice on supporting flatter union-style coding.

Correct, this is an artifact of development history, specifically "simple" unions and their elimination.

Quoted 4 lines

Must I create an ImageInfoSpecificNbdWrapper type, with the sole purpose of having the same (pointless, IMO) "data":{} wrapper as all the other branches of the union type, or am I okay with my addition using the flatter style?

I dislike these wrappers, possibly more than is reasonable. Still, I think local consistency is more important.

Precedence: commit 7f36a50ab4e (block/file: Add file-specific image info) added a new branch with a wrapper well after we eliminated "simple" unions.

I think you should add the silly wrapper.

#9 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

In reply to Eric Blake
Quoted 10 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

Quoted 8 lines
[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

Pardon my nitpicking, but it seems to me that the name "multi-conn" doesn't really suggest "number of simultaneous NBD client connections", especially when similarly named NBD_FLAG_CAN_MULTI_CONN_BIT represents binary logic: supported or not supported. Maybe smth like "multi_conns_num" would be better? Or anything else as long as it's clear it's an int, not bool.

Quoted 63 lines
qapi/block-core.json +7 -1
index 7f70ec6d3cb..5c10824f35b 100644
@@ -4545,6 +4545,11 @@
 #     until successful or until @open-timeout seconds have elapsed.
 #     Default 0 (Since 7.0)
 #
+# @multi-conn: Request the number of parallel client connections to make
+#     to the server, up to 16.  If the server does not advertise support
+#     for multiple connections, or if this value is 0 or 1, all traffic
+#     is sent through a single connection.  Default 1 (Since 10.1)
+#
 # Features:
 #
 # @unstable: Member @x-dirty-bitmap is experimental.
@@ -4558,7 +4563,8 @@
             '*tls-hostname': 'str',
             '*x-dirty-bitmap': { 'type': 'str', 'features': [ 'unstable' ] },
             '*reconnect-delay': 'uint32',
-            '*open-timeout': 'uint32' } }
+            '*open-timeout': 'uint32',
+            '*multi-conn': 'uint32' } }
 
 ##
 # @BlockdevOptionsRaw:
block/nbd.c +11 -0
index d5a2b21c6d1..5eb00e360af 100644
@@ -48,6 +48,7 @@
 
 #define EN_OPTSTR ":exportname="
 #define MAX_NBD_REQUESTS    16
+#define MAX_MULTI_CONN      16
 
 #define COOKIE_TO_INDEX(cookie) ((cookie) - 1)
 #define INDEX_TO_COOKIE(index)  ((index) + 1)
@@ -97,6 +98,7 @@ typedef struct BDRVNBDState {
     /* Connection parameters */
     uint32_t reconnect_delay;
     uint32_t open_timeout;
+    uint32_t multi_conn;
     SocketAddress *saddr;
     char *export;
     char *tlscredsid;
@@ -1840,6 +1842,15 @@ static QemuOptsList nbd_runtime_opts = {
                     "attempts until successful or until @open-timeout seconds "
                     "have elapsed. Default 0",
         },
+        {
+            .name = "multi-conn",
+            .type = QEMU_OPT_NUMBER,
+            .help = "If > 1 permit up to this number of connections to the "
+                    "server. The server must also advertise multi-conn "
+                    "support.  If <= 1, only a single connection is made "
+                    "to the server even if the server advertises multi-conn. "
+                    "Default 1",
+        },
         { /* end of list */ }
     },
 };

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1913

Quoted 8 lines
block/nbd.c +4 -0
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }
 

I agree with Markus that this setting value different from what's been directly requested by user shouldn't go silent. Having some kind of warning at the very least would be nice.

Quoted 4 lines
block/nbd.c +0 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1974

Quoted 10 lines
block/nbd.c +8 -0
     nbd_client_connection_enable_retry(s->conn);
 
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }

Same here.

Quoted 4 lines
block/nbd.c +1 -0
+
     return 0;
 
 fail:
author
#10 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 20 lines · Andrey Drobyshev
Quoted 10 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

Quoted 8 lines
[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
Quoted 6 lines

Pardon my nitpicking, but it seems to me that the name "multi-conn" doesn't really suggest "number of simultaneous NBD client connections", especially when similarly named NBD_FLAG_CAN_MULTI_CONN_BIT represents binary logic: supported or not supported. Maybe smth like "multi_conns_num" would be better? Or anything else as long as it's clear it's an int, not bool.

Maybe 'max-multi-conn-clients', since it is something that even if the user sets it to larger than 1 but the server doesn't advertise the bit, then we silently restrict to one client.

Quoted 13 lines

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1913

Quoted 9 lines
block/nbd.c +4 -0
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }
 

I agree with Markus that this setting value different from what's been directly requested by user shouldn't go silent. Having some kind of warning at the very least would be nice.

Okay, I'll make sure to warn if it exceeds the compile-time max.

Quoted 7 lines
Quoted 7 lines
block/nbd.c +0 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
Quoted 10 lines

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1974

Quoted 8 lines
block/nbd.c +8 -0
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }

Same here.

Here, I disagree. But that's where better naming comes into play: if there is 'max-' in the name, then the user should not be surprised if they don't actually achieve the max (because the server lacked support). On the other hand, I could see how you might want to know if you have a mismatched setup ("I think the server SHOULD be supporting multi-conn, so I request multiple clients, and I want to be informed if my expectations were not met because then I know to go reconfigure the server"). Thoughts?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
#11 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 66 lines · Eric Blake
Quoted 20 lines · Andrey Drobyshev
Quoted 10 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Add multi-conn option to the NBD client. This commit just adds the option, it is not functional.

Setting this to a value > 1 permits multiple connections to the NBD server; a typical value might be 4. The default is 1, meaning only a single connection is made. If the NBD server does not advertise that it is safe for multi-conn then this setting is forced to 1.

Quoted 8 lines
[eblake: also expose it through QMP]
---
 qapi/block-core.json |  8 +++++++-
 block/nbd.c          | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)
Quoted 6 lines

Pardon my nitpicking, but it seems to me that the name "multi-conn" doesn't really suggest "number of simultaneous NBD client connections", especially when similarly named NBD_FLAG_CAN_MULTI_CONN_BIT represents binary logic: supported or not supported. Maybe smth like "multi_conns_num" would be better? Or anything else as long as it's clear it's an int, not bool.

Maybe 'max-multi-conn-clients', since it is something that even if the user sets it to larger than 1 but the server doesn't advertise the bit, then we silently restrict to one client.

Quoted 13 lines

block/nbd.c static int nbd_process_options(BlockDriverState *bs, QDict *options, · line 1913

Quoted 9 lines
block/nbd.c +4 -0
@@ -1895,6 +1906,10 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
 
     s->reconnect_delay = qemu_opt_get_number(opts, "reconnect-delay", 0);
     s->open_timeout = qemu_opt_get_number(opts, "open-timeout", 0);
+    s->multi_conn = qemu_opt_get_number(opts, "multi-conn", 1);
+    if (s->multi_conn > MAX_MULTI_CONN) {
+        s->multi_conn = MAX_MULTI_CONN;
+    }
 

I agree with Markus that this setting value different from what's been directly requested by user shouldn't go silent. Having some kind of warning at the very least would be nice.

Okay, I'll make sure to warn if it exceeds the compile-time max.

Quoted 7 lines
Quoted 7 lines
block/nbd.c +0 -0
     ret = 0;
 
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
Quoted 10 lines

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1974

Quoted 8 lines
block/nbd.c +8 -0
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }

Same here.

Quoted 9 lines

Here, I disagree. But that's where better naming comes into play: if there is 'max-' in the name, then the user should not be surprised if they don't actually achieve the max (because the server lacked support). On the other hand, I could see how you might want to know if you have a mismatched setup ("I think the server SHOULD be supporting multi-conn, so I request multiple clients, and I want to be informed if my expectations were not met because then I know to go reconfigure the server"). Thoughts?

Doesn't the "max-" part suggest that there might be anything in between [1..N]? When in reality it's either of {1, min(N, MAX_MULTI_CONN)}. But you're right, my initial argument was that this mismatch shouldn't go unnoticed as well. Although I agree that it's part of the expected behavior and therefore might not deserve a warning. Maybe smth like info_report() will do? We might even print it unconditionally, so that there's always a way to tell the actual number of connections chosen. Somewhat similar to what Richard pointed out at in nbdcopy.

Andrey

author
#12 lore raw

Re: [PATCH v2 1/4] nbd: Add multi-conn option

Quoted 5 lines · Andrey Drobyshev
Quoted 5 lines
Quoted 3 lines

I agree with Markus that this setting value different from what's been directly requested by user shouldn't go silent. Having some kind of warning at the very least would be nice.

Okay, I'll make sure to warn if it exceeds the compile-time max.

Rather, refuse the QMP command with an error.

Quoted 29 lines
Quoted 19 lines
Quoted 8 lines
Quoted 8 lines

     ret = 0;
block/nbd.c +0 -0
@@ -1949,6 +1964,15 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
 
     nbd_client_connection_enable_retry(s->conn);
 
Quoted 10 lines

block/nbd.c static int nbd_open(BlockDriverState *bs, QDict *options, int flags, · line 1974

Quoted 8 lines
block/nbd.c +8 -0
+    /*
+     * We set s->multi_conn in nbd_process_options above, but now that
+     * we have connected if the server doesn't advertise that it is
+     * safe for multi-conn, force it to 1.
+     */
+    if (!(s->info.flags & NBD_FLAG_CAN_MULTI_CONN)) {
+        s->multi_conn = 1;
+    }

Same here.

Quoted 9 lines

Here, I disagree. But that's where better naming comes into play: if there is 'max-' in the name, then the user should not be surprised if they don't actually achieve the max (because the server lacked support). On the other hand, I could see how you might want to know if you have a mismatched setup ("I think the server SHOULD be supporting multi-conn, so I request multiple clients, and I want to be informed if my expectations were not met because then I know to go reconfigure the server"). Thoughts?

Quoted 8 lines

Doesn't the "max-" part suggest that there might be anything in between [1..N]? When in reality it's either of {1, min(N, MAX_MULTI_CONN)}. But you're right, my initial argument was that this mismatch shouldn't go unnoticed as well. Although I agree that it's part of the expected behavior and therefore might not deserve a warning. Maybe smth like info_report() will do? We might even print it unconditionally, so that there's always a way to tell the actual number of connections chosen. Somewhat similar to what Richard pointed out at in nbdcopy.

Warnings in QMP are difficult. It's easy to fail a command, but hard to print non-fatal messages. However, I'm not opposed to having a way to use a QMP query-* command as a followup for the user who is curious on how many connections a given block export is using (either we can already query all existing block exports, or that's something that I should add independent of the new multi-conn setting).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
author
#13 lore raw

PATCH v2 2/4 nbd: Split out block device state from underlying NBD connections

In reply to Eric Blake

From: "Richard W.M. Jones" <[email protected]>

To implement multi-conn, we will put multiple underlying NBD connections (ie. NBDClientConnection) inside the NBD block device handle (BDRVNBDState). This requires first breaking the one-to-one relationship between NBDClientConnection and BDRVNBDState.

To do this a new structure (NBDConnState) is implemented. NBDConnState takes all the per-connection state out of the block driver struct. BDRVNBDState now contains a conns[] array of pointers to NBDConnState, one for each underlying multi-conn connection.

After this change there is still a one-to-one relationship because we only ever use the zeroth slot in the conns[] array. Thus this does not implement multi-conn yet.

Message-ID: <[email protected]>

[eblake: rebase to changes in the meantime]
---
 block/coroutines.h |   5 +-
 block/nbd.c        | 671 ++++++++++++++++++++++++---------------------
 2 files changed, 356 insertions(+), 320 deletions(-)
block/coroutines.h +3 -2
index 892646bb7aa..d611408fa14 100644
@@ -62,7 +62,7 @@ int coroutine_fn GRAPH_RDLOCK
 bdrv_co_writev_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos);
 
 int coroutine_fn GRAPH_RDLOCK
-nbd_co_do_establish_connection(BlockDriverState *bs, bool blocking,
+nbd_co_do_establish_connection(BlockDriverState *bs, void *cs, bool blocking,
                                Error **errp);
 
 
@@ -87,6 +87,7 @@ bdrv_common_block_status_above(BlockDriverState *bs,
                                int *depth);
 
 int co_wrapper_mixed_bdrv_rdlock
-nbd_do_establish_connection(BlockDriverState *bs, bool blocking, Error **errp);
+nbd_do_establish_connection(BlockDriverState *bs, void *cs, bool blocking,
+                            Error **errp);
 
 #endif /* BLOCK_COROUTINES_H */
block/nbd.c +353 -318

1,476 lines not on this page · raw message

Signature · 1 line
2.49.0
author
#14 lore raw

PATCH v2 3/4 nbd: Open multiple NBD connections if multi-conn is set

In reply to Eric Blake

From: "Richard W.M. Jones" <[email protected]>

If the user opts in to multi-conn and the server advertises it, open multiple NBD connections. They are opened with the same parameters as the initial connection. Similarly on the close path the connections are closed.

However only the zeroth connection is used, so this commit does not actually enable multi-conn capabilities.

(XXX) The strategy here is very naive. Firstly if you were going to open them, you'd probably want to open them in parallel. Secondly it would make sense to delay opening until multiple parallel requests are seen (perhaps above some threshold), so that simple or shortlived NBD operations do not require multiple connections to be made.

---
 block/nbd.c | 128 ++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 90 insertions(+), 38 deletions(-)
block/nbd.c +90 -38
index 4690599bc61..19da1a7a1fe 100644
@@ -123,18 +123,23 @@ static void nbd_yank(void *opaque);
 static void nbd_clear_bdrvstate(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
+    size_t i;
 
-    nbd_client_connection_release(s->conns[0]->conn);
-    s->conns[0]->conn = NULL;
+    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+        if (s->conns[i]) {
+            nbd_client_connection_release(s->conns[i]->conn);
+            s->conns[i]->conn = NULL;
+
+            /* Must not leave timers behind that would access freed data */
+            assert(!s->conns[i]->reconnect_delay_timer);
+            assert(!s->conns[i]->open_timer);
+
+            g_free(s->conns[i]);
+        }
+    }
 
     yank_unregister_instance(BLOCKDEV_YANK_INSTANCE(bs->node_name));
 
-    /* Must not leave timers behind that would access freed data */
-    assert(!s->conns[0]->reconnect_delay_timer);
-    assert(!s->conns[0]->open_timer);
-
-    g_free(s->conns[0]);
-
     object_unref(OBJECT(s->tlscreds));
     qapi_free_SocketAddress(s->saddr);
     s->saddr = NULL;
@@ -1941,21 +1946,68 @@ static int nbd_process_options(BlockDriverState *bs, QDict *options,
     return ret;
 }
 
+static NBDConnState *init_conn_state(BDRVNBDState *s)
+{
+    NBDConnState *cs;
+
+    cs = g_new0(NBDConnState, 1);
+    cs->s = s;
+    qemu_mutex_init(&cs->requests_lock);
+    qemu_co_queue_init(&cs->free_sema);
+    qemu_co_mutex_init(&cs->send_mutex);
+    qemu_co_mutex_init(&cs->receive_mutex);
+
+    return cs;
+}
+
+static int conn_state_connect(BlockDriverState *bs, NBDConnState *cs,
+                              Error **errp)
+{
+    BDRVNBDState *s = cs->s;
+    int ret;
+
+    cs->conn =
+        nbd_client_connection_new(s->saddr, true, s->export,
+                                  s->x_dirty_bitmap, s->tlscreds,
+                                  s->tlshostname);
+
+    if (s->open_timeout) {
+        nbd_client_connection_enable_retry(cs->conn);
+        open_timer_init(cs, qemu_clock_get_ns(QEMU_CLOCK_REALTIME) +
+                        s->open_timeout * NANOSECONDS_PER_SECOND);
+    }
+
+    cs->state = NBD_CLIENT_CONNECTING_WAIT;
+    ret = nbd_do_establish_connection(bs, cs, true, errp);
+    if (ret < 0) {
+        goto fail;
+    }
+
+    /*
+     * The connect attempt is done, so we no longer need this timer.
+     * Delete it, because we do not want it to be around when this node
+     * is drained or closed.
+     */
+    open_timer_del(cs);
+
+    nbd_client_connection_enable_retry(cs->conn);
+
+    return 0;
+
+fail:
+    open_timer_del(cs);
+    return ret;
+}
+
 static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
                     Error **errp)
 {
     int ret;
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
+    size_t i;
 
     s->bs = bs;
 
-    s->conns[0] = g_new0(NBDConnState, 1);
-    s->conns[0]->s = s;
-    qemu_mutex_init(&s->conns[0]->requests_lock);
-    qemu_co_queue_init(&s->conns[0]->free_sema);
-    qemu_co_mutex_init(&s->conns[0]->send_mutex);
-    qemu_co_mutex_init(&s->conns[0]->receive_mutex);
-
     if (!yank_register_instance(BLOCKDEV_YANK_INSTANCE(bs->node_name), errp)) {
         return -EEXIST;
     }
@@ -1965,32 +2017,16 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
         goto fail;
     }
 
-    s->conns[0]->conn =
-        nbd_client_connection_new(s->saddr, true, s->export,
-                                  s->x_dirty_bitmap, s->tlscreds,
-                                  s->tlshostname);
+    /*
+     * Open the first NBD connection.
+     */
+    s->conns[0] = init_conn_state(s);
 
-    if (s->open_timeout) {
-        nbd_client_connection_enable_retry(s->conns[0]->conn);
-        open_timer_init(s->conns[0], qemu_clock_get_ns(QEMU_CLOCK_REALTIME) +
-                        s->open_timeout * NANOSECONDS_PER_SECOND);
-    }
-
-    s->conns[0]->state = NBD_CLIENT_CONNECTING_WAIT;
-    ret = nbd_do_establish_connection(bs, s->conns[0], true, errp);
+    ret = conn_state_connect(bs, s->conns[0], errp);
     if (ret < 0) {
         goto fail;
     }
 
-    /*
-     * The connect attempt is done, so we no longer need this timer.
-     * Delete it, because we do not want it to be around when this node
-     * is drained or closed.
-     */
-    open_timer_del(s->conns[0]);
-
-    nbd_client_connection_enable_retry(s->conns[0]->conn);
-
     /*
      * We set s->multi_conn in nbd_process_options above, but now that
      * we have connected if the server doesn't advertise that it is
@@ -2000,10 +2036,21 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags,
         s->multi_conn = 1;
     }
 
+    /*
+     * Open remaining multi-conn NBD connections (if any).
+     */
+    for (i = 1; i < s->multi_conn; ++i) {
+        s->conns[i] = init_conn_state(s);
+
+        ret = conn_state_connect(bs, s->conns[i], errp);
+        if (ret < 0) {
+            goto fail;
+        }
+    }
+
     return 0;
 
 fail:
-    open_timer_del(s->conns[0]);
     nbd_clear_bdrvstate(bs);
     return ret;
 }
@@ -2054,8 +2101,13 @@ static void nbd_refresh_limits(BlockDriverState *bs, Error **errp)
 static void nbd_close(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
+    size_t i;
 
-    nbd_client_close(s->conns[0]);
+    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+        if (s->conns[i]) {
+            nbd_client_close(s->conns[i]);
+        }
+    }
     nbd_clear_bdrvstate(bs);
 }
 
Signature · 1 line
2.49.0
author
#15 lore raw

PATCH v2 4/4 nbd: Enable multi-conn using round-robin

In reply to Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Enable NBD multi-conn by spreading operations across multiple connections.

(XXX) This uses a naive round-robin approach which could be improved. For example we could look at how many requests are in flight and assign operations to the connections with fewest. Or we could try to estimate (based on size of requests outstanding) the load on each connection. But this implementation doesn't do any of that.

Message-ID: <[email protected]>

---
 block/nbd.c | 67 +++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 49 insertions(+), 18 deletions(-)
block/nbd.c +49 -18
index 19da1a7a1fe..bf5bc57569c 100644
@@ -1264,6 +1264,26 @@ nbd_co_request(NBDConnState *cs, NBDRequest *request,
     return ret ? ret : request_ret;
 }
 
+/*
+ * If multi-conn, choose a connection for this operation.
+ */
+static NBDConnState *choose_connection(BDRVNBDState *s)
+{
+    static size_t next;
+    size_t i;
+
+    if (s->multi_conn <= 1) {
+        return s->conns[0];
+    }
+
+    /* XXX Stupid simple round robin. */
+    i = qatomic_fetch_inc(&next);
+    i %= s->multi_conn;
+
+    assert(s->conns[i] != NULL);
+    return s->conns[i];
+}
+
 static int coroutine_fn GRAPH_RDLOCK
 nbd_client_co_preadv(BlockDriverState *bs, int64_t offset, int64_t bytes,
                      QEMUIOVector *qiov, BdrvRequestFlags flags)
@@ -1276,7 +1296,7 @@ nbd_client_co_preadv(BlockDriverState *bs, int64_t offset, int64_t bytes,
         .from = offset,
         .len = bytes,
     };
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     assert(bytes <= NBD_MAX_BUFFER_SIZE);
 
@@ -1333,7 +1353,7 @@ nbd_client_co_pwritev(BlockDriverState *bs, int64_t offset, int64_t bytes,
         .from = offset,
         .len = bytes,
     };
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     assert(!(cs->info.flags & NBD_FLAG_READ_ONLY));
     if (flags & BDRV_REQ_FUA) {
@@ -1359,7 +1379,7 @@ nbd_client_co_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int64_t bytes,
         .from = offset,
         .len = bytes,
     };
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     /* rely on max_pwrite_zeroes */
     assert(bytes <= UINT32_MAX || cs->info.mode >= NBD_MODE_EXTENDED);
@@ -1391,7 +1411,13 @@ static int coroutine_fn GRAPH_RDLOCK nbd_client_co_flush(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
     NBDRequest request = { .type = NBD_CMD_FLUSH };
-    NBDConnState * const cs = s->conns[0];
+
+    /*
+     * Multi-conn (if used) guarantees that flushing on any connection
+     * flushes caches on all connections, so we can perform this
+     * operation on any.
+     */
+    NBDConnState * const cs = choose_connection(s);
 
     if (!(cs->info.flags & NBD_FLAG_SEND_FLUSH)) {
         return 0;
@@ -1412,7 +1438,7 @@ nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int64_t bytes)
         .from = offset,
         .len = bytes,
     };
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     /* rely on max_pdiscard */
     assert(bytes <= UINT32_MAX || cs->info.mode >= NBD_MODE_EXTENDED);
@@ -1433,7 +1459,7 @@ static int coroutine_fn GRAPH_RDLOCK nbd_client_co_block_status(
     NBDExtent64 extent = { 0 };
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
     Error *local_err = NULL;
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     NBDRequest request = {
         .type = NBD_CMD_BLOCK_STATUS,
@@ -2058,7 +2084,7 @@ fail:
 static void nbd_refresh_limits(BlockDriverState *bs, Error **errp)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
     uint32_t min = cs->info.min_block;
     uint32_t max = MIN_NON_ZERO(NBD_MAX_BUFFER_SIZE, cs->info.max_block);
 
@@ -2124,7 +2150,7 @@ static int coroutine_fn nbd_co_truncate(BlockDriverState *bs, int64_t offset,
                                         BdrvRequestFlags flags, Error **errp)
 {
     BDRVNBDState *s = bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     if (offset != cs->info.size && exact) {
         error_setg(errp, "Cannot resize NBD nodes");
@@ -2207,24 +2233,29 @@ static const char *const nbd_strong_runtime_opts[] = {
 static void nbd_cancel_in_flight(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    size_t i;
+    NBDConnState *cs;
 
-    reconnect_delay_timer_del(cs);
+    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+        cs = s->conns[i];
 
-    qemu_mutex_lock(&cs->requests_lock);
-    if (cs->state == NBD_CLIENT_CONNECTING_WAIT) {
-        cs->state = NBD_CLIENT_CONNECTING_NOWAIT;
+        reconnect_delay_timer_del(cs);
+
+        qemu_mutex_lock(&cs->requests_lock);
+        if (cs->state == NBD_CLIENT_CONNECTING_WAIT) {
+            cs->state = NBD_CLIENT_CONNECTING_NOWAIT;
+        }
+        qemu_mutex_unlock(&cs->requests_lock);
+
+        nbd_co_establish_connection_cancel(cs->conn);
     }
-    qemu_mutex_unlock(&cs->requests_lock);
-
-    nbd_co_establish_connection_cancel(cs->conn);
 }
 
 static void nbd_attach_aio_context(BlockDriverState *bs,
                                    AioContext *new_context)
 {
     BDRVNBDState *s = bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     /* The open_timer is used only during nbd_open() */
     assert(!cs->open_timer);
@@ -2244,7 +2275,7 @@ static void nbd_attach_aio_context(BlockDriverState *bs,
 static void nbd_detach_aio_context(BlockDriverState *bs)
 {
     BDRVNBDState *s = bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    NBDConnState * const cs = choose_connection(s);
 
     assert(!cs->open_timer);
     assert(!cs->reconnect_delay_timer);
Signature · 1 line
2.49.0
#16 lore raw

Re: [PATCH v2 4/4] nbd: Enable multi-conn using round-robin

On Mon, Apr 28, 2025 at 01:46:47PM -0500, Eric Blake wrote:
[...]

This all looks similar to when I posted it before. However I noted a couple of problems ...

Quoted 11 lines

(XXX) The strategy here is very naive. Firstly if you were going to open them, you'd probably want to open them in parallel. Secondly it would make sense to delay opening until multiple parallel requests are seen (perhaps above some threshold), so that simple or shortlived NBD operations do not require multiple connections to be made.

(XXX) This uses a naive round-robin approach which could be improved. For example we could look at how many requests are in flight and assign operations to the connections with fewest. Or we could try to estimate (based on size of requests outstanding) the load on each connection. But this implementation doesn't do any of that.

Plus there was a third rather more fundamental problem that apparently I didn't write about. That is that connections were serialised on a single thread (called from many coroutines). This bottleneck meant that there wasn't very much advantage to multi-conn, compared to what we get in libnbd / nbdcopy.

Are these fixed / planned to be fixed, especially the third?

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
author
#17 lore raw

Re: [PATCH v2 4/4] nbd: Enable multi-conn using round-robin

Quoted 18 lines · Richard W.M. Jones

On Mon, Apr 28, 2025 at 01:46:47PM -0500, Eric Blake wrote:
[...]

This all looks similar to when I posted it before. However I noted a couple of problems ...

Quoted 11 lines

(XXX) The strategy here is very naive. Firstly if you were going to open them, you'd probably want to open them in parallel. Secondly it would make sense to delay opening until multiple parallel requests are seen (perhaps above some threshold), so that simple or shortlived NBD operations do not require multiple connections to be made.

(XXX) This uses a naive round-robin approach which could be improved. For example we could look at how many requests are in flight and assign operations to the connections with fewest. Or we could try to estimate (based on size of requests outstanding) the load on each connection. But this implementation doesn't do any of that.

Quoted 7 lines

Plus there was a third rather more fundamental problem that apparently I didn't write about. That is that connections were serialised on a single thread (called from many coroutines). This bottleneck meant that there wasn't very much advantage to multi-conn, compared to what we get in libnbd / nbdcopy.

Are these fixed / planned to be fixed, especially the third?

That indeed is what I hope to address - to provide additional patches on top of this rebase to make it possible to specify a pool of more than one thread using the recent multiqueue work in the block layer. It's taking me longer than I want to get something that I'm happy with posting.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
#18 lore raw

Re: [PATCH v2 4/4] nbd: Enable multi-conn using round-robin

In reply to Eric Blake
Quoted 32 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Enable NBD multi-conn by spreading operations across multiple connections.

(XXX) This uses a naive round-robin approach which could be improved. For example we could look at how many requests are in flight and assign operations to the connections with fewest. Or we could try to estimate (based on size of requests outstanding) the load on each connection. But this implementation doesn't do any of that.

Message-ID: <[email protected]>

---
 block/nbd.c | 67 +++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 49 insertions(+), 18 deletions(-)
block/nbd.c +2 -1
index 19da1a7a1fe..bf5bc57569c 100644
 
[...]
@@ -2207,24 +2233,29 @@ static const char *const nbd_strong_runtime_opts[] = {
 static void nbd_cancel_in_flight(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    size_t i;
+    NBDConnState *cs;

block/nbd.c static const char *const nbd_strong_runtime_opts[] = { · line 2243

Quoted 10 lines
block/nbd.c +4 -4
 
-    reconnect_delay_timer_del(cs);
+    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+        cs = s->conns[i];
 
-    qemu_mutex_lock(&cs->requests_lock);
-    if (cs->state == NBD_CLIENT_CONNECTING_WAIT) {
-        cs->state = NBD_CLIENT_CONNECTING_NOWAIT;
+        reconnect_delay_timer_del(cs);
+

This code is causing iotests/{185,264,281} to segfault. E.g.:

Quoted 11 lines

(gdb) bt

#0  0x000055bbaec58119 in reconnect_delay_timer_del (cs=0x0) at ../block/nbd.c:205
#1  0x000055bbaec5d8e4 in nbd_cancel_in_flight (bs=0x55bbb1458020) at ../block/nbd.c:2242
#2  0x000055bbaec4ff16 in bdrv_cancel_in_flight (bs=0x55bbb1458020) at ../block/io.c:3737
#3  0x000055bbaec54ec1 in mirror_cancel (job=0x55bbb21ce800, force=true) at ../block/mirror.c:1335
#4  0x000055bbaec18278 in job_cancel_async_locked (job=0x55bbb21ce800, force=true) at ../job.c:893
#5  0x000055bbaec18df2 in job_cancel_locked (job=0x55bbb21ce800, force=true) at ../job.c:1143
#6  0x000055bbaec18ef3 in job_force_cancel_err_locked (job=0x55bbb21ce800, errp=0x7fff44f247a0) at ../job.c:1190
#7  0x000055bbaec192a4 in job_finish_sync_locked (job=0x55bbb21ce800, finish=0x55bbaec18ed2 <job_force_cancel_err_locked>, errp=0x0) at ../job.c:1253
#8  0x000055bbaec18f2e in job_cancel_sync_locked (job=0x55bbb21ce800, force=true) at ../job.c:1196
#9  0x000055bbaec19086 in job_cancel_sync_all () at ../job.c:1214
Quoted 3 lines

#10 0x000055bbaed55177 in qemu_cleanup (status=0) at ../system/runstate.c:949 #11 0x000055bbaedd0aad in qemu_default_main (opaque=0x0) at ../system/main.c:51 #12 0x000055bbaedd0b4f in main (argc=21, argv=0x7fff44f249d8) at ../system/main.c:80

We're dereferencing a pointer to NBDConnState that was never initialized. So it should either be

block/nbd.c

Quoted 2 lines
block/nbd.c +1 -1
-    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+    for (i = 0; i < s->multi_conn; ++i) {

or

block/nbd.c

Quoted 4 lines
block/nbd.c +3 -1
-        cs = s->conns[i];
+        if (s->conns[i]) {
+            cs = s->conns[i];
+        ...

[...]

Andrey

author
#19 lore raw

Re: [PATCH v2 4/4] nbd: Enable multi-conn using round-robin

Quoted 61 lines · Andrey Drobyshev
Quoted 32 lines · Eric Blake

From: "Richard W.M. Jones" <[email protected]>

Enable NBD multi-conn by spreading operations across multiple connections.

(XXX) This uses a naive round-robin approach which could be improved. For example we could look at how many requests are in flight and assign operations to the connections with fewest. Or we could try to estimate (based on size of requests outstanding) the load on each connection. But this implementation doesn't do any of that.

Message-ID: <[email protected]>

---
 block/nbd.c | 67 +++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 49 insertions(+), 18 deletions(-)
block/nbd.c +2 -1
index 19da1a7a1fe..bf5bc57569c 100644
 
[...]
@@ -2207,24 +2233,29 @@ static const char *const nbd_strong_runtime_opts[] = {
 static void nbd_cancel_in_flight(BlockDriverState *bs)
 {
     BDRVNBDState *s = (BDRVNBDState *)bs->opaque;
-    NBDConnState * const cs = s->conns[0];
+    size_t i;
+    NBDConnState *cs;

block/nbd.c static const char *const nbd_strong_runtime_opts[] = { · line 2243

Quoted 10 lines
block/nbd.c +4 -4
 
-    reconnect_delay_timer_del(cs);
+    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+        cs = s->conns[i];
 
-    qemu_mutex_lock(&cs->requests_lock);
-    if (cs->state == NBD_CLIENT_CONNECTING_WAIT) {
-        cs->state = NBD_CLIENT_CONNECTING_NOWAIT;
+        reconnect_delay_timer_del(cs);
+

This code is causing iotests/{185,264,281} to segfault. E.g.:

Quoted 11 lines

(gdb) bt

#0  0x000055bbaec58119 in reconnect_delay_timer_del (cs=0x0) at ../block/nbd.c:205
#1  0x000055bbaec5d8e4 in nbd_cancel_in_flight (bs=0x55bbb1458020) at ../block/nbd.c:2242
#2  0x000055bbaec4ff16 in bdrv_cancel_in_flight (bs=0x55bbb1458020) at ../block/io.c:3737
#3  0x000055bbaec54ec1 in mirror_cancel (job=0x55bbb21ce800, force=true) at ../block/mirror.c:1335
#4  0x000055bbaec18278 in job_cancel_async_locked (job=0x55bbb21ce800, force=true) at ../job.c:893
#5  0x000055bbaec18df2 in job_cancel_locked (job=0x55bbb21ce800, force=true) at ../job.c:1143
#6  0x000055bbaec18ef3 in job_force_cancel_err_locked (job=0x55bbb21ce800, errp=0x7fff44f247a0) at ../job.c:1190
#7  0x000055bbaec192a4 in job_finish_sync_locked (job=0x55bbb21ce800, finish=0x55bbaec18ed2 <job_force_cancel_err_locked>, errp=0x0) at ../job.c:1253
#8  0x000055bbaec18f2e in job_cancel_sync_locked (job=0x55bbb21ce800, force=true) at ../job.c:1196
#9  0x000055bbaec19086 in job_cancel_sync_all () at ../job.c:1214
Quoted 3 lines

#10 0x000055bbaed55177 in qemu_cleanup (status=0) at ../system/runstate.c:949 #11 0x000055bbaedd0aad in qemu_default_main (opaque=0x0) at ../system/main.c:51 #12 0x000055bbaedd0b4f in main (argc=21, argv=0x7fff44f249d8) at ../system/main.c:80

Quoted 5 lines

We're dereferencing a pointer to NBDConnState that was never initialized. So it should either be

Quoted 2 lines
block/nbd.c +1 -1
-    for (i = 0; i < MAX_MULTI_CONN; ++i) {
+    for (i = 0; i < s->multi_conn; ++i) {

Thanks for pointing it out; I'll fix that.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
#20 lore raw
In reply to Eric Blake
Quoted 14 lines · Eric Blake

This is a rebase of patches that Rich Jones first posted in 2023:
https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03320.html

I'm still leaving the series in an RFC state while I'm trying to improve the code to better play with all of the multi-queue changes in the block layer in the meantime. In my own testing, the patches as presented here are not making any noticeable difference in qemu-img convert to a local destination file (multi-conn=1 was not much different than multi-conn=4, although I did validate that multiple sockets were in use and the round robin code was working). Other benchmarks did show improvements, such as qemu convert targeting an nbdkit server on a curl backend getting a 20% boost when I ramped multi-conn from 1 to 4.

Quoted 2 lines

I have more patches on top of these to post to the list once I can get benchmark numbers that make more sense.

I'm curious if you are benchmarking UNIX or TCP sockets ?

If UNIX sockets, then that recent patch about increasing socket buffer size for UNIX sockets rings alarm bells. ie, I wonder if the artificially low UNIX socket buf size on Linux is preventing you seeing any significant multi-conn benefits ?

IOW, add that socket buf size patch on top of this series, before benchmarking again ?

Quoted 11 lines

Richard W.M. Jones (4):
nbd: Add multi-conn option
nbd: Split out block device state from underlying NBD connections
nbd: Open multiple NBD connections if multi-conn is set
nbd: Enable multi-conn using round-robin

 qapi/block-core.json |   8 +-
 block/coroutines.h   |   5 +-
 block/nbd.c          | 796 +++++++++++++++++++++++++------------------
 3 files changed, 479 insertions(+), 330 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#21 lore raw
In reply to Eric Blake
Quoted 23 lines

This is a rebase of patches that Rich Jones first posted in 2023:
https://lists.gnu.org/archive/html/qemu-devel/2023-03/msg03320.html

I'm still leaving the series in an RFC state while I'm trying to improve the code to better play with all of the multi-queue changes in the block layer in the meantime. In my own testing, the patches as presented here are not making any noticeable difference in qemu-img convert to a local destination file (multi-conn=1 was not much different than multi-conn=4, although I did validate that multiple sockets were in use and the round robin code was working). Other benchmarks did show improvements, such as qemu convert targeting an nbdkit server on a curl backend getting a 20% boost when I ramped multi-conn from 1 to 4.

I have more patches on top of these to post to the list once I can get benchmark numbers that make more sense.

Richard W.M. Jones (4):
nbd: Add multi-conn option
nbd: Split out block device state from underlying NBD connections
nbd: Open multiple NBD connections if multi-conn is set
nbd: Enable multi-conn using round-robin

Quoted 4 lines
 qapi/block-core.json |   8 +-
 block/coroutines.h   |   5 +-
 block/nbd.c          | 796 +++++++++++++++++++++++++------------------
 3 files changed, 479 insertions(+), 330 deletions(-)

This series makes my day.

I have a request from our partners to implement exactly this feature to improve the backup restoration performance in their scenario. The case is the following

+---------+  +----------+              +----------+  +------+
| Storage |->|IPsec gate|...Internet...|IPsec gate|->|Target|
+---------+  +----------+              +----------+  +------+

In this case the performance could be limited not by the Internet of something else, but with the raw CPU performance in VMs which are doing encryption/decryption.

I was specifically requested to implement multi-conn inside QEMU client, which was used to run the system from the backup while restoration is performed.

This makes a lot of sense for me even if there is no direct benefit without bottleneck in the middle.

Thank you in advance,
Den