Skip to contentSkip to conversation

nbd: MULTI_CONN for shared writable exports

PATCH Superseded Eric Blake 4 participants 15 messages Mar 14, 2022 – May 3, 2022

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v3 cover nbd: MULTI_CONN for shared writable exports

v2 was here:
https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg03314.html

Since then:
- expose the knob through 'qemu-nbd -m on|off|auto'
- reuse QAPI OnOffAuto type instead of rolling my own
- rewrite the iotest from bash into python (thanks: Nir, Vladimir)
- split out easy preliminary fixes (patches 1 and 2 are new)

Given that this is a new feature, it probably needs to be 7.1 material.

Eric Blake (3):
docs: Consistent typography for options of qemu-nbd
qemu-nbd: Pass max connections to blockdev layer nbd/server: Allow MULTI_CONN for shared writable exports

 docs/interop/nbd.txt                       |   1 +
 docs/tools/qemu-nbd.rst                    |  26 ++--
 qapi/block-export.json                     |  19 ++-
 include/block/nbd.h                        |   5 +-
 blockdev-nbd.c                             |  13 +-
 nbd/server.c                               |  27 +++-
 qemu-nbd.c                                 |  22 ++-
 MAINTAINERS                                |   1 +
 tests/qemu-iotests/tests/nbd-multiconn     | 157 +++++++++++++++++++++
 tests/qemu-iotests/tests/nbd-multiconn.out |   5 +
 10 files changed, 252 insertions(+), 24 deletions(-)
 create mode 100755 tests/qemu-iotests/tests/nbd-multiconn
 create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
Signature · 1 line
2.35.1
author
#1 lore raw

PATCH v3 1/3 docs: Consistent typography for options of qemu-nbd

Prefer the :option:--name form when cross-referencing other options from the qemu-nbd documentation.

---
 docs/tools/qemu-nbd.rst | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
docs/tools/qemu-nbd.rst +6 -6
index 2b8c90c35498..4c950f61998e 100644
@@ -27,7 +27,7 @@ Options
 .. program:: qemu-nbd
 
 *filename* is a disk image filename, or a set of block
-driver options if ``--image-opts`` is specified.
+driver options if :option:`--image-opts` is specified.
 
 *dev* is an NBD device.
 
@@ -165,9 +165,9 @@ driver options if ``--image-opts`` is specified.
 .. option:: --tls-creds=ID
 
   Enable mandatory TLS encryption for the server by setting the ID
-  of the TLS credentials object previously created with the --object
-  option; or provide the credentials needed for connecting as a client
-  in list mode.
+  of the TLS credentials object previously created with the
+  :option:`--object` option; or provide the credentials needed for
+  connecting as a client in list mode.
 
 .. option:: --tls-hostname=hostname
 
@@ -176,11 +176,11 @@ driver options if ``--image-opts`` is specified.
   against information in the server provided certificate. Sometimes
   it might be required to override the hostname used to perform this
   check. For example, if the NBD client is using a tunnel from localhost
-  to connect to the remote server, the `--tls-hostname` option should
+  to connect to the remote server, the :option:`--tls-hostname` option should
   be used to set the officially expected hostname of the remote NBD
   server. This can also be used if accessing NBD over a UNIX socket
   where there is no inherent hostname available. This is only permitted
-  when acting as a NBD client with the `--list` option.
+  when acting as a NBD client with the :option:`--list` option.
 
 .. option:: --fork
 
Signature · 1 line
2.35.1
Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCH v3 1/3] docs: Consistent typography for options of qemu-nbd

Quoted 7 lines · Eric Blake

Prefer the :option:--name form when cross-referencing other options from the qemu-nbd documentation.

---
 docs/tools/qemu-nbd.rst | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
author
#3 lore raw

PATCH v3 2/3 qemu-nbd: Pass max connections to blockdev layer

In reply to Eric Blake

The next patch wants to adjust whether the NBD server code advertises MULTI_CONN based on whether it is known if the server limits to exactly one client. For a server started by QMP, this information is obtained through nbd_server_start (which can support more than one export); but for qemu-nbd (which supports exactly one export), it is controlled only by the command-line option -e/--shared. Since we already have a hook function used by qemu-nbd, it's easiest to just alter its signature to fit our needs.

---
 include/block/nbd.h | 2 +-
 blockdev-nbd.c      | 8 ++++----
 qemu-nbd.c          | 2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)
include/block/nbd.h +1 -1
index a98eb665da04..c5a29ce1c61b 100644
@@ -344,7 +344,7 @@ void nbd_client_new(QIOChannelSocket *sioc,
 void nbd_client_get(NBDClient *client);
 void nbd_client_put(NBDClient *client);
 
-void nbd_server_is_qemu_nbd(bool value);
+void nbd_server_is_qemu_nbd(int max_connections);
 bool nbd_server_is_running(void);
 void nbd_server_start(SocketAddress *addr, const char *tls_creds,
                       const char *tls_authz, uint32_t max_connections,
blockdev-nbd.c +4 -4
index 9840d25a8298..add41a23af43 100644
@@ -30,18 +30,18 @@ typedef struct NBDServerData {
 } NBDServerData;
 
 static NBDServerData *nbd_server;
-static bool is_qemu_nbd;
+static int qemu_nbd_connections = -1; /* Non-negative if this is qemu-nbd */
 
 static void nbd_update_server_watch(NBDServerData *s);
 
-void nbd_server_is_qemu_nbd(bool value)
+void nbd_server_is_qemu_nbd(int max_connections)
 {
-    is_qemu_nbd = value;
+    qemu_nbd_connections = max_connections;
 }
 
 bool nbd_server_is_running(void)
 {
-    return nbd_server || is_qemu_nbd;
+    return nbd_server || qemu_nbd_connections >= 0;
 }
 
 static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
qemu-nbd.c +1 -1
index 713e7557a9eb..8c25ae93df83 100644
@@ -1087,7 +1087,7 @@ int main(int argc, char **argv)
 
     bs->detect_zeroes = detect_zeroes;
 
-    nbd_server_is_qemu_nbd(true);
+    nbd_server_is_qemu_nbd(shared);
 
     export_opts = g_new(BlockExportOptions, 1);
     *export_opts = (BlockExportOptions) {
Signature · 1 line
2.35.1
author
#4 lore raw

PATCH v3 3/3 nbd/server: Allow MULTI_CONN for shared writable exports

In reply to Eric Blake

According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We satisfy these conditions in qemu when our block layer is backed by the local filesystem (by virtue of the semantics of fdatasync(), and the fact that qemu itself is not buffering writes beyond flushes). It is harder to state whether we satisfy these conditions for network-based protocols, so the safest course of action is to allow users to opt-in to advertising multi-conn. We may later tweak defaults to advertise by default when the block layer can confirm that the underlying protocol driver is cache consistent between multiple writers, but for now, this at least allows savvy users (such as virt-v2v or nbdcopy) to explicitly start qemu-nbd (new -m command-line option) or qemu-storage-daemon (new qapi field 'multi-conn') with multi-conn advertisement in a known-safe setup where the client end can then benefit from parallel clients.

Note, however, that we don't want to advertise MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test.

---
 docs/interop/nbd.txt                       |   1 +
 docs/tools/qemu-nbd.rst                    |  14 +-
 qapi/block-export.json                     |  19 ++-
 include/block/nbd.h                        |   3 +-
 blockdev-nbd.c                             |   5 +
 nbd/server.c                               |  27 +++-
 qemu-nbd.c                                 |  20 ++-
 MAINTAINERS                                |   1 +
 tests/qemu-iotests/tests/nbd-multiconn     | 157 +++++++++++++++++++++
 tests/qemu-iotests/tests/nbd-multiconn.out |   5 +
 10 files changed, 240 insertions(+), 12 deletions(-)
 create mode 100755 tests/qemu-iotests/tests/nbd-multiconn
 create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
docs/interop/nbd.txt +1 -0
index bdb0f2a41aca..6c99070b99c8 100644
@@ -68,3 +68,4 @@ NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CACHE
 * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports,
 NBD_CMD_FLAG_FAST_ZERO
 * 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth"
+* 7.0: NBD_FLAG_CAN_MULTI_CONN for shareable writable exports
docs/tools/qemu-nbd.rst +12 -2
index 4c950f61998e..229414e3cdc7 100644
@@ -139,8 +139,18 @@ driver options if :option:`--image-opts` is specified.
 .. option:: -e, --shared=NUM
 
   Allow up to *NUM* clients to share the device (default
-  ``1``), 0 for unlimited. Safe for readers, but for now,
-  consistency is not guaranteed between multiple writers.
+  ``1``), 0 for unlimited.
+
+.. option:: -m, --multi-conn=MODE
+
+  When multiple clients are allowed at once (via :option:`--shared`),
+  controls whether the NBD protocol MULTI_CONN bit is advertised to
+  clients.  This defaults to ``auto``, which advertises the bit for
+  readonly connections (:option:`--read-only`) but not for writeable
+  connections; it can also be set to ``on`` or ``off`` to override the
+  default.  Advertising this bit informs clients whether a flush from
+  one client has guaranteed consistency to subsequent access from a
+  parallel client.
 
 .. option:: -t, --persistent
 
qapi/block-export.json +16 -3
index f183522d0d2c..d9d38620156f 100644
@@ -21,7 +21,9 @@
 #             recreated on the fly while the NBD server is active.
 #             If missing, it will default to denying access (since 4.0).
 # @max-connections: The maximum number of connections to allow at the same
-#                   time, 0 for unlimited. (since 5.2; default: 0)
+#                   time, 0 for unlimited. Setting this to 1 also stops
+#                   the server from advertising multiple client support
+#                   (since 5.2; default: 0)
 #
 # Since: 4.2
 ##
@@ -50,7 +52,9 @@
 #             recreated on the fly while the NBD server is active.
 #             If missing, it will default to denying access (since 4.0).
 # @max-connections: The maximum number of connections to allow at the same
-#                   time, 0 for unlimited. (since 5.2; default: 0)
+#                   time, 0 for unlimited. Setting this to 1 also stops
+#                   the server from advertising multiple client support
+#                   (since 5.2; default: 0).
 #
 # Returns: error if the server is already running.
 #
@@ -95,11 +99,20 @@
 #                    the metadata context name "qemu:allocation-depth" to
 #                    inspect allocation details. (since 5.2)
 #
+# @multi-conn: Controls whether multiple client support is advertised, if the
+#              server's max-connections is not 1. "auto" advertises support
+#              for read-only clients, but not for writeable (although this
+#              may be improved in the future for writeable devices that are
+#              known to be cache-consistent, such as the raw file driver),
+#              while "on" and "off" explicitly set the bit (default auto,
+#              since 7.0).
+#
 # Since: 5.2
 ##
 { 'struct': 'BlockExportOptionsNbd',
   'base': 'BlockExportOptionsNbdBase',
-  'data': { '*bitmaps': ['str'], '*allocation-depth': 'bool' } }
+  'data': { '*bitmaps': ['str'], '*allocation-depth': 'bool',
+            '*multi-conn': 'OnOffAuto' } }
 
 ##
 # @BlockExportOptionsVhostUserBlk:
include/block/nbd.h +2 -1
index c5a29ce1c61b..c74b7a9d2e6e 100644
@@ -1,5 +1,5 @@
 /*
- *  Copyright (C) 2016-2020 Red Hat, Inc.
+ *  Copyright (C) 2016-2022 Red Hat, Inc.
  *  Copyright (C) 2005  Anthony Liguori <[email protected]>
  *
  *  Network Block Device
@@ -346,6 +346,7 @@ void nbd_client_put(NBDClient *client);
 
 void nbd_server_is_qemu_nbd(int max_connections);
 bool nbd_server_is_running(void);
+int nbd_server_max_connections(void);
 void nbd_server_start(SocketAddress *addr, const char *tls_creds,
                       const char *tls_authz, uint32_t max_connections,
                       Error **errp);
blockdev-nbd.c +5 -0
index add41a23af43..15ad7b8dbfd5 100644
@@ -44,6 +44,11 @@ bool nbd_server_is_running(void)
     return nbd_server || qemu_nbd_connections >= 0;
 }
 
+int nbd_server_max_connections(void)
+{
+    return nbd_server ? nbd_server->max_connections : -1;
+}
+
 static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
 {
     nbd_client_put(client);
nbd/server.c +22 -5
index 5da884c2fc35..2cfdd56cda19 100644
@@ -1,5 +1,5 @@
 /*
- *  Copyright (C) 2016-2021 Red Hat, Inc.
+ *  Copyright (C) 2016-2022 Red Hat, Inc.
  *  Copyright (C) 2005  Anthony Liguori <[email protected]>
  *
  *  Network Block Device Server Side
@@ -1642,7 +1642,7 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     int64_t size;
     uint64_t perm, shared_perm;
     bool readonly = !exp_args->writable;
-    bool shared = !exp_args->writable;
+    bool multi_conn;
     strList *bitmaps;
     size_t i;
     int ret;
@@ -1680,6 +1680,23 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
         return size;
     }
 
+    /*
+     * Determine whether to advertise multi-conn.  Default is auto,
+     * which resolves to on for read-only and off for writable.  But
+     * if the server has max-connections 1, that forces the flag off.
+     */
+    if (!arg->has_multi_conn) {
+        arg->multi_conn = ON_OFF_AUTO_AUTO;
+    }
+    if (nbd_server_max_connections() == 1) {
+        arg->multi_conn = ON_OFF_AUTO_OFF;
+    }
+    if (arg->multi_conn == ON_OFF_AUTO_AUTO) {
+        multi_conn = readonly;
+    } else {
+        multi_conn = arg->multi_conn == ON_OFF_AUTO_ON;
+    }
+
     /* Don't allow resize while the NBD server is running, otherwise we don't
      * care what happens with the node. */
     blk_get_perm(blk, &perm, &shared_perm);
@@ -1693,11 +1710,11 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     exp->description = g_strdup(arg->description);
     exp->nbdflags = (NBD_FLAG_HAS_FLAGS | NBD_FLAG_SEND_FLUSH |
                      NBD_FLAG_SEND_FUA | NBD_FLAG_SEND_CACHE);
+    if (multi_conn) {
+        exp->nbdflags |= NBD_FLAG_CAN_MULTI_CONN;
+    }
     if (readonly) {
         exp->nbdflags |= NBD_FLAG_READ_ONLY;
-        if (shared) {
-            exp->nbdflags |= NBD_FLAG_CAN_MULTI_CONN;
-        }
     } else {
         exp->nbdflags |= (NBD_FLAG_SEND_TRIM | NBD_FLAG_SEND_WRITE_ZEROES |
                           NBD_FLAG_SEND_FAST_ZERO);
qemu-nbd.c +19 -1
index 8c25ae93df83..ddeb2fcf7f40 100644
@@ -70,6 +70,7 @@
 #define QEMU_NBD_OPT_PID_FILE      265
 #define QEMU_NBD_OPT_SELINUX_LABEL 266
 #define QEMU_NBD_OPT_TLSHOSTNAME   267
+#define QEMU_NBD_OPT_MULTI_CONN    268
 
 #define MBR_SIZE 512
 
@@ -79,6 +80,7 @@ static SocketAddress *saddr;
 static int persistent = 0;
 static enum { RUNNING, TERMINATE, TERMINATED } state;
 static int shared = 1;
+static OnOffAuto multi_conn = ON_OFF_AUTO_AUTO;
 static int nb_fds;
 static QIONetListener *server;
 static QCryptoTLSCreds *tlscreds;
@@ -100,6 +102,8 @@ static void usage(const char *name)
 "  -k, --socket=PATH         path to the unix socket\n"
 "                            (default '"SOCKET_PATH"')\n"
 "  -e, --shared=NUM          device can be shared by NUM clients (default '1')\n"
+"  -m, --multi-conn=MODE     control multi-conn advertisement with -e, MODE is\n"
+"                            'on', 'off', or 'auto' (default based on -r)\n"
 "  -t, --persistent          don't exit on the last connection\n"
 "  -v, --verbose             display extra debugging information\n"
 "  -x, --export-name=NAME    expose export by name (default is empty string)\n"
@@ -513,7 +517,7 @@ int main(int argc, char **argv)
     char *device = NULL;
     QemuOpts *sn_opts = NULL;
     const char *sn_id_or_name = NULL;
-    const char *sopt = "hVb:o:p:rsnc:dvk:e:f:tl:x:T:D:AB:L";
+    const char *sopt = "hVb:o:p:rsnc:dvk:e:m:f:tl:x:T:D:AB:L";
     struct option lopt[] = {
         { "help", no_argument, NULL, 'h' },
         { "version", no_argument, NULL, 'V' },
@@ -536,6 +540,7 @@ int main(int argc, char **argv)
         { "detect-zeroes", required_argument, NULL,
           QEMU_NBD_OPT_DETECT_ZEROES },
         { "shared", required_argument, NULL, 'e' },
+        { "multi-conn", required_argument, NULL, 'm' },
         { "format", required_argument, NULL, 'f' },
         { "persistent", no_argument, NULL, 't' },
         { "verbose", no_argument, NULL, 'v' },
@@ -709,6 +714,17 @@ int main(int argc, char **argv)
                 exit(EXIT_FAILURE);
             }
             break;
+        case 'm':
+        {
+            Error *err = NULL;
+            multi_conn = qapi_enum_parse(&OnOffAuto_lookup, optarg,
+                                         ON_OFF_AUTO_AUTO, &err);
+            if (err) {
+                error_report_err(err);
+                exit(EXIT_FAILURE);
+            }
+            break;
+        }
         case 'f':
             fmt = optarg;
             break;
@@ -1107,6 +1123,8 @@ int main(int argc, char **argv)
             .bitmaps              = bitmaps,
             .has_allocation_depth = alloc_depth,
             .allocation_depth     = alloc_depth,
+            .has_multi_conn       = true,
+            .multi_conn           = multi_conn,
         },
     };
     blk_exp_add(export_opts, &error_fatal);
MAINTAINERS +1 -0
index f2e9ce1da2ac..678c34e8077f 100644
@@ -3362,6 +3362,7 @@ F: qemu-nbd.*
 F: blockdev-nbd.c
 F: docs/interop/nbd.txt
 F: docs/tools/qemu-nbd.rst
+F: tests/qemu-iotests/tests/*nbd*
 T: git https://repo.or.cz/qemu/ericb.git nbd
 T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd
 
tests/qemu-iotests/tests/nbd-multiconn new file +157 -0

157 lines not on this page · raw message

tests/qemu-iotests/tests/nbd-multiconn.out new file +5 -0
new file mode 100644
index 000000000000..89968f35d783
@@ -0,0 +1,5 @@
+....
+----------------------------------------------------------------------
+Ran 4 tests
+
+OK
Signature · 1 line
2.35.1
#5 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

The patches seem OK to me, but I don't really know enough about the internals of qemu-nbd to give a line-by-line review. I did however build and test qemu-nbd with the patches:

  $ ./build/qemu-nbd /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false
  $ ./build/qemu-nbd -e 2 /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

^^^ Is this expected? It also happens with -e 0.

  $ ./build/qemu-nbd -e 2 -m on /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: true
  $ ./build/qemu-nbd -e 2 -m off /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
author
#6 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Quoted 10 lines · Richard W.M. Jones

The patches seem OK to me, but I don't really know enough about the internals of qemu-nbd to give a line-by-line review. I did however build and test qemu-nbd with the patches:

  $ ./build/qemu-nbd /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false
Quoted 6 lines
  $ ./build/qemu-nbd -e 2 /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

^^^ Is this expected? It also happens with -e 0.

Yes, because qemu-nbd defaults to read-write connections, but to be conservative, this patch defaults '-m auto' to NOT advertise multi-conn for read-write; you need to be explicit:

Quoted 6 lines

  $ ./build/qemu-nbd -e 2 -m on /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: true

either with '-m on' as you did here, or with

build/qemu-nbd -r -e 2 /var/tmp/test.qcow2

where the '-m auto' default exposes multi-conn for a readonly client.

Quoted 9 lines

  $ ./build/qemu-nbd -e 2 -m off /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

Rich.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
author
#7 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Quoted 29 lines · Eric Blake
Quoted 10 lines · Richard W.M. Jones

The patches seem OK to me, but I don't really know enough about the internals of qemu-nbd to give a line-by-line review. I did however build and test qemu-nbd with the patches:

  $ ./build/qemu-nbd /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false
Quoted 6 lines
  $ ./build/qemu-nbd -e 2 /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

^^^ Is this expected? It also happens with -e 0.

Yes, because qemu-nbd defaults to read-write connections, but to be conservative, this patch defaults '-m auto' to NOT advertise multi-conn for read-write; you need to be explicit:

Quoted 6 lines

  $ ./build/qemu-nbd -e 2 -m on /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: true
Quoted 5 lines

either with '-m on' as you did here, or with

build/qemu-nbd -r -e 2 /var/tmp/test.qcow2

where the '-m auto' default exposes multi-conn for a readonly client.

I hit send prematurely - one more thought I wanted to make clear. For _this_ series, the behavior of '-m auto' depends solely on readonly vs. read-write; that being the most conservative choice of preserving pre-existing qemu-nbd behavior (that is, if you don't use -m, the only change in behavior should be the fact that --help output now lists -m).

But in future versions of qemu, we might be able to improve '-m auto' to also take into consideration whether the backing image of a read-write device is known-cache-consistent (such as a local file system), where we can then manage to default to advertising multi-conn for those writable images, while still avoiding the advertisement when exposing other devices such as network-mounted devices where we are less confident on whether there are sufficient cache consistency guarantees. Use of explicit '-m off' or '-m on' gives guaranteed results no matter the qemu version, so it is only explicit (or implied) '-m auto' where we might get smarter defaults over time.

Thus testing whether advertising multi-conn makes a difference in other tools like nbdcopy thus requires checking if qemu-nbd is new enough to support -m, and then being explicit that we are opting in to using it.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#8 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Quoted 54 lines · Eric Blake
Quoted 29 lines · Eric Blake
Quoted 10 lines · Richard W.M. Jones

The patches seem OK to me, but I don't really know enough about the internals of qemu-nbd to give a line-by-line review. I did however build and test qemu-nbd with the patches:

  $ ./build/qemu-nbd /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false
Quoted 6 lines
  $ ./build/qemu-nbd -e 2 /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: false

^^^ Is this expected? It also happens with -e 0.

Yes, because qemu-nbd defaults to read-write connections, but to be conservative, this patch defaults '-m auto' to NOT advertise multi-conn for read-write; you need to be explicit:

Quoted 6 lines

  $ ./build/qemu-nbd -e 2 -m on /var/tmp/test.qcow2 
  $ nbdinfo nbd://localhost
  ...
	can_multi_conn: true
Quoted 5 lines

either with '-m on' as you did here, or with

build/qemu-nbd -r -e 2 /var/tmp/test.qcow2

where the '-m auto' default exposes multi-conn for a readonly client.

I hit send prematurely - one more thought I wanted to make clear. For _this_ series, the behavior of '-m auto' depends solely on readonly vs. read-write; that being the most conservative choice of preserving pre-existing qemu-nbd behavior (that is, if you don't use -m, the only change in behavior should be the fact that --help output now lists -m).

But in future versions of qemu, we might be able to improve '-m auto' to also take into consideration whether the backing image of a read-write device is known-cache-consistent (such as a local file system), where we can then manage to default to advertising multi-conn for those writable images, while still avoiding the advertisement when exposing other devices such as network-mounted devices where we are less confident on whether there are sufficient cache consistency guarantees. Use of explicit '-m off' or '-m on' gives guaranteed results no matter the qemu version, so it is only explicit (or implied) '-m auto' where we might get smarter defaults over time.

Quoted 4 lines

Thus testing whether advertising multi-conn makes a difference in other tools like nbdcopy thus requires checking if qemu-nbd is new enough to support -m, and then being explicit that we are opting in to using it.

I see. The commit message of patch 3 confused me because superficially it seems to say that multi-conn is safe (and therefore I assumed safe == enabled) when backed by a local filesystem. Could the commit message be improved to list the default for common combinations of flags?

Rich.

Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
#9 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

In reply to Eric Blake

Am 14.03.2022 um 21:38 hat Eric Blake geschrieben:

Quoted 11 lines

According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We satisfy these conditions in qemu when our block layer is backed by the local filesystem (by virtue of the semantics of fdatasync(), and the fact that qemu itself is not buffering writes beyond flushes). It is harder to state whether we satisfy these conditions for network-based protocols, so the safest course of action is to allow users to opt-in to advertising multi-conn.

Do you have an example of how this could be unsafe?

As I understand it, the NBD server has a single BlockBackend and therefore is a single client for the backend, be it file-posix or any network-based protocol. It doesn't really make a difference for the storage from how many different NBD clients the requests are coming.

I would have expected that cache coherency of the protocol level driver would only matter if you had two QEMU processes accessing the same file concurrently.

In fact, I don't think we even need the flush restriction from the NBD spec. All clients see the same state (that of the NBD server BlockBackend) even without anyone issuing any flush. The flush is only needed to make sure that cached data is written to the backing storage when writeback caches are involved.

Please correct me if I'm misunderstanding something here.

Quoted 27 lines

We may later tweak defaults to advertise
by default when the block layer can confirm that the underlying protocol driver is cache consistent between multiple writers, but for now, this at least allows savvy users (such as virt-v2v or nbdcopy) to explicitly start qemu-nbd (new -m command-line option) or qemu-storage-daemon (new qapi field 'multi-conn') with multi-conn advertisement in a known-safe setup where the client end can then benefit from parallel clients.

Note, however, that we don't want to advertise MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test.

qemu-nbd.c +0 -0
@@ -709,6 +714,17 @@ int main(int argc, char **argv)
                 exit(EXIT_FAILURE);
             }

qemu-nbd.c int main(int argc, char **argv) · line 725

Quoted 10 lines
qemu-nbd.c +9 -0
             break;
+        case 'm':
+        {
+            Error *err = NULL;
+            multi_conn = qapi_enum_parse(&OnOffAuto_lookup, optarg,
+                                         ON_OFF_AUTO_AUTO, &err);
+            if (err) {
+                error_report_err(err);
+                exit(EXIT_FAILURE);
+            }

I think this is the same as passing &error_fatal.

Quoted 25 lines
qemu-nbd.c +2 -0
+            break;
+        }
         case 'f':
             fmt = optarg;
             break;
tests/qemu-iotests/tests/nbd-multiconn new file +14 -0
new file mode 100755
index 000000000000..7d1179b33b05
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+# group: rw auto quick
+#
+# Test cases for NBD multi-conn advertisement
+#
+# Copyright (C) 2022 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of

tests/qemu-iotests/tests/nbd-multiconn line 24

Quoted 10 lines
tests/qemu-iotests/tests/nbd-multiconn new file +10 -0
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+from contextlib import contextmanager
+import iotests
+from iotests import qemu_img_create, qemu_io_silent

qemu_io_silent() doesn't exist any more, commit 72cfb937 removed it.

Kevin

author
#10 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Quoted 13 lines · Kevin Wolf

Am 14.03.2022 um 21:38 hat Eric Blake geschrieben:

Quoted 11 lines

According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We satisfy these conditions in qemu when our block layer is backed by the local filesystem (by virtue of the semantics of fdatasync(), and the fact that qemu itself is not buffering writes beyond flushes). It is harder to state whether we satisfy these conditions for network-based protocols, so the safest course of action is to allow users to opt-in to advertising multi-conn.

Quoted 1 line

Do you have an example of how this could be unsafe?

Nothing direct. I tried to turn this on unconditionally in an earlier version, and we waffled about whether we could prove that network block backends (such as gluster) provide us the safety that the NBD spec demands:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00038.html
https://lists.gnu.org/archive/html/qemu-devel/2021-10/msg06744.html

Quoted 9 lines

As I understand it, the NBD server has a single BlockBackend and therefore is a single client for the backend, be it file-posix or any network-based protocol. It doesn't really make a difference for the storage from how many different NBD clients the requests are coming.

I would have expected that cache coherency of the protocol level driver would only matter if you had two QEMU processes accessing the same file concurrently.

Or a multi-pathed connection to network storage, where one QEMU process accesses the network device, but those accesses may round-robin which server they reach, and where any caching at an individual server may be inconsistent with what is seen on another server unless flushing is used to force the round-robin access to synchronize between the multi-path views.

Quoted 8 lines

In fact, I don't think we even need the flush restriction from the NBD spec. All clients see the same state (that of the NBD server BlockBackend) even without anyone issuing any flush. The flush is only needed to make sure that cached data is written to the backing storage when writeback caches are involved.

Please correct me if I'm misunderstanding something here.

Likewise me, if I'm being overly cautious.

I can certainly write a simpler v4 that just always advertises MULTI_CONN if we allow more than one client, without any knob to override it; it's just that it is harder to write a commit message justifying why I think it is safe to do so.

Quoted 25 lines
Quoted 24 lines

We may later tweak defaults to advertise
by default when the block layer can confirm that the underlying protocol driver is cache consistent between multiple writers, but for now, this at least allows savvy users (such as virt-v2v or nbdcopy) to explicitly start qemu-nbd (new -m command-line option) or qemu-storage-daemon (new qapi field 'multi-conn') with multi-conn advertisement in a known-safe setup where the client end can then benefit from parallel clients.

Note, however, that we don't want to advertise MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test.

Quoted 15 lines

qemu-nbd.c int main(int argc, char **argv) · line 725

Quoted 13 lines
qemu-nbd.c +9 -0
@@ -709,6 +714,17 @@ int main(int argc, char **argv)
                 exit(EXIT_FAILURE);
             }
             break;
+        case 'm':
+        {
+            Error *err = NULL;
+            multi_conn = qapi_enum_parse(&OnOffAuto_lookup, optarg,
+                                         ON_OFF_AUTO_AUTO, &err);
+            if (err) {
+                error_report_err(err);
+                exit(EXIT_FAILURE);
+            }

I think this is the same as passing &error_fatal.

Yes, sounds right.

Quoted 14 lines
Quoted 13 lines
qemu-nbd.c +2 -0
+            break;
+        }
         case 'f':
             fmt = optarg;
             break;
tests/qemu-iotests/tests/nbd-multiconn new file +1 -0
new file mode 100755
index 000000000000..7d1179b33b05
@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
Quoted 7 lines

tests/qemu-iotests/tests/nbd-multiconn line 24

Quoted 5 lines
tests/qemu-iotests/tests/nbd-multiconn new file +5 -0
+
+import os
+from contextlib import contextmanager
+import iotests
+from iotests import qemu_img_create, qemu_io_silent

qemu_io_silent() doesn't exist any more, commit 72cfb937 removed it.

Sounds like I need to rebase and post v4 anyways, then.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#11 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Am 27.04.2022 um 23:39 hat Eric Blake geschrieben:

Quoted 34 lines · Kevin Wolf
Quoted 13 lines

Am 14.03.2022 um 21:38 hat Eric Blake geschrieben:

Quoted 11 lines

According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We satisfy these conditions in qemu when our block layer is backed by the local filesystem (by virtue of the semantics of fdatasync(), and the fact that qemu itself is not buffering writes beyond flushes). It is harder to state whether we satisfy these conditions for network-based protocols, so the safest course of action is to allow users to opt-in to advertising multi-conn.

Quoted 1 line

Do you have an example of how this could be unsafe?

Nothing direct. I tried to turn this on unconditionally in an earlier version, and we waffled about whether we could prove that network block backends (such as gluster) provide us the safety that the NBD spec demands:

https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00038.html
https://lists.gnu.org/archive/html/qemu-devel/2021-10/msg06744.html

Quoted 9 lines

As I understand it, the NBD server has a single BlockBackend and therefore is a single client for the backend, be it file-posix or any network-based protocol. It doesn't really make a difference for the storage from how many different NBD clients the requests are coming.

I would have expected that cache coherency of the protocol level driver would only matter if you had two QEMU processes accessing the same file concurrently.

Quoted 6 lines

Or a multi-pathed connection to network storage, where one QEMU process accesses the network device, but those accesses may round-robin which server they reach, and where any caching at an individual server may be inconsistent with what is seen on another server unless flushing is used to force the round-robin access to synchronize between the multi-path views.

I don't think this is a realistic scenario. It would mean that you successfully write data to the storage, and when you then read the same location, you get different data back. This would be inconsistent even with a single client. So I'd call this broken storage that should be replaced as soon as possible.

I could imagine problems of this kind with two separate connections to the network storage, but here all the NBD clients share a single BlockBackend, so for the storage they are a single connection.

Quoted 6 lines
Quoted 6 lines

In fact, I don't think we even need the flush restriction from the NBD spec. All clients see the same state (that of the NBD server BlockBackend) even without anyone issuing any flush. The flush is only needed to make sure that cached data is written to the backing storage when writeback caches are involved.

Quoted 8 lines
Quoted 1 line

Please correct me if I'm misunderstanding something here.

Likewise me, if I'm being overly cautious.

I can certainly write a simpler v4 that just always advertises MULTI_CONN if we allow more than one client, without any knob to override it; it's just that it is harder to write a commit message justifying why I think it is safe to do so.

Having an explicit option doesn't hurt, but it's the reasoning in the commit message that feels wrong to me.

We could consider changing "auto" to advertise MULTI_CONN even for writable exports. There might still be a good reason not to do this by default, though, because of the NBD clients. I'm quite sure that the backend won't make any trouble, but client might if someone else is writing to the same image (this is why we require an explicit share-rw=on for guest devices in the same case).

Kevin

author
#12 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

On Fri, Apr 29, 2022 at 02:49:35PM +0200, Kevin Wolf wrote:
...

Quoted 13 lines
Quoted 6 lines

Or a multi-pathed connection to network storage, where one QEMU process accesses the network device, but those accesses may round-robin which server they reach, and where any caching at an individual server may be inconsistent with what is seen on another server unless flushing is used to force the round-robin access to synchronize between the multi-path views.

I don't think this is a realistic scenario. It would mean that you successfully write data to the storage, and when you then read the same location, you get different data back. This would be inconsistent even with a single client. So I'd call this broken storage that should be replaced as soon as possible.

Quoted 3 lines

I could imagine problems of this kind with two separate connections to the network storage, but here all the NBD clients share a single BlockBackend, so for the storage they are a single connection.

I like that chain of reasoning.

Quoted 19 lines
Quoted 7 lines
Quoted 7 lines

In fact, I don't think we even need the flush restriction from the NBD spec. All clients see the same state (that of the NBD server BlockBackend) even without anyone issuing any flush. The flush is only needed to make sure that cached data is written to the backing storage when writeback caches are involved.

Quoted 8 lines
Quoted 1 line

Please correct me if I'm misunderstanding something here.

Likewise me, if I'm being overly cautious.

I can certainly write a simpler v4 that just always advertises MULTI_CONN if we allow more than one client, without any knob to override it; it's just that it is harder to write a commit message justifying why I think it is safe to do so.

Having an explicit option doesn't hurt, but it's the reasoning in the commit message that feels wrong to me.

Quoted 6 lines

We could consider changing "auto" to advertise MULTI_CONN even for writable exports. There might still be a good reason not to do this by default, though, because of the NBD clients. I'm quite sure that the backend won't make any trouble, but client might if someone else is writing to the same image (this is why we require an explicit share-rw=on for guest devices in the same case).

If your worry is about a client trying to determine if writing to an NBD server is going to race with some external process writing to the direct image, I don't see how not advertising MULTI_CONN will make things safer - the NBD client to qemu-nbd will still be going through a single backend, and that race is present even if there is only one NBD client. The point of MULTI_CONN is informing the client that it can open multiple sockets and see a consistent view across all of them, and in your scenario of the server competing with some external process over the underlying data file, that competition is not controlled by how many NBD clients connect to the server, but by the external process having access at the same time the server has access through a single BlockBackend (and would be just as risky as if MULTI_CONN were not advertised and the client limits itself to one NBD connection).

If we can argue that our single BlockBackend point of access is safe enough to default to advertising MULTI_CONN for writable connections (when we support parallel clients), then exposing an OnOffAuto knob is overkill. I'm not even sure I can envision a case where needing to not advertise the bit would matter to a client (clients are supposed to ignore unknown feature bits).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#13 lore raw

Re: [PATCH v3 3/3] nbd/server: Allow MULTI_CONN for shared writable exports

Am 02.05.2022 um 23:12 hat Eric Blake geschrieben:

Quoted 47 lines · Kevin Wolf

...

Quoted 13 lines
Quoted 6 lines

Or a multi-pathed connection to network storage, where one QEMU process accesses the network device, but those accesses may round-robin which server they reach, and where any caching at an individual server may be inconsistent with what is seen on another server unless flushing is used to force the round-robin access to synchronize between the multi-path views.

I don't think this is a realistic scenario. It would mean that you successfully write data to the storage, and when you then read the same location, you get different data back. This would be inconsistent even with a single client. So I'd call this broken storage that should be replaced as soon as possible.

Quoted 3 lines

I could imagine problems of this kind with two separate connections to the network storage, but here all the NBD clients share a single BlockBackend, so for the storage they are a single connection.

I like that chain of reasoning.

Quoted 19 lines
Quoted 7 lines
Quoted 7 lines

In fact, I don't think we even need the flush restriction from the NBD spec. All clients see the same state (that of the NBD server BlockBackend) even without anyone issuing any flush. The flush is only needed to make sure that cached data is written to the backing storage when writeback caches are involved.

Quoted 8 lines
Quoted 1 line

Please correct me if I'm misunderstanding something here.

Likewise me, if I'm being overly cautious.

I can certainly write a simpler v4 that just always advertises MULTI_CONN if we allow more than one client, without any knob to override it; it's just that it is harder to write a commit message justifying why I think it is safe to do so.

Having an explicit option doesn't hurt, but it's the reasoning in the commit message that feels wrong to me.

Quoted 6 lines

We could consider changing "auto" to advertise MULTI_CONN even for writable exports. There might still be a good reason not to do this by default, though, because of the NBD clients. I'm quite sure that the backend won't make any trouble, but client might if someone else is writing to the same image (this is why we require an explicit share-rw=on for guest devices in the same case).

Quoted 3 lines

If your worry is about a client trying to determine if writing to an NBD server is going to race with some external process writing to the direct image

No, I'm not really thinking of a race, but just the fact that someone else could write to the image at all. Guest OSes don't generally expect the disk magically changing under their feet (unless you use cluster filesystems etc.), so the safe default is not allowing multiple writers to the same image.

Quoted 4 lines

I don't see how not advertising MULTI_CONN will make things safer - the NBD client to qemu-nbd will still be going through a single backend, and that race is present even if there is only one NBD client. The point of MULTI_CONN is informing the client that it can

Quoted 8 lines

open multiple sockets and see a consistent view across all of them, and in your scenario of the server competing with some external process over the underlying data file, that competition is not controlled by how many NBD clients connect to the server, but by the external process having access at the same time the server has access through a single BlockBackend (and would be just as risky as if MULTI_CONN were not advertised and the client limits itself to one NBD connection).

Oh, right, that makes sense. So the other connections should really be considered the same client in this case, not some random other writer.

Can we somehow tell whether a new connection is just the same client opening an additional connection or whether someone else is connecting?

But maybe it's not worth spending too much time with this for a check that would only try to catch configurations that the user set up in a way they didn't actually intend.

Quoted 6 lines

If we can argue that our single BlockBackend point of access is safe enough to default to advertising MULTI_CONN for writable connections (when we support parallel clients), then exposing an OnOffAuto knob is overkill. I'm not even sure I can envision a case where needing to not advertise the bit would matter to a client (clients are supposed to ignore unknown feature bits).

Yes, my concern seems unrelated to MULTI_CONN after all.

Kevin

author
#14 lore raw
In reply to Eric Blake

Ping. Now that 7.1 is open, I'd like to include this series in my next NBD pull request.

Quoted 34 lines · Eric Blake

v2 was here:
https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg03314.html

Since then:
- expose the knob through 'qemu-nbd -m on|off|auto'
- reuse QAPI OnOffAuto type instead of rolling my own
- rewrite the iotest from bash into python (thanks: Nir, Vladimir)
- split out easy preliminary fixes (patches 1 and 2 are new)

Given that this is a new feature, it probably needs to be 7.1 material.

Eric Blake (3):
docs: Consistent typography for options of qemu-nbd
qemu-nbd: Pass max connections to blockdev layer nbd/server: Allow MULTI_CONN for shared writable exports

 docs/interop/nbd.txt                       |   1 +
 docs/tools/qemu-nbd.rst                    |  26 ++--
 qapi/block-export.json                     |  19 ++-
 include/block/nbd.h                        |   5 +-
 blockdev-nbd.c                             |  13 +-
 nbd/server.c                               |  27 +++-
 qemu-nbd.c                                 |  22 ++-
 MAINTAINERS                                |   1 +
 tests/qemu-iotests/tests/nbd-multiconn     | 157 +++++++++++++++++++++
 tests/qemu-iotests/tests/nbd-multiconn.out |   5 +
 10 files changed, 252 insertions(+), 24 deletions(-)
 create mode 100755 tests/qemu-iotests/tests/nbd-multiconn
 create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
Signature · 1 line
2.35.1
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org