Skip to contentSkip to conversation

nbd: MULTI_CONN for shared writable exports

PATCH Applied Eric Blake 2 participants 4 messages May 12, 2022

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v4 cover nbd: MULTI_CONN for shared writable exports

v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html
with additional review here:
https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00166.html

Since then:
- patch 1 of v3 applied independently
- patch 3 simplified with beefed up commit message (now unconditional, because we have convinced ourselves that all shared clients go through a single block backend with no cache inconsistency) [kwolf]
- patch 3 rebase iotest addition to recent iotest improvements
- patch 3 s/7.0/7.1/ to match another slip in time

Eric Blake (2):
qemu-nbd: Pass max connections to blockdev layer nbd/server: Allow MULTI_CONN for shared writable exports

 docs/interop/nbd.txt                          |   1 +
 docs/tools/qemu-nbd.rst                       |   3 +-
 qapi/block-export.json                        |   8 +-
 include/block/nbd.h                           |   5 +-
 blockdev-nbd.c                                |  13 +-
 nbd/server.c                                  |  10 +-
 qemu-nbd.c                                    |   2 +-
 MAINTAINERS                                   |   1 +
 tests/qemu-iotests/tests/nbd-multiconn        | 145 ++++++++++++++++++
 tests/qemu-iotests/tests/nbd-multiconn.out    |   5 +
 .../tests/nbd-qemu-allocation.out             |   2 +-
 11 files changed, 178 insertions(+), 17 deletions(-)
 create mode 100755 tests/qemu-iotests/tests/nbd-multiconn
 create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
Signature · 1 line
2.36.1
author
#1 lore raw

PATCH v4 1/2 qemu-nbd: Pass max connections to blockdev layer

The next patch wants to adjust whether the NBD server code advertises MULTI_CONN based on whether it is known if the server limits to exactly one client. For a server started by QMP, this information is obtained through nbd_server_start (which can support more than one export); but for qemu-nbd (which supports exactly one export), it is controlled only by the command-line option -e/--shared. Since we already have a hook function used by qemu-nbd, it's easiest to just alter its signature to fit our needs.

---
 include/block/nbd.h | 2 +-
 blockdev-nbd.c      | 8 ++++----
 qemu-nbd.c          | 2 +-
 3 files changed, 6 insertions(+), 6 deletions(-)
include/block/nbd.h +1 -1
index a98eb665da..c5a29ce1c6 100644
@@ -344,7 +344,7 @@ void nbd_client_new(QIOChannelSocket *sioc,
 void nbd_client_get(NBDClient *client);
 void nbd_client_put(NBDClient *client);
 
-void nbd_server_is_qemu_nbd(bool value);
+void nbd_server_is_qemu_nbd(int max_connections);
 bool nbd_server_is_running(void);
 void nbd_server_start(SocketAddress *addr, const char *tls_creds,
                       const char *tls_authz, uint32_t max_connections,
blockdev-nbd.c +4 -4
index 7f6531cba0..711e0e72bd 100644
@@ -30,18 +30,18 @@ typedef struct NBDServerData {
 } NBDServerData;
 
 static NBDServerData *nbd_server;
-static bool is_qemu_nbd;
+static int qemu_nbd_connections = -1; /* Non-negative if this is qemu-nbd */
 
 static void nbd_update_server_watch(NBDServerData *s);
 
-void nbd_server_is_qemu_nbd(bool value)
+void nbd_server_is_qemu_nbd(int max_connections)
 {
-    is_qemu_nbd = value;
+    qemu_nbd_connections = max_connections;
 }
 
 bool nbd_server_is_running(void)
 {
-    return nbd_server || is_qemu_nbd;
+    return nbd_server || qemu_nbd_connections >= 0;
 }
 
 static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
qemu-nbd.c +1 -1
index 2382b5042a..0cd5aa6f02 100644
@@ -1095,7 +1095,7 @@ int main(int argc, char **argv)
 
     bs->detect_zeroes = detect_zeroes;
 
-    nbd_server_is_qemu_nbd(true);
+    nbd_server_is_qemu_nbd(shared);
 
     export_opts = g_new(BlockExportOptions, 1);
     *export_opts = (BlockExportOptions) {
Signature · 1 line
2.36.1
author
#2 lore raw

PATCH v4 2/2 nbd/server: Allow MULTI_CONN for shared writable exports

In reply to Eric Blake

According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush.

We always satisfy these conditions in qemu - even when we support multiple clients, ALL clients go through a single point of reference into the block layer, with no local caching. The effect of one client is instantly visible to the next client. Even if our backend were a network device, we argue that any multi-path caching effects that would cause inconsistencies in back-to-back actions not seeing the effect of previous actions would be a bug in that backend, and not the fault of caching in qemu. As such, it is safe to unconditionally advertise CAN_MULTI_CONN for any qemu NBD server situation that supports parallel clients.

Note, however, that we don't want to advertise CAN_MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). This is visible by no longer advertising MULTI_CONN for 'qemu-nbd -r' without -e, as in the iotest nbd-qemu-allocation.

The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test.

---
 docs/interop/nbd.txt                          |   1 +
 docs/tools/qemu-nbd.rst                       |   3 +-
 qapi/block-export.json                        |   8 +-
 include/block/nbd.h                           |   3 +-
 blockdev-nbd.c                                |   5 +
 nbd/server.c                                  |  10 +-
 MAINTAINERS                                   |   1 +
 tests/qemu-iotests/tests/nbd-multiconn        | 145 ++++++++++++++++++
 tests/qemu-iotests/tests/nbd-multiconn.out    |   5 +
 .../tests/nbd-qemu-allocation.out             |   2 +-
 10 files changed, 172 insertions(+), 11 deletions(-)
 create mode 100755 tests/qemu-iotests/tests/nbd-multiconn
 create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
docs/interop/nbd.txt +1 -0
index bdb0f2a41a..f5ca25174a 100644
@@ -68,3 +68,4 @@ NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CACHE
 * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports,
 NBD_CMD_FLAG_FAST_ZERO
 * 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth"
+* 7.1: NBD_FLAG_CAN_MULTI_CONN for shareable writable exports
docs/tools/qemu-nbd.rst +1 -2
index 4c950f6199..8e08a29e89 100644
@@ -139,8 +139,7 @@ driver options if :option:`--image-opts` is specified.
 .. option:: -e, --shared=NUM
 
   Allow up to *NUM* clients to share the device (default
-  ``1``), 0 for unlimited. Safe for readers, but for now,
-  consistency is not guaranteed between multiple writers.
+  ``1``), 0 for unlimited.
 
 .. option:: -t, --persistent
 
qapi/block-export.json +6 -2
index 1de16d2589..7776248435 100644
@@ -22,7 +22,9 @@
 #             recreated on the fly while the NBD server is active.
 #             If missing, it will default to denying access (since 4.0).
 # @max-connections: The maximum number of connections to allow at the same
-#                   time, 0 for unlimited. (since 5.2; default: 0)
+#                   time, 0 for unlimited. Setting this to 1 also stops
+#                   the server from advertising multiple client support
+#                   (since 5.2; default: 0)
 #
 # Since: 4.2
 ##
@@ -51,7 +53,9 @@
 #             recreated on the fly while the NBD server is active.
 #             If missing, it will default to denying access (since 4.0).
 # @max-connections: The maximum number of connections to allow at the same
-#                   time, 0 for unlimited. (since 5.2; default: 0)
+#                   time, 0 for unlimited. Setting this to 1 also stops
+#                   the server from advertising multiple client support
+#                   (since 5.2; default: 0).
 #
 # Returns: error if the server is already running.
 #
include/block/nbd.h +2 -1
index c5a29ce1c6..c74b7a9d2e 100644
@@ -1,5 +1,5 @@
 /*
- *  Copyright (C) 2016-2020 Red Hat, Inc.
+ *  Copyright (C) 2016-2022 Red Hat, Inc.
  *  Copyright (C) 2005  Anthony Liguori <[email protected]>
  *
  *  Network Block Device
@@ -346,6 +346,7 @@ void nbd_client_put(NBDClient *client);
 
 void nbd_server_is_qemu_nbd(int max_connections);
 bool nbd_server_is_running(void);
+int nbd_server_max_connections(void);
 void nbd_server_start(SocketAddress *addr, const char *tls_creds,
                       const char *tls_authz, uint32_t max_connections,
                       Error **errp);
blockdev-nbd.c +5 -0
index 711e0e72bd..012256bb02 100644
@@ -44,6 +44,11 @@ bool nbd_server_is_running(void)
     return nbd_server || qemu_nbd_connections >= 0;
 }
 
+int nbd_server_max_connections(void)
+{
+    return nbd_server ? nbd_server->max_connections : qemu_nbd_connections;
+}
+
 static void nbd_blockdev_client_closed(NBDClient *client, bool ignored)
 {
     nbd_client_put(client);
nbd/server.c +5 -5
index 4cdbc062c1..213e00e761 100644
@@ -1,5 +1,5 @@
 /*
- *  Copyright (C) 2016-2021 Red Hat, Inc.
+ *  Copyright (C) 2016-2022 Red Hat, Inc.
  *  Copyright (C) 2005  Anthony Liguori <[email protected]>
  *
  *  Network Block Device Server Side
@@ -1642,7 +1642,6 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     int64_t size;
     uint64_t perm, shared_perm;
     bool readonly = !exp_args->writable;
-    bool shared = !exp_args->writable;
     BlockDirtyBitmapOrStrList *bitmaps;
     size_t i;
     int ret;
@@ -1693,11 +1692,12 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,
     exp->description = g_strdup(arg->description);
     exp->nbdflags = (NBD_FLAG_HAS_FLAGS | NBD_FLAG_SEND_FLUSH |
                      NBD_FLAG_SEND_FUA | NBD_FLAG_SEND_CACHE);
+
+    if (nbd_server_max_connections() != 1) {
+        exp->nbdflags |= NBD_FLAG_CAN_MULTI_CONN;
+    }
     if (readonly) {
         exp->nbdflags |= NBD_FLAG_READ_ONLY;
-        if (shared) {
-            exp->nbdflags |= NBD_FLAG_CAN_MULTI_CONN;
-        }
     } else {
         exp->nbdflags |= (NBD_FLAG_SEND_TRIM | NBD_FLAG_SEND_WRITE_ZEROES |
                           NBD_FLAG_SEND_FAST_ZERO);
MAINTAINERS +1 -0
index 571556d279..fbc0662627 100644
@@ -3367,6 +3367,7 @@ F: qemu-nbd.*
 F: blockdev-nbd.c
 F: docs/interop/nbd.txt
 F: docs/tools/qemu-nbd.rst
+F: tests/qemu-iotests/tests/*nbd*
 T: git https://repo.or.cz/qemu/ericb.git nbd
 T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd
 
tests/qemu-iotests/tests/nbd-multiconn new file +145 -0

145 lines not on this page · raw message

tests/qemu-iotests/tests/nbd-multiconn.out new file +5 -0
new file mode 100644
index 0000000000..8d7e996700
@@ -0,0 +1,5 @@
+...
+----------------------------------------------------------------------
+Ran 3 tests
+
+OK
tests/qemu-iotests/tests/nbd-qemu-allocation.out +1 -1
index 0bf1abb063..9d938db24e 100644
@@ -17,7 +17,7 @@ wrote 2097152/2097152 bytes at offset 1048576
 exports available: 1
  export: ''
   size:  4194304
-  flags: 0x58f ( readonly flush fua df multi cache )
+  flags: 0x48f ( readonly flush fua df cache )
   min block: 1
   opt block: 4096
   max block: 33554432
Signature · 1 line
2.36.1