nbd: MULTI_CONN for shared writable exports
v4 cover nbd: MULTI_CONN for shared writable exports
v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html
with additional review here:
https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00166.html
Since then:
- patch 1 of v3 applied independently
- patch 3 simplified with beefed up commit message (now unconditional, because we have convinced ourselves that all shared clients go through a single block backend with no cache inconsistency) [kwolf]
- patch 3 rebase iotest addition to recent iotest improvements
- patch 3 s/7.0/7.1/ to match another slip in time
Eric Blake (2):
qemu-nbd: Pass max connections to blockdev layer nbd/server: Allow MULTI_CONN for shared writable exports
docs/interop/nbd.txt | 1 + docs/tools/qemu-nbd.rst | 3 +- qapi/block-export.json | 8 +- include/block/nbd.h | 5 +- blockdev-nbd.c | 13 +- nbd/server.c | 10 +- qemu-nbd.c | 2 +- MAINTAINERS | 1 + tests/qemu-iotests/tests/nbd-multiconn | 145 ++++++++++++++++++ tests/qemu-iotests/tests/nbd-multiconn.out | 5 + .../tests/nbd-qemu-allocation.out | 2 +- 11 files changed, 178 insertions(+), 17 deletions(-) create mode 100755 tests/qemu-iotests/tests/nbd-multiconn create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
Signature · 1 line
2.36.1
PATCH v4 1/2 qemu-nbd: Pass max connections to blockdev layer
The next patch wants to adjust whether the NBD server code advertises MULTI_CONN based on whether it is known if the server limits to exactly one client. For a server started by QMP, this information is obtained through nbd_server_start (which can support more than one export); but for qemu-nbd (which supports exactly one export), it is controlled only by the command-line option -e/--shared. Since we already have a hook function used by qemu-nbd, it's easiest to just alter its signature to fit our needs.
--- include/block/nbd.h | 2 +- blockdev-nbd.c | 8 ++++---- qemu-nbd.c | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-)
include/block/nbd.h +1 -1
@@ -344,7 +344,7 @@ void nbd_client_new(QIOChannelSocket *sioc,void nbd_client_get (NBDClient *client);void nbd_client_put (NBDClient *client); -void nbd_server_is_qemu_nbd (bool value); +void nbd_server_is_qemu_nbd (int max_connections);bool nbd_server_is_running (void ); void nbd_server_start(SocketAddress *addr, const char *tls_creds, const char *tls_authz, uint32_t max_connections,
blockdev-nbd.c +4 -4
@@ -30,18 +30,18 @@ typedef struct NBDServerData { } NBDServerData;static NBDServerData *nbd_server; -static bool is_qemu_nbd; +static int qemu_nbd_connections =-1 ;/* Non-negative if this is qemu-nbd */ static void nbd_update_server_watch (NBDServerData *s); -void nbd_server_is_qemu_nbd (bool value) +void nbd_server_is_qemu_nbd (int max_connections) { - is_qemu_nbd = value; + qemu_nbd_connections = max_connections; }bool nbd_server_is_running (void ) { -return nbd_server || is_qemu_nbd; +return nbd_server || qemu_nbd_connections >=0 ; }static void nbd_blockdev_client_closed (NBDClient *client,bool ignored)
qemu-nbd.c +1 -1
@@ -1095,7 +1095,7 @@ int main(int argc, char **argv) bs->detect_zeroes = detect_zeroes; -nbd_server_is_qemu_nbd (true); +nbd_server_is_qemu_nbd (shared); export_opts =g_new (BlockExportOptions,1 ); *export_opts = (BlockExportOptions ) {
Signature · 1 line
2.36.1
PATCH v4 2/2 nbd/server: Allow MULTI_CONN for shared writable exports
In reply to Eric BlakeAccording to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush.
We always satisfy these conditions in qemu - even when we support multiple clients, ALL clients go through a single point of reference into the block layer, with no local caching. The effect of one client is instantly visible to the next client. Even if our backend were a network device, we argue that any multi-path caching effects that would cause inconsistencies in back-to-back actions not seeing the effect of previous actions would be a bug in that backend, and not the fault of caching in qemu. As such, it is safe to unconditionally advertise CAN_MULTI_CONN for any qemu NBD server situation that supports parallel clients.
Note, however, that we don't want to advertise CAN_MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). This is visible by no longer advertising MULTI_CONN for 'qemu-nbd -r' without -e, as in the iotest nbd-qemu-allocation.
The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test.
--- docs/interop/nbd.txt | 1 + docs/tools/qemu-nbd.rst | 3 +- qapi/block-export.json | 8 +- include/block/nbd.h | 3 +- blockdev-nbd.c | 5 + nbd/server.c | 10 +- MAINTAINERS | 1 + tests/qemu-iotests/tests/nbd-multiconn | 145 ++++++++++++++++++ tests/qemu-iotests/tests/nbd-multiconn.out | 5 + .../tests/nbd-qemu-allocation.out | 2 +- 10 files changed, 172 insertions(+), 11 deletions(-) create mode 100755 tests/qemu-iotests/tests/nbd-multiconn create mode 100644 tests/qemu-iotests/tests/nbd-multiconn.out
docs/interop/nbd.txt +1 -0
@@ -68,3 +68,4 @@ NBD_CMD_BLOCK_STATUS for "qemu:dirty-bitmap:", NBD_CMD_CACHE * 4.2: NBD_FLAG_CAN_MULTI_CONN for shareable read-only exports, NBD_CMD_FLAG_FAST_ZERO * 5.2: NBD_CMD_BLOCK_STATUS for "qemu:allocation-depth" +* 7.1: NBD_FLAG_CAN_MULTI_CONN for shareable writable exports
docs/tools/qemu-nbd.rst +1 -2
@@ -139,8 +139,7 @@ driver options if :option:`--image-opts` is specified. .. option:: -e, --shared=NUM Allow up to *NUM* clients to share the device (default - ``1``), 0 for unlimited. Safe for readers, but for now, - consistency is not guaranteed between multiple writers. + ``1``), 0 for unlimited. .. option:: -t, --persistent
qapi/block-export.json +6 -2
@@ -22,7 +22,9 @@ # recreated on the fly while the NBD server is active. # If missing, it will default to denying access (since 4.0). # @max-connections: The maximum number of connections to allow at the same -# time, 0 for unlimited. (since 5.2; default: 0) +# time, 0 for unlimited. Setting this to 1 also stops +# the server from advertising multiple client support +# (since 5.2; default: 0) # # Since: 4.2 ##
@@ -51,7 +53,9 @@ # recreated on the fly while the NBD server is active. # If missing, it will default to denying access (since 4.0). # @max-connections: The maximum number of connections to allow at the same -# time, 0 for unlimited. (since 5.2; default: 0) +# time, 0 for unlimited. Setting this to 1 also stops +# the server from advertising multiple client support +# (since 5.2; default: 0). # # Returns: error if the server is already running. #
include/block/nbd.h +2 -1
@@ -1,5 +1,5 @@/* -* Copyright (C) 2016-2020 Red Hat, Inc. +* Copyright (C) 2016-2022 Red Hat, Inc. * Copyright (C) 2005 Anthony Liguori <[email protected]> * * Network Block Device
@@ -346,6 +346,7 @@ void nbd_client_put(NBDClient *client);void nbd_server_is_qemu_nbd (int max_connections);bool nbd_server_is_running (void ); +int nbd_server_max_connections (void );void nbd_server_start (SocketAddress *addr,const char *tls_creds,const char *tls_authz,uint32_t max_connections,Error **errp);
blockdev-nbd.c +5 -0
@@ -44,6 +44,11 @@ bool nbd_server_is_running(void)return nbd_server || qemu_nbd_connections >=0 ; } +int nbd_server_max_connections (void ) +{ +return nbd_server ? nbd_server->max_connections : qemu_nbd_connections; +} +static void nbd_blockdev_client_closed (NBDClient *client,bool ignored) {nbd_client_put (client);
nbd/server.c +5 -5
@@ -1,5 +1,5 @@/* -* Copyright (C) 2016-2021 Red Hat, Inc. +* Copyright (C) 2016-2022 Red Hat, Inc. * Copyright (C) 2005 Anthony Liguori <[email protected]> * * Network Block Device Server Side
@@ -1642,7 +1642,6 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args,int64_t size;uint64_t perm, shared_perm;bool readonly = !exp_args->writable; -bool shared = !exp_args->writable;BlockDirtyBitmapOrStrList *bitmaps;size_t i;int ret;
@@ -1693,11 +1692,12 @@ static int nbd_export_create(BlockExport *blk_exp, BlockExportOptions *exp_args, exp->description =g_strdup (arg->description); exp->nbdflags = (NBD_FLAG_HAS_FLAGS |NBD_FLAG_SEND_FLUSH |NBD_FLAG_SEND_FUA |NBD_FLAG_SEND_CACHE ); + +if (nbd_server_max_connections () !=1 ) { + exp->nbdflags |=NBD_FLAG_CAN_MULTI_CONN ; + }if (readonly) { exp->nbdflags |=NBD_FLAG_READ_ONLY ; -if (shared) { - exp->nbdflags |=NBD_FLAG_CAN_MULTI_CONN ; - } }else { exp->nbdflags |= (NBD_FLAG_SEND_TRIM |NBD_FLAG_SEND_WRITE_ZEROES |NBD_FLAG_SEND_FAST_ZERO );
MAINTAINERS +1 -0
@@ -3367,6 +3367,7 @@ F: qemu-nbd.* F: blockdev-nbd.c F: docs/interop/nbd.txt F: docs/tools/qemu-nbd.rst +F: tests/qemu-iotests/tests/*nbd* T: git https://repo.or.cz/qemu/ericb.git nbd T: git https://src.openvz.org/scm/~vsementsov/qemu.git nbd
tests/qemu-iotests/tests/nbd-multiconn new file +145 -0
145 lines not on this page · raw message
tests/qemu-iotests/tests/nbd-multiconn.out new file +5 -0
@@ -0,0 +1,5 @@ +... +---------------------------------------------------------------------- +Ran 3 tests + +OK
tests/qemu-iotests/tests/nbd-qemu-allocation.out +1 -1
@@ -17,7 +17,7 @@ wrote 2097152/2097152 bytes at offset 1048576 exports available: 1 export: '' size: 4194304 - flags: 0x58f ( readonly flush fua df multi cache ) + flags: 0x48f ( readonly flush fua df cache ) min block: 1 opt block: 4096 max block: 33554432
Signature · 1 line
2.36.1
Kevin Wolf
AppliedAm 12.05.2022 um 02:49 hat Eric Blake geschrieben:
Quoted 4 lines
v3 was here:
https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html
with additional review here:
https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00166.html
Thanks, applied to the block branch.
Kevin