nbd: BLOCK_STATUS
cover nbd: BLOCK_STATUS
Hi all!
We really need exporting dirty bitmaps feature as well as remote get_block_status for nbd devices. So, here is minimalistic and restricted realization of 'structured reply' and 'block status' nbd protocol extension (as second is developed over the first the combined spec may be found here:
https://github.com/NetworkBlockDevice/nbd/blob/extension-blockstatus/doc/proto.md)
What is done:
server:
- can send not fragmented structured replies for CMD_READ (this was done only because the spec doesn't allow structure reply feature without maintaining structured read)
- can export dirty bitmap through BLOCK_STATUS. Only one bitmap can be exported, negotiation query should be 'qemu-dirty-bitmap:<bitmap name>'
- cab export block status through BLOCK_STATUS. Client can negotiate only one entity for exporting through BLOCK_STATUS - bitmap _or_ block status. negotiation query should be 'base:allocation', as defined in the spec. server sends only one extent on each BLOCK_STATUS query.
client:
- can receive not fragmented structured replies for CMD_READ
- can get load dirty bitmap through nbd. Be careful: bitmap for export is is selected during nbd negotiation - actually in open(). So, name argument for qmp block-dirty-bitmap-load is just a _new_ name for loaded bitmap. (any way, for us block-dirty-bitmap-load is just a way to test the feature, really we need only server part)
- get_block_status works now through nbd CMD_BLOCK_STATUS, if base:allocation is negotiated for export.
It should be minimal but fully compatible realization of the spec.
web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=nbd-block-status-v1
git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag nbd-block-status-v1)
Vladimir Sementsov-Ogievskiy (18):
nbd: rename NBD_REPLY_MAGIC to NBD_SIMPLE_REPLY_MAGIC
nbd-server: refactor simple reply sending
nbd: Minimal structured read for server
nbd/client: refactor nbd_receive_starttls
nbd/client: fix drop_sync
nbd/client: refactor drop_sync
nbd: Minimal structured read for client
hbitmap: add next_zero function
block/dirty-bitmap: add bdrv_dirty_bitmap_next() block/dirty-bitmap: add bdrv_load_dirty_bitmap
nbd: BLOCK_STATUS for bitmap export: server part
nbd: BLOCK_STATUS for bitmap export: client part
nbd: add nbd_dirty_bitmap_load
qmp: add x-debug-block-dirty-bitmap-sha256
qmp: add block-dirty-bitmap-load
iotests: add test for nbd dirty bitmap export
nbd: BLOCK_STATUS for standard get_block_status function: server part
nbd: BLOCK_STATUS for standard get_block_status function: client part
block/dirty-bitmap.c | 70 ++++ block/nbd-client.c | 230 ++++++++++- block/nbd-client.h | 13 + block/nbd.c | 44 +- blockdev.c | 57 +++ include/block/block_int.h | 4 + include/block/dirty-bitmap.h | 10 + include/block/nbd.h | 73 +++- include/qemu/hbitmap.h | 16 + nbd/client.c | 373 ++++++++++++++--- nbd/nbd-internal.h | 25 +- nbd/server.c | 661 ++++++++++++++++++++++++++++++- qapi/block-core.json | 46 ++- qemu-nbd.c | 2 +- tests/Makefile.include | 2 +- tests/qemu-iotests/180 | 133 +++++++ tests/qemu-iotests/180.out | 5 + tests/qemu-iotests/group | 1 + tests/qemu-iotests/nbd-fault-injector.py | 4 +- util/hbitmap.c | 37 ++ 20 files changed, 1713 insertions(+), 93 deletions(-) create mode 100755 tests/qemu-iotests/180 create mode 100644 tests/qemu-iotests/180.out
Signature · 1 line
2.11.0
PATCH 1/18 nbd: rename NBD_REPLY_MAGIC to NBD_SIMPLE_REPLY_MAGIC
To be consistent when NBD_STRUCTURED_REPLY_MAGIC will be introduced.
--- nbd/client.c | 4 ++-- nbd/nbd-internal.h | 2 +- nbd/server.c | 4 ++-- tests/qemu-iotests/nbd-fault-injector.py | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-)
nbd/client.c +2 -2
@@ -788,7 +788,7 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) }/* Reply -[ 0 .. 3] magic (NBD_REPLY_MAGIC) +[ 0 .. 3] magic (NBD_SIMPLE_REPLY_MAGIC) [ 4 .. 7] error (0 == no error) [ 7 .. 15] handle */
@@ -808,7 +808,7 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply)", handle = %" PRIu64" }" , magic, reply->error, reply->handle); -if (magic !=NBD_REPLY_MAGIC ) { +if (magic !=NBD_SIMPLE_REPLY_MAGIC ) {LOG ("invalid magic (got 0x%" PRIx32")" , magic);return -EINVAL ; }
nbd/nbd-internal.h +1 -1
@@ -59,7 +59,7 @@#define NBD_REQUEST_SIZE (4 + 2 + 2 + 8 + 8 + 4)#define NBD_REPLY_SIZE (4 + 4 + 8)#define NBD_REQUEST_MAGIC 0x25609513 -#define NBD_REPLY_MAGIC 0x67446698 +#define NBD_SIMPLE_REPLY_MAGIC 0x67446698#define NBD_OPTS_MAGIC 0x49484156454F5054LL#define NBD_CLIENT_MAGIC 0x0000420281861253LL#define NBD_REP_MAGIC 0x0003e889045565a9LL
nbd/server.c +2 -2
@@ -750,11 +750,11 @@ static ssize_t nbd_send_reply(QIOChannel *ioc, NBDReply *reply) reply->error, reply->handle);/* Reply -[ 0 .. 3] magic (NBD_REPLY_MAGIC) +[ 0 .. 3] magic (NBD_SIMPLE_REPLY_MAGIC) [ 4 .. 7] error (0 == no error) [ 7 .. 15] handle */ -stl_be_p (buf,NBD_REPLY_MAGIC ); +stl_be_p (buf,NBD_SIMPLE_REPLY_MAGIC );stl_be_p (buf +4 , reply->error);stq_be_p (buf +8 , reply->handle);
tests/qemu-iotests/nbd-fault-injector.py +2 -2
@@ -56,7 +56,7 @@ NBD_CMD_READ = 0 NBD_CMD_WRITE = 1 NBD_CMD_DISC = 2 NBD_REQUEST_MAGIC = 0x25609513 -NBD_REPLY_MAGIC = 0x67446698 +NBD_SIMPLE_REPLY_MAGIC = 0x67446698 NBD_PASSWD = 0x4e42444d41474943 NBD_OPTS_MAGIC = 0x49484156454F5054 NBD_CLIENT_MAGIC = 0x0000420281861253
@@ -166,7 +166,7 @@ def read_request(conn): return req def write_reply(conn, error, handle): - buf = reply_struct.pack(NBD_REPLY_MAGIC, error, handle) + buf = reply_struct.pack(NBD_SIMPLE_REPLY_MAGIC, error, handle) conn.send(buf, event='reply') def handle_connection(conn, use_export):
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 01/18] nbd: rename NBD_REPLY_MAGIC to NBD_SIMPLE_REPLY_MAGIC
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
To be consistent when
NBD_STRUCTURED_REPLY_MAGICwill be introduced.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 4 ++-- nbd/nbd-internal.h | 2 +- nbd/server.c | 4 ++-- tests/qemu-iotests/nbd-fault-injector.py | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-)
Mechanical, and makes sense.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 2/18 nbd-server: refactor simple reply sending
In reply to Vladimir Sementsov-OgievskiyRename functions appropriately and also make a separate copy of NBDReply
- NBDSimpleReply, to replace NBDReply for the server. NBDReply itself will be upgraded in future patches to handle both simple and structured replies in the client.
--- include/block/nbd.h | 7 +++++++ nbd/server.c | 25 +++++++++++++------------ 2 files changed, 20 insertions(+), 12 deletions(-)
include/block/nbd.h +7 -0
@@ -63,6 +63,13 @@ struct NBDReply { };typedef struct NBDReply NBDReply ; +struct NBDSimpleReply { +/* uint32_t NBD_SIMPLE_REPLY_MAGIC */ +uint64_t handle; +uint32_t error; +}; +typedef struct NBDSimpleReply NBDSimpleReply ; +/* Transmission (export) flags: sent from server to client during handshake, but describe what will happen during transmission */ #define NBD_FLAG_HAS_FLAGS (1 << 0)/* Flags are there */
nbd/server.c +13 -12
@@ -738,7 +738,7 @@ static ssize_t nbd_receive_request(QIOChannel *ioc, NBDRequest *request)return 0 ; } -static ssize_t nbd_send_reply (QIOChannel *ioc,NBDReply *reply) +static ssize_t nbd_send_simple_reply (QIOChannel *ioc,NBDSimpleReply *reply) {uint8_t buf[NBD_REPLY_SIZE ];ssize_t ret;
@@ -1036,8 +1036,8 @@ void nbd_export_close_all(void) } } -static ssize_t nbd_co_send_reply (NBDRequestData *req,NBDReply *reply, -int len) +static ssize_t nbd_co_send_simple_reply (NBDRequestData *req, +NBDSimpleReply *reply,int len) {NBDClient *client = req->client;ssize_t rc, ret;
@@ -1048,10 +1048,10 @@ static ssize_t nbd_co_send_reply(NBDRequestData *req, NBDReply *reply,nbd_set_handlers (client);if (!len) { - rc =nbd_send_reply (client->ioc, reply); + rc =nbd_send_simple_reply (client->ioc, reply); }else {qio_channel_set_cork (client->ioc, true); - rc =nbd_send_reply (client->ioc, reply); + rc =nbd_send_simple_reply (client->ioc, reply);if (rc >=0 ) { ret =write_sync (client->ioc, req->data, len);if (ret != len) {
@@ -1174,7 +1174,7 @@ static void nbd_trip(void *opaque)NBDExport *exp = client->exp;NBDRequestData *req;NBDRequest request; -NBDReply reply; +NBDSimpleReply reply;ssize_t ret;int flags;
@@ -1231,8 +1231,9 @@ static void nbd_trip(void *opaque) }TRACE ("Read %" PRIu32" byte(s)" , request.len); -if (nbd_co_send_reply (req, &reply, request.len) <0 ) +if (nbd_co_send_simple_reply (req, &reply, request.len) <0 ) { goto out; + }break ;case NBD_CMD_WRITE :TRACE ("Request type is WRITE" );
@@ -1257,7 +1258,7 @@ static void nbd_trip(void *opaque) goto error_reply; } -if (nbd_co_send_reply (req, &reply,0 ) <0 ) { +if (nbd_co_send_simple_reply (req, &reply,0 ) <0 ) { goto out; }break ;
@@ -1288,7 +1289,7 @@ static void nbd_trip(void *opaque) goto error_reply; } -if (nbd_co_send_reply (req, &reply,0 ) <0 ) { +if (nbd_co_send_simple_reply (req, &reply,0 ) <0 ) { goto out; }break ;
@@ -1305,7 +1306,7 @@ static void nbd_trip(void *opaque)LOG ("flush failed" ); reply.error = -ret; } -if (nbd_co_send_reply (req, &reply,0 ) <0 ) { +if (nbd_co_send_simple_reply (req, &reply,0 ) <0 ) { goto out; }break ;
@@ -1317,7 +1318,7 @@ static void nbd_trip(void *opaque)LOG ("discard failed" ); reply.error = -ret; } -if (nbd_co_send_reply (req, &reply,0 ) <0 ) { +if (nbd_co_send_simple_reply (req, &reply,0 ) <0 ) { goto out; }break ;
@@ -1328,7 +1329,7 @@ static void nbd_trip(void *opaque)/* We must disconnect after NBD_CMD_WRITE if we did not * read the payload. */ -if (nbd_co_send_reply (req, &reply,0 ) <0 || !req->complete) { +if (nbd_co_send_simple_reply (req, &reply,0 ) <0 || !req->complete) { goto out; }break ;
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 02/18] nbd-server: refactor simple reply sending
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Rename functions appropriately and also make a separate copy of NBDReply
- NBDSimpleReply, to replace NBDReply for the server. NBDReply itself will be upgraded in future patches to handle both simple and structured replies in the client.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 7 +++++++ nbd/server.c | 25 +++++++++++++------------ 2 files changed, 20 insertions(+), 12 deletions(-)
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 3/18 nbd: Minimal structured read for server
In reply to Vladimir Sementsov-OgievskiyMinimal implementation of structured read: one data chunk + finishing none chunk. No segmentation.
Minimal structured error implementation: no text message. Support DF flag, but just ignore it, as there is no segmentation any way.
--- include/block/nbd.h | 31 +++++++++++++ nbd/nbd-internal.h | 2 + nbd/server.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 154 insertions(+), 4 deletions(-)
include/block/nbd.h +31 -0
@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; + +typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; + +typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError; +/* Transmission (export) flags: sent from server to client during handshake, but describe what will happen during transmission */ #define NBD_FLAG_HAS_FLAGS (1 << 0)/* Flags are there */
@@ -79,6 +98,7 @@ typedef struct NBDSimpleReply NBDSimpleReply;#define NBD_FLAG_ROTATIONAL (1 << 4)/* Use elevator algorithm - rotational media */ #define NBD_FLAG_SEND_TRIM (1 << 5)/* Send TRIM (discard) */ #define NBD_FLAG_SEND_WRITE_ZEROES (1 << 6)/* Send WRITE_ZEROES */ +#define NBD_FLAG_SEND_DF (1 << 7)/* Send DF (Do not Fragment) */ /* New-style handshake (global) flags, sent from server to client, and control what will happen during handshake phase. */
@@ -106,6 +126,7 @@ typedef struct NBDSimpleReply NBDSimpleReply;/* Request flags, sent from client to server during transmission phase */ #define NBD_CMD_FLAG_FUA (1 << 0)/* 'force unit access' during write */ #define NBD_CMD_FLAG_NO_HOLE (1 << 1)/* don't punch hole on zero run */ +#define NBD_CMD_FLAG_DF (1 << 2)/* don't fragment structured read */ /* Supported request types */ enum {
@@ -130,6 +151,16 @@ enum {* aren't overflowing some other buffer. */ #define NBD_MAX_NAME_SIZE 256 +/* Structured reply flags */ +#define NBD_REPLY_FLAG_DONE 1 + +/* Structured reply types */ +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1) +#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2) + ssize_t nbd_wr_syncv(QIOChannel *ioc, struct iovec *iov, size_t niov,
nbd/nbd-internal.h +2 -0
@@ -60,6 +60,7 @@#define NBD_REPLY_SIZE (4 + 4 + 8)#define NBD_REQUEST_MAGIC 0x25609513#define NBD_SIMPLE_REPLY_MAGIC 0x67446698 +#define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef#define NBD_OPTS_MAGIC 0x49484156454F5054LL#define NBD_CLIENT_MAGIC 0x0000420281861253LL#define NBD_REP_MAGIC 0x0003e889045565a9LL
@@ -81,6 +82,7 @@#define NBD_OPT_LIST (3)#define NBD_OPT_PEEK_EXPORT (4)#define NBD_OPT_STARTTLS (5) +#define NBD_OPT_STRUCTURED_REPLY (8)/* NBD errors are based on errno numbers, so there is a 1:1 mapping, * but only a limited set of errno values is specified in the protocol.
nbd/server.c +121 -4
164 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 03/18] nbd: Minimal structured read for server
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation of structured read: one data chunk + finishing none chunk. No segmentation.
Minimal structured error implementation: no text message. Support DF flag, but just ignore it, as there is no segmentation any way.
Might be worth adding that this is still an experimental extension to the NBD spec, and therefore that this implementation serves as proof of concept and may still need tweaking if anything major turns up before promoting it to stable. It might also be worth a link to:
https://github.com/NetworkBlockDevice/nbd/blob/extension-structured-reply/doc/proto.md
Quoted 24 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 31 +++++++++++++ nbd/nbd-internal.h | 2 + nbd/server.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 154 insertions(+), 4 deletions(-)include/block/nbd.h +8
-0@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; +
include/block/nbd.h struct NBDSimpleReply { · line 90
Quoted 10 lines
include/block/nbd.h +10
-0+typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; + +typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError;
Definitely a subset of all types added in the NBD protocol extension, but reasonable for a minimal implementation. Might be worth adding comments to the types...
include/block/nbd.h enum { · line 162
Quoted 10 lines
include/block/nbd.h +9
-0+/* Structured reply flags */ +#define NBD_REPLY_FLAG_DONE 1 + +/* Structured reply types */ +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1) +#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2)
...that correspond to these constants that will be used in the [h.]type field.
Also, it's a bit odd that you are defining constants that aren't implemented here; I don't know if it is any cleaner to save the definition for the unimplemented types until you actually implement them (NBD_REPLY_TYPE_OFFSET_HOLE, NBD_REPLY_TYPE_ERROR_OFFSET).
include/block/nbd.h line 66
Quoted 9 lines
+++
b/nbd/nbd-internal.hinclude/block/nbd.h +1
-0@@ -60,6 +60,7 @@#define NBD_REPLY_SIZE (4 + 4 + 8)#define NBD_REQUEST_MAGIC 0x25609513#define NBD_SIMPLE_REPLY_MAGIC 0x67446698 +#define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef#define NBD_OPTS_MAGIC 0x49484156454F5054LL#define NBD_CLIENT_MAGIC 0x0000420281861253LL#define NBD_REP_MAGIC 0x0003e889045565a9LL
I would not be bothered if you wanted to reindent the other lines by 3 spaces so that all the macro definitions start on the same column. But I also won't require it.
include/block/nbd.h line 85
Quoted 5 lines
include/block/nbd.h +1
-0@@ -81,6 +82,7 @@#define NBD_OPT_LIST (3)#define NBD_OPT_PEEK_EXPORT (4)#define NBD_OPT_STARTTLS (5) +#define NBD_OPT_STRUCTURED_REPLY (8)
Similar comments about consistency in the definition column.
Quoted 14 lines
include/block/nbd.h +3
-0+++ b/nbd/server.c@@ -100,6 +100,8 @@ struct NBDClient {QTAILQ_ENTRY (NBDClient) next;int nb_requests;bool closing; + +bool structured_reply; };/* That's all folks */ @@ -573,6 +575,16 @@ static int nbd_negotiate_options(NBDClient *client)return ret; }break ;
include/block/nbd.h static int nbd_negotiate_options(NBDClient *client) · line 587
Quoted 10 lines
include/block/nbd.h +10
-0+ +case NBD_OPT_STRUCTURED_REPLY : + client->structured_reply = true; + ret =nbd_negotiate_send_rep (client->ioc,NBD_REP_ACK , + clientflags); +if (ret <0 ) { +return ret; + } +break ; +
As written, you allow the client to negotiate this more than once. On the one hand, we are idempotent, so it doesn't hurt if they do so; on the other hand, it is a waste of bandwidth, and a client could abuse it by sending an infinite stream of NBD_OPT_STRUCTURED_REPLY requests and never moving into transmission phase, which is a mild form of Denial-of-Service (they're hogging a socket from accomplishing useful work for some other client). It would be acceptable if we wanted to disconnect any client that sends this option more than once, although the NBD spec does not require us to do so. Up to you if you think that's worth adding.
include/block/nbd.h static int nbd_negotiate_options(NBDClient *client) · line 589
Quoted 3 lines
include/block/nbd.h +2
-0+static void set_be_chunk (NBDStructuredReplyChunk *chunk,uint16_t flags, +uint16_t type,uint64_t handle,uint32_t length)
I'm not sure I like the name of this helper. I know what you are doing:
go from native-endian local variables into network-byte-order storage in preparation for transmission, done at the last possible moment. But I also don't know if I have a good suggestion for a better name off hand.
include/block/nbd.h static int nbd_negotiate_options(NBDClient *client)
Quoted 9 lines
include/block/nbd.h +9
-0+{ +stl_be_p (&chunk->magic,NBD_STRUCTURED_REPLY_MAGIC ); +stw_be_p (&chunk->flags, flags); +stw_be_p (&chunk->type, type); +stq_be_p (&chunk->handle, handle); +stl_be_p (&chunk->length, length); +} + +static int nbd_co_send_iov (NBDClient *client,struct iovec *iov,unsigned niov)
Probably should add the coroutine_fn annotation to this function and its friends (yeah, the NBD code doesn't consistently use it yet, but it should).
include/block/nbd.h static ssize_t nbd_co_receive_request(NBDRequestData *req, · line 1246
Quoted 10 lines
include/block/nbd.h +2
-1@@ -1147,7 +1239,8 @@ static ssize_t nbd_co_receive_request(NBDRequestData *req, rc = request->type ==NBD_CMD_WRITE ? -ENOSPC : -EINVAL ; goto out; } -if (request->flags & ~(NBD_CMD_FLAG_FUA |NBD_CMD_FLAG_NO_HOLE )) { +if (request->flags & ~(NBD_CMD_FLAG_FUA |NBD_CMD_FLAG_NO_HOLE | +NBD_CMD_FLAG_DF )) {LOG ("unsupported flags (got 0x%x)" , request->flags); rc = -EINVAL ; goto out;
Missing a check that NBD_CMD_FLAG_DF is only set for NBD_CMD_READ (it is not valid on any other command, at least in the current version of the extension specification).
include/block/nbd.h void nbd_client_new(NBDExport *exp, · line 1562
Quoted 5 lines
include/block/nbd.h +1
-0@@ -1444,6 +1559,8 @@ void nbd_client_new(NBDExport *exp, client->can_read = true; client->close = close_fn; + client->structured_reply = false;
Dead assignment, since we used 'client = g_malloc0()' above.
Overall looks like it matches the spec.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 03/18] nbd: Minimal structured read for server
Quoted 7 lines · Eric Blake
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation of structured read: one data chunk + finishing none chunk. No segmentation.
Minimal structured error implementation: no text message. Support DF flag, but just ignore it, as there is no segmentation any way.
Quoted 6 lines
Might be worth adding that this is still an experimental extension to the NBD spec, and therefore that this implementation serves as proof of concept and may still need tweaking if anything major turns up before promoting it to stable. It might also be worth a link to:
https://github.com/NetworkBlockDevice/nbd/blob/extension-structured-reply/doc/proto.md
Wouter's slides from FOSDEM said the state is "discussion complete, not yet implemented".
Paolo
Quoted 163 lines
Quoted 24 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 31 +++++++++++++ nbd/nbd-internal.h | 2 + nbd/server.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 154 insertions(+), 4 deletions(-)include/block/nbd.h +8
-0@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; +include/block/nbd.h struct NBDSimpleReply { · line 90
Quoted 10 lines
include/block/nbd.h +10
-0+typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; + +typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError;Definitely a subset of all types added in the NBD protocol extension, but reasonable for a minimal implementation. Might be worth adding comments to the types...
include/block/nbd.h enum { · line 162
Quoted 10 lines
include/block/nbd.h +9
-0+/* Structured reply flags */ +#define NBD_REPLY_FLAG_DONE 1 + +/* Structured reply types */ +#define NBD_REPLY_TYPE_NONE 0 +#define NBD_REPLY_TYPE_OFFSET_DATA 1 +#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1) +#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2)...that correspond to these constants that will be used in the [h.]type field.
Also, it's a bit odd that you are defining constants that aren't implemented here; I don't know if it is any cleaner to save the definition for the unimplemented types until you actually implement them (
NBD_REPLY_TYPE_OFFSET_HOLE,NBD_REPLY_TYPE_ERROR_OFFSET).include/block/nbd.h line 66
Quoted 9 lines
+++
b/nbd/nbd-internal.hinclude/block/nbd.h +1
-0@@ -60,6 +60,7 @@#define NBD_REPLY_SIZE (4 + 4 + 8)#define NBD_REQUEST_MAGIC 0x25609513#define NBD_SIMPLE_REPLY_MAGIC 0x67446698 +#define NBD_STRUCTURED_REPLY_MAGIC 0x668e33ef#define NBD_OPTS_MAGIC 0x49484156454F5054LL#define NBD_CLIENT_MAGIC 0x0000420281861253LL#define NBD_REP_MAGIC 0x0003e889045565a9LLI would not be bothered if you wanted to reindent the other lines by 3 spaces so that all the macro definitions start on the same column. But I also won't require it.
include/block/nbd.h line 85
Quoted 5 lines
include/block/nbd.h +1
-0@@ -81,6 +82,7 @@#define NBD_OPT_LIST (3)#define NBD_OPT_PEEK_EXPORT (4)#define NBD_OPT_STARTTLS (5) +#define NBD_OPT_STRUCTURED_REPLY (8)Similar comments about consistency in the definition column.
Quoted 14 lines
include/block/nbd.h +3
-0+++ b/nbd/server.c@@ -100,6 +100,8 @@ struct NBDClient {QTAILQ_ENTRY (NBDClient) next;int nb_requests;bool closing; + +bool structured_reply; };/* That's all folks */ @@ -573,6 +575,16 @@ static int nbd_negotiate_options(NBDClient *client)return ret; }break ;include/block/nbd.h static int nbd_negotiate_options(NBDClient *client) · line 587
Quoted 10 lines
include/block/nbd.h +10
-0+ +case NBD_OPT_STRUCTURED_REPLY : + client->structured_reply = true; + ret =nbd_negotiate_send_rep (client->ioc,NBD_REP_ACK , + clientflags); +if (ret <0 ) { +return ret; + } +break ; +As written, you allow the client to negotiate this more than once. On the one hand, we are idempotent, so it doesn't hurt if they do so; on the other hand, it is a waste of bandwidth, and a client could abuse it by sending an infinite stream of
NBD_OPT_STRUCTURED_REPLYrequests and never moving into transmission phase, which is a mild form of Denial-of-Service (they're hogging a socket from accomplishing useful work for some other client). It would be acceptable if we wanted to disconnect any client that sends this option more than once, although the NBD spec does not require us to do so. Up to you if you think that's worth adding.include/block/nbd.h static int nbd_negotiate_options(NBDClient *client) · line 589
Quoted 3 lines
include/block/nbd.h +2
-0+static void set_be_chunk (NBDStructuredReplyChunk *chunk,uint16_t flags, +uint16_t type,uint64_t handle,uint32_t length)I'm not sure I like the name of this helper. I know what you are doing:
go from native-endian local variables into network-byte-order storage in preparation for transmission, done at the last possible moment. But I also don't know if I have a good suggestion for a better name off hand.include/block/nbd.h static int nbd_negotiate_options(NBDClient *client)
Quoted 9 lines
include/block/nbd.h +9
-0+{ +stl_be_p (&chunk->magic,NBD_STRUCTURED_REPLY_MAGIC ); +stw_be_p (&chunk->flags, flags); +stw_be_p (&chunk->type, type); +stq_be_p (&chunk->handle, handle); +stl_be_p (&chunk->length, length); +} + +static int nbd_co_send_iov (NBDClient *client,struct iovec *iov,unsigned niov)Probably should add the
coroutine_fnannotation to this function and its friends (yeah, the NBD code doesn't consistently use it yet, but it should).include/block/nbd.h static ssize_t nbd_co_receive_request(NBDRequestData *req, · line 1246
Quoted 10 lines
include/block/nbd.h +2
-1@@ -1147,7 +1239,8 @@ static ssize_t nbd_co_receive_request(NBDRequestData *req, rc = request->type ==NBD_CMD_WRITE ? -ENOSPC : -EINVAL ; goto out; } -if (request->flags & ~(NBD_CMD_FLAG_FUA |NBD_CMD_FLAG_NO_HOLE )) { +if (request->flags & ~(NBD_CMD_FLAG_FUA |NBD_CMD_FLAG_NO_HOLE | +NBD_CMD_FLAG_DF )) {LOG ("unsupported flags (got 0x%x)" , request->flags); rc = -EINVAL ; goto out;Missing a check that
NBD_CMD_FLAG_DFis only set forNBD_CMD_READ(it is not valid on any other command, at least in the current version of the extension specification).include/block/nbd.h void nbd_client_new(NBDExport *exp, · line 1562
Quoted 5 lines
include/block/nbd.h +1
-0@@ -1444,6 +1559,8 @@ void nbd_client_new(NBDExport *exp, client->can_read = true; client->close = close_fn; + client->structured_reply = false;Dead assignment, since we used 'client =
g_malloc0()' above.Overall looks like it matches the spec.
Re: [Qemu-devel] [PATCH 03/18] nbd: Minimal structured read for server
In reply to Eric BlakeQuoted 41 lines · Eric Blake
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation of structured read: one data chunk + finishing none chunk. No segmentation.
Minimal structured error implementation: no text message. Support DF flag, but just ignore it, as there is no segmentation any way.Might be worth adding that this is still an experimental extension to the NBD spec, and therefore that this implementation serves as proof of concept and may still need tweaking if anything major turns up before promoting it to stable. It might also be worth a link to:
https://github.com/NetworkBlockDevice/nbd/blob/extension-structured-reply/doc/proto.md
Quoted 28 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 31 +++++++++++++ nbd/nbd-internal.h | 2 + nbd/server.c | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 154 insertions(+), 4 deletions(-)include/block/nbd.h +13
-0@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; + +typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; +
Quoted 8 lines
include/block/nbd.h struct NBDSimpleReply { · line 90
Quoted 5 lines
include/block/nbd.h +5
-0+typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError;Definitely a subset of all types added in the NBD protocol extension, but reasonable for a minimal implementation. Might be worth adding comments to the types...
Hmm, for me their names looks descriptive enough, but my sight may be biased.. What kind of comments do you want?
[...]
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 03/18] nbd: Minimal structured read for server
Quoted 25 lines · Vladimir Sementsov-Ogievskiy
Quoted 16 lines
Quoted 16 lines
include/block/nbd.h +12
-0@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; + +typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; +Quoted 8 lines
include/block/nbd.h struct NBDSimpleReply { · line 90
Quoted 5 lines
include/block/nbd.h +5
-0+typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError;Definitely a subset of all types added in the NBD protocol extension, but reasonable for a minimal implementation. Might be worth adding comments to the types...
Quoted 2 lines
Hmm, for me their names looks descriptive enough, but my sight may be biased.. What kind of comments do you want?
I guess I was thinking of existing structs in include/block/nbd.h:
/* Handshake phase structs - this struct is passed on the wire */
struct nbd_option {
uint64_t magic; /* NBD_OPTS_MAGIC */
uint32_t option; /* NBD_OPT_* */
uint32_t length;
} QEMU_PACKED;typedef struct nbd_option nbd_option;
and compared to:
/* Transmission phase structs
*
* Note: these are _NOT_ the same as the network representation of an NBD
* request and reply!
*/
struct NBDRequest {
uint64_t handle;
uint64_t from;
uint32_t len;
uint16_t flags; /* NBD_CMD_FLAG_* */
uint16_t type; /* NBD_CMD_* */
};typedef struct NBDRequest NBDRequest;
where the comments make it obvious whether QEMU_PACKED matters because we are using the struct to directly map bytes read/written on the wire.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 03/18] nbd: Minimal structured read for server
Quoted 54 lines · Eric Blake
Quoted 24 lines · Vladimir Sementsov-Ogievskiy
Quoted 16 lines
Quoted 16 lines
include/block/nbd.h +12
-0@@ -70,6 +70,25 @@ struct NBDSimpleReply { };typedef struct NBDSimpleReply NBDSimpleReply ; +typedef struct NBDStructuredReplyChunk { +uint32_t magic; +uint16_t flags; +uint16_t type; +uint64_t handle; +uint32_t length; +} QEMU_PACKED NBDStructuredReplyChunk; + +typedef struct NBDStructuredRead { +NBDStructuredReplyChunk h; +uint64_t offset; +} QEMU_PACKED NBDStructuredRead; +Quoted 8 lines
include/block/nbd.h struct NBDSimpleReply { · line 90
Quoted 5 lines
include/block/nbd.h +5
-0+typedef struct NBDStructuredError { +NBDStructuredReplyChunk h; +uint32_t error; +uint16_t message_length; +} QEMU_PACKED NBDStructuredError;Definitely a subset of all types added in the NBD protocol extension, but reasonable for a minimal implementation. Might be worth adding comments to the types...
Quoted 2 lines
Hmm, for me their names looks descriptive enough, but my sight may be biased.. What kind of comments do you want?
I guess I was thinking of existing structs in
include/block/nbd.h:/* Handshake phase structs - this struct is passed on the wire */
struct nbd_option { uint64_t magic; /* NBD_OPTS_MAGIC */ uint32_t option; /* NBD_OPT_* */ uint32_t length; } QEMU_PACKED;typedef struct
nbd_optionnbd_option;and compared to:
/* Transmission phase structs
*
* Note: these are _NOT_ the same as the network representation of an NBD
* request and reply!
*/struct NBDRequest { uint64_t handle; uint64_t from; uint32_t len; uint16_t flags; /* NBD_CMD_FLAG_* */ uint16_t type; /* NBD_CMD_* */ };
Quoted 5 lines
typedef struct NBDRequest NBDRequest;
where the comments make it obvious whether
QEMU_PACKEDmatters because we are using the struct to directly map bytes read/written on the wire.
Ok, thank you, will add.
Signature · 2 lines
Best regards, Vladimir
PATCH 4/18 nbd/client: refactor nbd_receive_starttls
In reply to Vladimir Sementsov-OgievskiySplit out nbd_receive_simple_option to be reused for structured reply option.
--- nbd/client.c | 54 +++++++++++++++++++++++++++++++++++------------------- nbd/nbd-internal.h | 14 ++++++++++++++ 2 files changed, 49 insertions(+), 19 deletions(-)
nbd/client.c +35 -19
@@ -395,39 +395,55 @@ static int nbd_receive_query_exports(QIOChannel *ioc, } } -static QIOChannel *nbd_receive_starttls (QIOChannel *ioc, -QCryptoTLSCreds *tlscreds, -const char *hostname,Error **errp) +static int nbd_receive_simple_option (QIOChannel *ioc,int opt, +bool abort_on_notsup,Error **errp) {nbd_opt_reply reply; -QIOChannelTLS *tioc; -struct NBDTLSHandshakeData data = {0 }; -TRACE ("Requesting TLS from server" ); -if (nbd_send_option_request (ioc,NBD_OPT_STARTTLS ,0 ,NULL , errp) <0 ) { -return NULL ; +TRACE ("Requesting '%s' option from server" ,nbd_opt_name (opt)); +if (nbd_send_option_request (ioc, opt,0 ,NULL , errp) <0 ) { +return -1 ; } -TRACE ("Getting TLS reply from server" ); -if (nbd_receive_option_reply (ioc,NBD_OPT_STARTTLS , &reply, errp) <0 ) { -return NULL ; +TRACE ("Getting '%s' option reply from server" ,nbd_opt_name (opt)); +if (nbd_receive_option_reply (ioc, opt, &reply, errp) <0 ) { +return -1 ; }if (reply.type !=NBD_REP_ACK ) { -error_setg (errp,"Server rejected request to start TLS %" PRIx32, - reply.type); -nbd_send_opt_abort (ioc); -return NULL ; +error_setg (errp,"Server rejected request for '%s' option: %" PRIx32, +nbd_opt_name (opt), reply.type); +if (abort_on_notsup) { +nbd_send_opt_abort (ioc); + } +return -1 ; }if (reply.length !=0 ) { -error_setg (errp,"Start TLS response was not zero %" PRIu32, - reply.length); -nbd_send_opt_abort (ioc); +error_setg (errp,"'%s' option response was not zero %" PRIu32, +nbd_opt_name (opt), reply.length); +if (abort_on_notsup) { +nbd_send_opt_abort (ioc); + } +return -1 ; + } + +TRACE ("%s 'option' approved" ,nbd_opt_name (opt)); +return 0 ; +} + +static QIOChannel *nbd_receive_starttls (QIOChannel *ioc, +QCryptoTLSCreds *tlscreds, +const char *hostname,Error **errp) +{ +QIOChannelTLS *tioc; +struct NBDTLSHandshakeData data = {0 }; + +if (nbd_receive_simple_option (ioc,NBD_OPT_STARTTLS , true, errp) <0 ) {return NULL ; } -TRACE ("TLS request approved, setting up TLS" ); +TRACE ("Setting up TLS" ); tioc =qio_channel_tls_new_client (ioc, tlscreds, hostname, errp);if (!tioc) {return NULL ;
nbd/nbd-internal.h +14 -0
@@ -96,6 +96,20 @@#define NBD_ENOSPC 28#define NBD_ESHUTDOWN 108 +static inline const char *nbd_opt_name (int opt) +{ +switch (opt) { +case NBD_OPT_EXPORT_NAME :return "export_name" ; +case NBD_OPT_ABORT :return "abort" ; +case NBD_OPT_LIST :return "list" ; +case NBD_OPT_PEEK_EXPORT :return "peek_export" ; +case NBD_OPT_STARTTLS :return "tls" ; +case NBD_OPT_STRUCTURED_REPLY :return "structured_reply" ; + } + +return "<unknown option>" ; +} +static inline ssize_t read_sync (QIOChannel *ioc,void *buffer,size_t size) {struct iovec iov = { .iov_base = buffer, .iov_len = size };
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
Split out
nbd_receive_simple_optionto be reused for structured reply option.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 54 +++++++++++++++++++++++++++++++++++------------------- nbd/nbd-internal.h | 14 ++++++++++++++ 2 files changed, 49 insertions(+), 19 deletions(-)+++
b/nbd/nbd-internal.h
nbd/nbd-internal.h line 102
Quoted 7 lines
nbd/nbd-internal.h +4
-0@@ -96,6 +96,20 @@#define NBD_ENOSPC 28#define NBD_ESHUTDOWN 108 +static inline const char *nbd_opt_name (int opt) +{ +switch (opt) { +case NBD_OPT_EXPORT_NAME :return "export_name" ;
Does this really get past checkpatch?
nbd/nbd-internal.h line 106
Quoted 4 lines
nbd/nbd-internal.h +4
-0+case NBD_OPT_ABORT :return "abort" ; +case NBD_OPT_LIST :return "list" ; +case NBD_OPT_PEEK_EXPORT :return "peek_export" ; +case NBD_OPT_STARTTLS :return "tls" ;
Why just 'tls' instead of 'starttls'?
nbd/nbd-internal.h line 110
Quoted 4 lines
nbd/nbd-internal.h +4
-0+case NBD_OPT_STRUCTURED_REPLY :return "structured_reply" ; + } + +return "<unknown option>" ;
Can you please consider making this include the %d representation of the unknown option; perhaps by snprintf'ing into static storage? While it is unlikely that a well-behaved server will respond to a client with an option the client doesn't recognize, it is much more likely that this reverse lookup function will be used in a server to respond to an unknown option from a client.
In fact, I might have split this into two patches: one providing nbd_opt_name() and using it throughout the code base where appropriate, and the other refactoring starttls in the client.
I'm not sure if the reverse lookup function needs to be inline in the header; it could reasonably live in nbd/common.c, particularly if you are going to take my advice to have it format a message for unknown values.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 25 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Split out
nbd_receive_simple_optionto be reused for structured reply option.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 54 +++++++++++++++++++++++++++++++++++------------------- nbd/nbd-internal.h | 14 ++++++++++++++ 2 files changed, 49 insertions(+), 19 deletions(-)+++
b/nbd/nbd-internal.hnbd/nbd-internal.h line 102
Quoted 7 lines
nbd/nbd-internal.h +4
-0@@ -96,6 +96,20 @@#define NBD_ENOSPC 28#define NBD_ESHUTDOWN 108 +static inline const char *nbd_opt_name (int opt) +{ +switch (opt) { +case NBD_OPT_EXPORT_NAME :return "export_name" ;Does this really get past checkpatch?
nbd/nbd-internal.h line 106
Quoted 4 lines
nbd/nbd-internal.h +4
-0+case NBD_OPT_ABORT :return "abort" ; +case NBD_OPT_LIST :return "list" ; +case NBD_OPT_PEEK_EXPORT :return "peek_export" ; +case NBD_OPT_STARTTLS :return "tls" ;
Quoted 8 lines
Why just 'tls' instead of 'starttls'?
nbd/nbd-internal.h line 110
Quoted 4 lines
nbd/nbd-internal.h +4
-0+case NBD_OPT_STRUCTURED_REPLY :return "structured_reply" ; + } + +return "<unknown option>" ;Can you please consider making this include the %d representation of the unknown option; perhaps by snprintf'ing into static storage? While it
Hmm.. The caller should free it in this case. Currently, the get_opt_name is called only on error path or if tracing enabled. How to achieve this with variable string return value not complicating the code a lot? When there is unknown message there should be other mechanism to inform user
Quoted 13 lines
is unlikely that a well-behaved server will respond to a client with an option the client doesn't recognize, it is much more likely that this reverse lookup function will be used in a server to respond to an unknown option from a client.
In fact, I might have split this into two patches: one providing
nbd_opt_name()and using it throughout the code base where appropriate, and the other refactoring starttls in the client.I'm not sure if the reverse lookup function needs to be inline in the header; it could reasonably live in
nbd/common.c, particularly if you are going to take my advice to have it format a message for unknown values.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Eric Blake
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
Split out
nbd_receive_simple_optionto be reused for structured reply option.+ return "<unknown option>";
Can you please consider making this include the %d representation of the unknown option; perhaps by snprintf'ing into static storage? While it
Hmm.. The caller should free it in this case.
Only if you print it into malloc'd space. I think that printing it into static storage may be sufficient (although then we have a race if more than one thread is trying to use that static storage at the same time - but do we ever have more than one thread trying to handle an error at the same time?).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 6 lines · Eric Blake
Quoted 5 lines · Vladimir Sementsov-Ogievskiy
Quoted 4 lines · Eric Blake
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Split out
nbd_receive_simple_optionto be reused for structured reply option.
+ return "<unknown option>";
Quoted 9 lines
Quoted 3 lines
Quoted 2 lines
Can you please consider making this include the %d representation of the unknown option; perhaps by snprintf'ing into static storage? While it
Hmm.. The caller should free it in this case.
Only if you print it into malloc'd space. I think that printing it into static storage may be sufficient (although then we have a race if more than one thread is trying to use that static storage at the same time - but do we ever have more than one thread trying to handle an error at the same time?).
This race would be if one thread decides to print two option names in one message. Or save one option in a var, then print other, then print var.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Eric Blake
Quoted 8 lines · Vladimir Sementsov-Ogievskiy
Quoted 7 lines · Eric Blake
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Split out
nbd_receive_simple_optionto be reused for structured reply option.
+ return "<unknown option>";Can you please consider making this include the %d representation of the
unknown option; perhaps by snprintf'ing into static storage? While itQuoted 7 lines
Quoted 1 line
Hmm.. The caller should free it in this case.
Only if you print it into malloc'd space. I think that printing it into static storage may be sufficient (although then we have a race if more than one thread is trying to use that static storage at the same time - but do we ever have more than one thread trying to handle an error at the same time?).
Quoted 2 lines
This race would be if one thread decides to print two option names in one message. Or save one option in a var, then print other, then print var.
NBD_OPT_ are supposed to be handled sequentially. The NBD spec does NOT allow for a single client to send a second NBD_OPT_ until after the first one has received a final response. So the only time we would have two threads dealing with things in nbd/client.c during handshake is if we have a single qemu process connecting as client to two different NBD servers in parallel, where both servers issue an otherwise unknown opt response at the same time, which I don't see as likely enough to be worth avoiding static storage.
If you're still worried about the race (I'm not), to the point that you don't want to use static storage just to avoid g_malloc(), then another option is to make nbd_opt_name() take an input parameter for a buffer and max size, and let the caller provide stack-based storage for computing the resulting message (similar to how strerror_r does things).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 04/18] nbd/client: refactor nbd_receive_starttls
Quoted 8 lines · Eric Blake
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines
Split out
nbd_receive_simple_optionto be reused for structured reply option.
+ return "<unknown option>";Can you please consider making this include the %d representation of the
unknown option; perhaps by snprintf'ing into static storage? While it
Quoted 5 lines
If you're still worried about the race (I'm not), to the point that you don't want to use static storage just to avoid
g_malloc(), then another option is to makenbd_opt_name()take an input parameter for a buffer and max size, and let the caller provide stack-based storage for computing the resulting message (similar to howstrerror_rdoes things).
Or, as long as the caller prints the value as well as the reverse-mapped name, that would work too. I'm going to add such a reverse-mapping for my NBD_INFO_* patches about to land later today, so I'll make my counter-proposal for NBD_OPT_* along those lines as part of my series.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 5/18 nbd/client: fix drop_sync
In reply to Vladimir Sementsov-OgievskiyComparison symbol is misused. It may lead to memory corruption.
--- nbd/client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
nbd/client.c +1 -1
@@ -94,7 +94,7 @@ static ssize_t drop_sync(QIOChannel *ioc, size_t size)char small[1024 ];char *buffer; - buffer =sizeof (small) < size ? small :g_malloc (MIN (65536 , size)); + buffer =sizeof (small) > size ? small :g_malloc (MIN (65536 , size));while (size >0 ) {ssize_t count =read_sync (ioc, buffer,MIN (65536 , size));
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 05/18] nbd/client: fix drop_sync
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Comparison symbol is misused. It may lead to memory corruption.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Adding qemu-stable; this needs to be back-ported, and can be applied independently from your series.
Quoted 15 lines
nbd/client.c +1
-1@@ -94,7 +94,7 @@ static ssize_t drop_sync(QIOChannel *ioc, size_t size)char small[1024 ];char *buffer; - buffer =sizeof (small) < size ? small :g_malloc (MIN (65536 , size)); + buffer =sizeof (small) > size ? small :g_malloc (MIN (65536 , size));while (size >0 ) {ssize_t count =read_sync (ioc, buffer,MIN (65536 , size));
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 05/18] nbd/client: fix drop_sync
Quoted 8 lines · Eric Blake
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Comparison symbol is misused. It may lead to memory corruption.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 4 lines
Adding qemu-stable; this needs to be back-ported, and can be applied independently from your series.
Reviewed-by Eric Blake <[email protected]>
For the record, this is now assigned CVE-2017-2630. My apologies for introducing the bug in the first place (commit 7d3123e). The maintainer may want to touch up the commit message to give those further details, since it is security-related.
Quoted 17 lines
Quoted 16 lines
nbd/client.c +1
-1@@ -94,7 +94,7 @@ static ssize_t drop_sync(QIOChannel *ioc, size_t size)char small[1024 ];char *buffer; - buffer =sizeof (small) < size ? small :g_malloc (MIN (65536 , size)); + buffer =sizeof (small) > size ? small :g_malloc (MIN (65536 , size));while (size >0 ) {ssize_t count =read_sync (ioc, buffer,MIN (65536 , size));
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 6/18 nbd/client: refactor drop_sync
In reply to Vladimir Sementsov-OgievskiyReturn 0 on success to simplify success checking.
--- nbd/client.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-)
nbd/client.c +19 -16
@@ -86,31 +86,34 @@ static QTAILQ_HEAD(, NBDExport) exports = QTAILQ_HEAD_INITIALIZER(exports);*/ -/* Discard length bytes from channel. Return -errno on failure, or -* the amount of bytes consumed. */ -static ssize_t drop_sync (QIOChannel *ioc,size_t size) +/* Discard length bytes from channel. +* Return 0 on success and -errno on fail. +*/ +static int drop_sync (QIOChannel *ioc,size_t size) { -ssize_t ret =0 ; +ssize_t ret;char small[1024 ];char *buffer; buffer =sizeof (small) > size ? small :g_malloc (MIN (65536 , size));while (size >0 ) { -ssize_t count =read_sync (ioc, buffer,MIN (65536 , size)); - -if (count <=0 ) { - goto cleanup; + ret =read_sync (ioc, buffer,MIN (65536 , size)); +if (ret ==0 ) { + ret = -EIO ; + } +if (ret <0 ) { +break ; } -assert (count <= size); - size -= count; - ret += count; + +assert (ret <= size); + size -= ret; } - cleanup:if (buffer != small) {g_free (buffer); } -return ret; + +return ret <0 ? ret :0 ; }/* Send an option request.
@@ -334,7 +337,7 @@ static int nbd_receive_list(QIOChannel *ioc, const char *want, bool *match,return -1 ; }if (namelen !=strlen (want)) { -if (drop_sync (ioc, len) != len) { +if (drop_sync (ioc, len) <0 ) {error_setg (errp,"failed to skip export name with wrong length" );nbd_send_opt_abort (ioc);return -1 ;
@@ -350,7 +353,7 @@ static int nbd_receive_list(QIOChannel *ioc, const char *want, bool *match, } name[namelen] ='\0' ; len -= namelen; -if (drop_sync (ioc, len) != len) { +if (drop_sync (ioc, len) <0 ) {error_setg (errp,"failed to read export description" );nbd_send_opt_abort (ioc);return -1 ;
@@ -635,7 +638,7 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, }TRACE ("Size is %" PRIu64", export flags %" PRIx16, *size, *flags); -if (zeroes &&drop_sync (ioc,124 ) !=124 ) { +if (zeroes &&drop_sync (ioc,124 ) <0 ) {error_setg (errp,"Failed to read reserved block" ); goto fail; }
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 06/18] nbd/client: refactor drop_sync
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Return 0 on success to simplify success checking.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-)
I'm not sure that this simplifies anything. You have a net addition in lines of code, so unless some later patch is improved because of this, I'm inclined to say this is needless churn.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 06/18] nbd/client: refactor drop_sync
Quoted 11 lines · Eric Blake
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Return 0 on success to simplify success checking.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-)I'm not sure that this simplifies anything. You have a net addition in lines of code, so unless some later patch is improved because of this, I'm inclined to say this is needless churn.
I just dislike duplicating information like "drop_sync(ioc, 124) != 124". In the code there is no place where positive and not equal to size return value actually handled. But it is not so important, if you are against i'll drop this, no problem.
One note: I don't have good understanding of the following: actually read can return positive value < queried size, which means that we should read again. But it is not handled in the code (handled, but just as an error), except drop_sync.. (With drop_sync it is side effect of using limited buffer size, yes?). Is it all ok?
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 06/18] nbd/client: refactor drop_sync
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
Quoted 11 lines · Eric Blake
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Return 0 on success to simplify success checking.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/client.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-)I'm not sure that this simplifies anything. You have a net addition in lines of code, so unless some later patch is improved because of this, I'm inclined to say this is needless churn.
Quoted 4 lines
I just dislike duplicating information like "
drop_sync(ioc, 124)!= 124". In the code there is no place where positive and not equal to size return value actually handled. But it is not so important, if you are against i'll drop this, no problem.
I think I agree with Vladimir.
Quoted 5 lines
One note: I don't have good understanding of the following: actually read can return positive value < queried size, which means that we should read again. But it is not handled in the code (handled, but just as an error), except
drop_sync.. (Withdrop_syncit is side effect of using limited buffer size, yes?). Is it all ok?
It is handled in nbd_wr_syncv.
Paolo
PATCH 7/18 nbd: Minimal structured read for client
In reply to Vladimir Sementsov-OgievskiyMinimal implementation: always send DF flag, to not deal with fragmented replies.
--- block/nbd-client.c | 47 +++++++++++---- block/nbd-client.h | 2 + include/block/nbd.h | 15 +++-- nbd/client.c | 170 ++++++++++++++++++++++++++++++++++++++++++++++------ qemu-nbd.c | 2 +- 5 files changed, 203 insertions(+), 33 deletions(-)
block/nbd-client.c +37 -10
@@ -180,13 +180,20 @@ static void nbd_co_receive_reply(NBDClientSession *s, *reply = s->reply;if (reply->handle != request->handle || !s->ioc) { + reply->simple = true; reply->error =EIO ; }else { -if (qiov && reply->error ==0 ) { - ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, - true); -if (ret != request->len) { - reply->error =EIO ; +if (qiov) { +if ((reply->simple ? reply->error ==0 : + reply->type ==NBD_REPLY_TYPE_OFFSET_DATA )) { + ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, + true); +if (ret != request->len) { + reply->error =EIO ; + } + }else if (!reply->simple && + reply->type ==NBD_REPLY_TYPE_OFFSET_HOLE ) { +qemu_iovec_memset (qiov,0 ,0 , request->len); } }
@@ -227,6 +234,7 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, .type = NBD_CMD_READ, .from = offset, .len = bytes, + .flags = client->structured_reply ? NBD_CMD_FLAG_DF : 0, };NBDReply reply;ssize_t ret;
@@ -237,12 +245,30 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset,nbd_coroutine_start (client, &request); ret =nbd_co_send_request (bs, &request,NULL );if (ret <0 ) { - reply.error = -ret; - }else { -nbd_co_receive_reply (client, &request, &reply, qiov); + goto out; } + +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) { + ret = -reply.error; + } + +if (!reply.simple) { +while (!(reply.flags &NBD_REPLY_FLAG_DONE )) { +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) { + ret = -reply.error; + } +if (reply.simple) { + ret = -EIO ; + goto out; + } + } + } + +out:nbd_coroutine_end (client, &request); -return -reply.error; +return ret; } int nbd_client_co_pwritev(BlockDriverState *bs, uint64_t offset,
@@ -408,7 +434,8 @@ int nbd_client_init(BlockDriverState *bs, &client->nbdflags, tlscreds, hostname, &client->ioc, - &client->size, errp); + &client->size, + &client->structured_reply, errp);if (ret <0 ) {logout ("Failed to negotiate with the NBD server\n" );return ret;
block/nbd-client.h +2 -0
@@ -32,6 +32,8 @@ typedef struct NBDClientSession {NBDReply reply;bool is_unix; + +bool structured_reply; } NBDClientSession;NBDClientSession *nbd_get_client_session (BlockDriverState *bs);
include/block/nbd.h +10 -5
@@ -57,11 +57,16 @@ struct NBDRequest { };typedef struct NBDRequest NBDRequest ; -struct NBDReply { +typedef struct NBDReply { +bool simple;uint64_t handle;uint32_t error; -}; -typedef struct NBDReply NBDReply ; + +uint16_t flags; +uint16_t type; +uint32_t length; +uint64_t offset; +}NBDReply ;struct NBDSimpleReply {/* uint32_t NBD_SIMPLE_REPLY_MAGIC */
@@ -169,10 +174,10 @@ ssize_t nbd_wr_syncv(QIOChannel *ioc,int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,Error **errp); +off_t *size,bool *structured_reply,Error **errp);int nbd_init (int fd,QIOChannelSocket *sioc,uint16_t flags,off_t size);ssize_t nbd_send_request (QIOChannel *ioc,NBDRequest *request); -ssize_t nbd_receive_reply (QIOChannel *ioc,NBDReply *reply); +int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply);int nbd_client (int fd);int nbd_disconnect (int fd);
nbd/client.c +153 -17
215 lines not on this page · raw message
qemu-nbd.c +1 -1
@@ -272,7 +272,7 @@ static void *nbd_client_thread(void *arg) ret =nbd_receive_negotiate (QIO_CHANNEL (sioc),NULL , &nbdflags,NULL ,NULL ,NULL , - &size, &local_error); + &size,NULL , &local_error);if (ret <0 ) {if (local_error) {error_report_err (local_error);
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 07/18] nbd: Minimal structured read for client
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation: always send DF flag, to not deal with fragmented replies.
This works well with your minimal server implementation, but I worry that it will cause us to fall over when talking to a fully-compliant server that chooses to send EOVERFLOW errors for any request larger than 64k when DF is set; it also makes it impossible to benefit from sparse reads. I guess that means we need to start thinking about followup patches to flush out our implementation. But maybe I can live with this patch as is, since the goal of your series was not so much the full power of structured reads, but getting to a point where we could use structured reply for block status, even if it means your client can only communicate with qemu-nbd as server for now, as long as we do get to the rest of the patches for a full-blown structured read.
Quoted 9 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd-client.c | 47 +++++++++++---- block/nbd-client.h | 2 + include/block/nbd.h | 15 +++-- nbd/client.c | 170 ++++++++++++++++++++++++++++++++++++++++++++++------ qemu-nbd.c | 2 +- 5 files changed, 203 insertions(+), 33 deletions(-)
Hmm - no change to the testsuite. Structured reads seems like the sort of thing that it would be nice to test with some canned server replies, particularly with server behavior that is permitted by the NBD protocol but does not happen by default in qemu-nbd.
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 184
Quoted 11 lines
block/nbd-client.c +1
-0@@ -180,13 +180,20 @@ static void nbd_co_receive_reply(NBDClientSession *s, *reply = s->reply;if (reply->handle != request->handle || !s->ioc) { + reply->simple = true; reply->error =EIO ;
I don't think this is quite right - by setting reply->simple to true, you are forcing the caller to treat this as the final packet related to this request->handle, even though that might not be the case.
As it is, I wonder if this code is correct, even before your patch - the server is allowed to give responses out-of-order (if we request multiple reads without waiting for the first response) - I don't see how setting reply->error to EIO if the request->handle indicates that we are receiving an out-of-order response to some other packet, but that our request is still awaiting traffic.
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 190
Quoted 11 lines
block/nbd-client.c +5
-5}else { -if (qiov && reply->error ==0 ) { - ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, - true); -if (ret != request->len) { - reply->error =EIO ; +if (qiov) { +if ((reply->simple ? reply->error ==0 : + reply->type ==NBD_REPLY_TYPE_OFFSET_DATA )) { + ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, + true);
This works only because you used the DF flag. If we allow fragmenting, then you have to be careful to write the reply into the correct offset of the iov.
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 197
Quoted 7 lines
block/nbd-client.c +6
-0+if (ret != request->len) { + reply->error =EIO ; + } + }else if (!reply->simple && + reply->type ==NBD_REPLY_TYPE_OFFSET_HOLE ) { +qemu_iovec_memset (qiov,0 ,0 , request->len); }
Up to here, you didn't do any inspection for NBD_REPLY_FLAG_DONE (so you don't know if this is the last packet the server is sending for this reqeust->handle), and didn't do any special casing for NBD_REPLY_TYPE_NONE or for the various error replies. I'm not sure if this will always do what you want. In fact, I'm not even sure if reply->error is set correctly for all structured packets.
Quoted 22 lines
block/nbd-client.c +5
-3}@@ -227,6 +234,7 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, .type = NBD_CMD_READ, .from = offset, .len = bytes, + .flags = client->structured_reply ? NBD_CMD_FLAG_DF : 0, };NBDReply reply;ssize_t ret;@@ -237,12 +245,30 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset,nbd_coroutine_start (client, &request); ret =nbd_co_send_request (bs, &request,NULL );if (ret <0 ) { - reply.error = -ret; - }else { -nbd_co_receive_reply (client, &request, &reply, qiov); + goto out; } + +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) {
block/nbd-client.c int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, · line 262
Quoted 10 lines
block/nbd-client.c +10
-0+ ret = -reply.error; + } + +if (!reply.simple) { +while (!(reply.flags &NBD_REPLY_FLAG_DONE )) { +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) { + ret = -reply.error; + } +if (reply.simple) {
Hmm. It looks like this part of the loop is only triggered if nbd_co_receive_reply() detects a handle mismatch and slams reply.simple to true. As long as we use the DF flag, it looks like the server should never send an error packet followed by a data packet, and your particular server implementation always set the DONE flag on the error packet, so it got past your testing. But if we don't rely on the DF flag, a server could reasonable send an ERROR_OFFSET packet for half the buffer, followed by a data packet for the other half of the buffer, which may wipe out reply.error from the error packet.
include/block/nbd.h
Quoted 40 lines
block/nbd-client.c +10
-2+ ret = -EIO ; + goto out; + } + } + } + +out:nbd_coroutine_end (client, &request); -return -reply.error; +return ret; } int nbd_client_co_pwritev(BlockDriverState *bs, uint64_t offset,@@ -408,7 +434,8 @@ int nbd_client_init(BlockDriverState *bs, &client->nbdflags, tlscreds, hostname, &client->ioc, - &client->size, errp); + &client->size, + &client->structured_reply, errp);if (ret <0 ) {logout ("Failed to negotiate with the NBD server\n" );return ret;block/nbd-client.h +2
-0@@ -32,6 +32,8 @@ typedef struct NBDClientSession {NBDReply reply;bool is_unix; + +bool structured_reply; } NBDClientSession;NBDClientSession *nbd_get_client_session (BlockDriverState *bs);include/block/nbd.h +0
-0
Can you add the use of an order file to list .h files first in your diffs? See
https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg00288.html for tips.
Quoted 7 lines
include/block/nbd.h +2
-1@@ -57,11 +57,16 @@ struct NBDRequest { };typedef struct NBDRequest NBDRequest ; -struct NBDReply { +typedef struct NBDReply { + bool simple;
include/block/nbd.h struct NBDRequest { · line 69
Quoted 10 lines
include/block/nbd.h +6
-2uint64_t handle;uint32_t error; -}; -typedef struct NBDReply NBDReply ; + +uint16_t flags; +uint16_t type; +uint32_t length; +uint64_t offset; +} NBDReply;
I don't know if this is the best way to represent things; I might have used a union type, since not all fields are valid in all reply packets.
Quoted 34 lines
include/block/nbd.h +2
-2struct NBDSimpleReply {/* uint32_t NBD_SIMPLE_REPLY_MAGIC */ @@ -169,10 +174,10 @@ ssize_t nbd_wr_syncv(QIOChannel *ioc,int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,Error **errp); +off_t *size,bool *structured_reply,Error **errp);int nbd_init (int fd,QIOChannelSocket *sioc,uint16_t flags,off_t size);ssize_t nbd_send_request (QIOChannel *ioc,NBDRequest *request); -ssize_t nbd_receive_reply (QIOChannel *ioc,NBDReply *reply); +int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply);int nbd_client (int fd);int nbd_disconnect (int fd);nbd/client.c +1
-2@@ -472,11 +472,10 @@ static QIOChannel *nbd_receive_starttls(QIOChannel *ioc,return QIO_CHANNEL (tioc); } -int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,Error **errp) +off_t *size,bool *structured_reply,Error **errp) {char buf[256 ];uint64_t magic, s;
nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 590
Quoted 8 lines
nbd/client.c +5
-0@@ -584,6 +583,12 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags,if (nbd_receive_query_exports (ioc, name, errp) <0 ) { goto fail; } + +if (structured_reply !=NULL ) { + *structured_reply = +nbd_receive_simple_option (ioc,NBD_OPT_STRUCTURED_REPLY , + false,NULL ) ==0 ;
Okay, you're allowing the server to reject the option, in which case we set structured_reply to false. But re-reading patch 4/18, nbd_receive_simple_option() can return -1 for multiple reasons, some of them where it is still in sync (for sending more options), but others where it is out of sync (such as failure to write, in which case the connection MUST be dropped rather than trying to carry on). I don't think this handles errors correctly, and therefore I'm not even sure that the refactoring in 4/18 is correct.
I think you may be better off with nbd_receive_simple_option() in 4/18 being tri-state: return -1 if the connection is unrecoverable (such as after a write or read error, where we must not send or receive any more data), 0 if the server replied with an error but the connection is still in sync for trying something else, and 1 if the server replies with success. Then this code should check if the return is < 0 (kill negotiation), == 0 (*structured_reply = false), or == 1 (*structured_reply = true).
nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 612
Quoted 10 lines
nbd/client.c +3
-0+ } }/* write the export name request */ if (nbd_send_option_request(ioc, NBD_OPT_EXPORT_NAME, -1, name,@@ -603,6 +608,14 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, goto fail; }be16_to_cpus (flags); + + if (!!structured_reply && *structured_reply &&
Why do you need !! to coerce to bool, when && also coerces to bool?
nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 616
Quoted 4 lines
nbd/client.c +4
-0+ !(*flags & NBD_CMD_FLAG_DF)) + { +error_setg (errp,"Structured reply is negotiated, " +"but DF flag is not." );
No trailing '.' for error_setg() messages.
Also, I'm not quite sure the NBD protocol allows an implementation that supports structured read but does not support the DF flag. Maybe that's an NBD spec bug that we should get clarified. (Ideally, the server always advertises the DF flag if structured replies are negotiated, because the common implementation of user-space handshake followed by kernel transmission phase works best when the already-existing ioctl(NBD_SET_FLAGS) can then be used to tell the kernel to use/expect structure replies.)
Quoted 31 lines
nbd/client.c +10
-7+ goto fail; + } }else if (magic ==NBD_CLIENT_MAGIC ) {uint32_t oldflags;@@ -790,20 +803,33 @@ ssize_t nbd_send_request(QIOChannel *ioc, NBDRequest *request)return 0 ; } -ssize_t nbd_receive_reply (QIOChannel *ioc,NBDReply *reply) +static inline int read_sync_check (QIOChannel *ioc,void *buffer,size_t size) { -uint8_t buf[NBD_REPLY_SIZE ]; -uint32_t magic;ssize_t ret; - ret =read_sync (ioc, buf,sizeof (buf)); + ret =read_sync (ioc, buffer, size);if (ret <0 ) {return ret; } - -if (ret !=sizeof (buf)) { +if (ret != size) {LOG ("read failed" ); -return -EINVAL ; +return -EIO ; + } + +return 0 ; +}
nbd/client.c ssize_t nbd_send_request(QIOChannel *ioc, NBDRequest *request) · line 830
Quoted 10 lines
nbd/client.c +10
-0+ +/* nbd_receive_simple_reply +* Read simple reply except magic field (which should be already read) +*/ +static int nbd_receive_simple_reply (QIOChannel *ioc,NBDReply *reply) +{ +uint8_t buf[NBD_REPLY_SIZE -4 ]; +ssize_t ret; + + ret =read_sync_check (ioc, buf,sizeof (buf));
This patch is fairly long; I might have split the creation and initial use of the read_sync_check() function into its own patch.
Quoted 12 lines
nbd/client.c +2
-3+if (ret <0 ) { +return ret; }/* Reply @@ -812,9 +838,124 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply)[ 7 .. 15] handle */ - magic =ldl_be_p (buf); - reply->error =ldl_be_p (buf +4 ); - reply->handle =ldq_be_p (buf +8 );
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 850
Quoted 10 lines
nbd/client.c +10
-0+ reply->error =ldl_be_p (buf); + reply->handle =ldq_be_p (buf +4 ); + +return 0 ; +} + +/* nbd_receive_structured_reply_chunk +* Read structured reply chunk except magic field (which should be already read) +* Data for NBD_REPLY_TYPE_OFFSET_DATA is not read too. +* Length field of reply out parameter corresponds to unread part of reply.
Awkward wording; maybe:
Read the header portion of a structured reply chunk (except for magic, which should already be read). On success, the length field of reply corresponds to number of bytes in the reply that still need to be read.
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 858
Quoted 8 lines
nbd/client.c +8
-0+*/ +static int nbd_receive_structured_reply_chunk (QIOChannel *ioc,NBDReply *reply) +{ +NBDStructuredReplyChunk chunk; +ssize_t ret; +uint16_t message_size; + + ret = read_sync_check(ioc, (uint8_t *)&chunk + sizeof(chunk.magic),
This looks like some ugly pointer math. I wonder if you'd be better served by having a separate packed struct without the magic field, particularly since you have to read the magic in first to decide whether to dispatch to simple/structured for the rest of the structure. In other words, note how patch 2/18 omits a uint32_t magic in NBDSimpleReply; maybe patch 3/18 could do the same with NBDStructuredReplyChunk. Then again, that would impact how you code things for subcasses like NBDStructuredRead, so I don't know how much churn to request here.
Quoted 14 lines
nbd/client.c +14
-0+ sizeof(chunk) - sizeof(chunk.magic)); +if (ret <0 ) { +return ret; + } + + reply->flags =be16_to_cpu (chunk.flags); + reply->type =be16_to_cpu (chunk.type); + reply->handle =be64_to_cpu (chunk.handle); + reply->length =be32_to_cpu (chunk.length); + +switch (reply->type) { +case NBD_REPLY_TYPE_NONE : +break ; +case NBD_REPLY_TYPE_OFFSET_DATA :
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 882
Quoted 10 lines
nbd/client.c +10
-0+case NBD_REPLY_TYPE_OFFSET_HOLE : + ret =read_sync_check (ioc, &reply->offset,sizeof (reply->offset)); +if (ret <0 ) { +return ret; + } +be64_to_cpus (&reply->offset); + reply->length -=sizeof (reply->offset); +break ; +case NBD_REPLY_TYPE_ERROR : +case NBD_REPLY_TYPE_ERROR_OFFSET :
Here, I think that you want:
default:
if (reply->type & 0x8000) {
rather than specific NBD_REPLY_TYPE_ERROR labels, so that you can gracefully handle ALL error packets sent by a server even if you haven't explicitly coded for them (a good server should probably not be sending an error packet you don't recognize, but the protocol also went to good efforts to describe a common behavior of all error packets).
/me reads on...
Quoted 20 lines
nbd/client.c +20
-0+ ret =read_sync_check (ioc, &reply->error,sizeof (reply->error)); +if (ret <0 ) { +return ret; + } +be32_to_cpus (&reply->error); + + ret =read_sync_check (ioc, &message_size,sizeof (message_size)); +if (ret <0 ) { +return ret; + } +be16_to_cpus (&message_size); + +if (message_size >0 ) { +/* TODO: provide error message to user */ + ret =drop_sync (ioc, message_size); +if (ret <0 ) { +return ret; + } + } +
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 912
Quoted 10 lines
nbd/client.c +10
-0+if (reply->type ==NBD_REPLY_TYPE_ERROR_OFFSET ) { +/* drop 64bit offset */ + ret =drop_sync (ioc,8 ); +if (ret <0 ) { +return ret; + } + } +break ; +default : +if (reply->type & (1 <<15 )) {
...oh, you did, mostly.
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 914
Quoted 2 lines
nbd/client.c +2
-0+/* unknown error */ + ret =drop_sync (ioc, reply->length);
Still, even if it is an unknown error, you should still be able to populate reply->error, rather than ignoring it.
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 919
Quoted 5 lines
nbd/client.c +5
-0+if (ret <0 ) { +return ret; + } + + reply->error =NBD_EINVAL ;
Meanwhile, you should also probably ensure that reply->error is non-zero even if the server was buggy and sent an error flag without telling you an error value (your choice of EINVAL seems reasonable).
Quoted 24 lines
nbd/client.c +24
-0+ reply->length =0 ; + }else { +/* unknown non-error reply type */ +return -EINVAL ; + } + } + +return 0 ; +} + +int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply) +{ +uint32_t magic; +int ret; + + ret =read_sync_check (ioc, &magic,sizeof (magic)); +if (ret <0 ) { +return ret; + } + +be32_to_cpus (&magic); + +switch (magic) { +case NBD_SIMPLE_REPLY_MAGIC :
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 953
Quoted 10 lines
nbd/client.c +10
-0+ reply->simple = true; + ret =nbd_receive_simple_reply (ioc, reply); +break ; +case NBD_STRUCTURED_REPLY_MAGIC : + reply->simple = false; + ret =nbd_receive_structured_reply_chunk (ioc, reply); +break ; +default : +LOG ("invalid magic (got 0x%" PRIx32")" , magic); +return -EINVAL ;
This part looks good.
qemu-nbd.c
Quoted 20 lines
nbd/client.c +5
-5+ } + +if (ret <0 ) { +return ret; + } reply->error =nbd_errno_to_system_errno (reply->error);@@ -827,10 +968,5 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply)", handle = %" PRIu64" }" , magic, reply->error, reply->handle); -if (magic !=NBD_SIMPLE_REPLY_MAGIC ) { -LOG ("invalid magic (got 0x%" PRIx32")" , magic); -return -EINVAL ; - }return 0 ; } -qemu-nbd.c +0
-0
Why are you deleting a blank line from the end of the file? Is that rebase churn that should have been avoided in an earlier patch? </me goes and reads> - oh, it's been like that since commit 798bfe000 created the file. Thanks for cleaning it up (and I'm surprised checkpatch doesn't flag it).
qemu-nbd.c static void *nbd_client_thread(void *arg) · line 275
Quoted 9 lines
index
c734f627b4..de0099e333100644qemu-nbd.c +1
-1@@ -272,7 +272,7 @@ static void *nbd_client_thread(void *arg) ret =nbd_receive_negotiate (QIO_CHANNEL (sioc),NULL , &nbdflags,NULL ,NULL ,NULL , - &size, &local_error); + &size,NULL , &local_error);
We could reasonably allow new-style structured reads when we're handling the entire client side; but I agree that when qemu-nbd is used as the handshaking phase before handing things over to the kernel that we can't request structured reads, at least not until a) the kernel nbd module implements structured read supports, and b) we have a way to tell that the kernel is willing to accept structured read negotiation (and that's where my earlier comments about the DF flag being a witness of structured reads comes into play).
Umm, how does this patch compile? You changed the signature of nbd_receive_negotiate() to add a parameter, but did not modify the call in block/nbd-client.c to pass the new parameter. (So far, I've been reviewing based just on email content; I also plan on applying and testing your patches before all is said and done, but I sometimes surprise myself with my ability to do a quality read-only review even without spending time compiling things).
qemu-nbd.c static void *nbd_client_thread(void *arg) · line 278
Quoted 4 lines
qemu-nbd.c +0
-0if (ret <0 ) {if (local_error) {error_report_err (local_error);
We'll definitely need a v2, but I'm grateful that you've tackled this work.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 07/18] nbd: Minimal structured read for client
Quoted 4 lines · Eric Blake
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation: always send DF flag, to not deal with fragmented replies.
Quoted 11 lines
This works well with your minimal server implementation, but I worry that it will cause us to fall over when talking to a fully-compliant server that chooses to send EOVERFLOW errors for any request larger than 64k when DF is set; it also makes it impossible to benefit from sparse reads. I guess that means we need to start thinking about followup patches to flush out our implementation. But maybe I can live with this patch as is, since the goal of your series was not so much the full power of structured reads, but getting to a point where we could use structured reply for block status, even if it means your client can only communicate with qemu-nbd as server for now, as long as we do get to the rest of the patches for a full-blown structured read.
Can you post a diff that expresses this as a comment? I'll squash the comment into this commit.
Quoted 10 lines
Quoted 9 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd-client.c | 47 +++++++++++---- block/nbd-client.h | 2 + include/block/nbd.h | 15 +++-- nbd/client.c | 170 ++++++++++++++++++++++++++++++++++++++++++++++------ qemu-nbd.c | 2 +- 5 files changed, 203 insertions(+), 33 deletions(-)
Quoted 4 lines
Hmm - no change to the testsuite. Structured reads seems like the sort of thing that it would be nice to test with some canned server replies, particularly with server behavior that is permitted by the NBD protocol but does not happen by default in qemu-nbd.
This would require implementing some kind of mock server support. That would be a very good thing but not something we have much infrastructure for (you could use either a real socket or a mock QIOChannel).
Thanks,
Paolo
Quoted 532 lines
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 184
Quoted 11 lines
block/nbd-client.c +1
-0@@ -180,13 +180,20 @@ static void nbd_co_receive_reply(NBDClientSession *s, *reply = s->reply;if (reply->handle != request->handle || !s->ioc) { + reply->simple = true; reply->error =EIO ;I don't think this is quite right - by setting
reply->simpleto true, you are forcing the caller to treat this as the final packet related to thisrequest->handle, even though that might not be the case.As it is, I wonder if this code is correct, even before your patch - the server is allowed to give responses out-of-order (if we request multiple reads without waiting for the first response) - I don't see how setting
reply->errorto EIO if therequest->handleindicates that we are receiving an out-of-order response to some other packet, but that our request is still awaiting traffic.block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 190
Quoted 11 lines
block/nbd-client.c +5
-5}else { -if (qiov && reply->error ==0 ) { - ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, - true); -if (ret != request->len) { - reply->error =EIO ; +if (qiov) { +if ((reply->simple ? reply->error ==0 : + reply->type ==NBD_REPLY_TYPE_OFFSET_DATA )) { + ret =nbd_wr_syncv (s->ioc, qiov->iov, qiov->niov, request->len, + true);This works only because you used the DF flag. If we allow fragmenting, then you have to be careful to write the reply into the correct offset of the iov.
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 197
Quoted 7 lines
block/nbd-client.c +6
-0+if (ret != request->len) { + reply->error =EIO ; + } + }else if (!reply->simple && + reply->type ==NBD_REPLY_TYPE_OFFSET_HOLE ) { +qemu_iovec_memset (qiov,0 ,0 , request->len); }Up to here, you didn't do any inspection for
NBD_REPLY_FLAG_DONE(so you don't know if this is the last packet the server is sending for thisreqeust->handle), and didn't do any special casing forNBD_REPLY_TYPE_NONEor for the various error replies. I'm not sure if this will always do what you want. In fact, I'm not even sure ifreply->erroris set correctly for all structured packets.Quoted 22 lines
block/nbd-client.c +5
-3}@@ -227,6 +234,7 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, .type = NBD_CMD_READ, .from = offset, .len = bytes, + .flags = client->structured_reply ? NBD_CMD_FLAG_DF : 0, };NBDReply reply;ssize_t ret;@@ -237,12 +245,30 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset,nbd_coroutine_start (client, &request); ret =nbd_co_send_request (bs, &request,NULL );if (ret <0 ) { - reply.error = -ret; - }else { -nbd_co_receive_reply (client, &request, &reply, qiov); + goto out; } + +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) {block/nbd-client.c int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, · line 262
Quoted 10 lines
block/nbd-client.c +10
-0+ ret = -reply.error; + } + +if (!reply.simple) { +while (!(reply.flags &NBD_REPLY_FLAG_DONE )) { +nbd_co_receive_reply (client, &request, &reply, qiov); +if (reply.error !=0 ) { + ret = -reply.error; + } +if (reply.simple) {Hmm. It looks like this part of the loop is only triggered if
nbd_co_receive_reply()detects a handle mismatch and slams reply.simple to true. As long as we use the DF flag, it looks like the server should never send an error packet followed by a data packet, and your particular server implementation always set the DONE flag on the error packet, so it got past your testing. But if we don't rely on the DF flag, a server could reasonable send anERROR_OFFSETpacket for half the buffer, followed by a data packet for the other half of the buffer, which may wipe out reply.error from the error packet.include/block/nbd.h
Quoted 40 lines
block/nbd-client.c +10
-2+ ret = -EIO ; + goto out; + } + } + } + +out:nbd_coroutine_end (client, &request); -return -reply.error; +return ret; } int nbd_client_co_pwritev(BlockDriverState *bs, uint64_t offset,@@ -408,7 +434,8 @@ int nbd_client_init(BlockDriverState *bs, &client->nbdflags, tlscreds, hostname, &client->ioc, - &client->size, errp); + &client->size, + &client->structured_reply, errp);if (ret <0 ) {logout ("Failed to negotiate with the NBD server\n" );return ret;block/nbd-client.h +2
-0@@ -32,6 +32,8 @@ typedef struct NBDClientSession {NBDReply reply;bool is_unix; + +bool structured_reply; } NBDClientSession;NBDClientSession *nbd_get_client_session (BlockDriverState *bs);include/block/nbd.h +0
-0Can you add the use of an order file to list .h files first in your diffs? See
https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg00288.html for tips.Quoted 7 lines
include/block/nbd.h +2
-1@@ -57,11 +57,16 @@ struct NBDRequest { };typedef struct NBDRequest NBDRequest ; -struct NBDReply { +typedef struct NBDReply { + bool simple;include/block/nbd.h struct NBDRequest { · line 69
Quoted 10 lines
include/block/nbd.h +6
-2uint64_t handle;uint32_t error; -}; -typedef struct NBDReply NBDReply ; + +uint16_t flags; +uint16_t type; +uint32_t length; +uint64_t offset; +} NBDReply;I don't know if this is the best way to represent things; I might have used a union type, since not all fields are valid in all reply packets.
Quoted 34 lines
include/block/nbd.h +2
-2struct NBDSimpleReply {/* uint32_t NBD_SIMPLE_REPLY_MAGIC */ @@ -169,10 +174,10 @@ ssize_t nbd_wr_syncv(QIOChannel *ioc,int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,Error **errp); +off_t *size,bool *structured_reply,Error **errp);int nbd_init (int fd,QIOChannelSocket *sioc,uint16_t flags,off_t size);ssize_t nbd_send_request (QIOChannel *ioc,NBDRequest *request); -ssize_t nbd_receive_reply (QIOChannel *ioc,NBDReply *reply); +int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply);int nbd_client (int fd);int nbd_disconnect (int fd);nbd/client.c +1
-2@@ -472,11 +472,10 @@ static QIOChannel *nbd_receive_starttls(QIOChannel *ioc,return QIO_CHANNEL (tioc); } -int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,Error **errp) +off_t *size,bool *structured_reply,Error **errp) {char buf[256 ];uint64_t magic, s;nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 590
Quoted 8 lines
nbd/client.c +5
-0@@ -584,6 +583,12 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags,if (nbd_receive_query_exports (ioc, name, errp) <0 ) { goto fail; } + +if (structured_reply !=NULL ) { + *structured_reply = +nbd_receive_simple_option (ioc,NBD_OPT_STRUCTURED_REPLY , + false,NULL ) ==0 ;Okay, you're allowing the server to reject the option, in which case we set
structured_replyto false. But re-reading patch 4/18,nbd_receive_simple_option()can return -1 for multiple reasons, some of them where it is still in sync (for sending more options), but others where it is out of sync (such as failure to write, in which case the connection MUST be dropped rather than trying to carry on). I don't think this handles errors correctly, and therefore I'm not even sure that the refactoring in 4/18 is correct.I think you may be better off with
nbd_receive_simple_option()in 4/18 being tri-state: return -1 if the connection is unrecoverable (such as after a write or read error, where we must not send or receive any more data), 0 if the server replied with an error but the connection is still in sync for trying something else, and 1 if the server replies with success. Then this code should check if the return is < 0 (kill negotiation), == 0 (*structured_reply= false), or == 1 (*structured_reply= true).nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 612
Quoted 10 lines
nbd/client.c +3
-0+ } }/* write the export name request */ if (nbd_send_option_request(ioc, NBD_OPT_EXPORT_NAME, -1, name,@@ -603,6 +608,14 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, goto fail; }be16_to_cpus (flags); + + if (!!structured_reply && *structured_reply &&Why do you need !! to coerce to bool, when && also coerces to bool?
nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 616
Quoted 4 lines
nbd/client.c +4
-0+ !(*flags & NBD_CMD_FLAG_DF)) + { +error_setg (errp,"Structured reply is negotiated, " +"but DF flag is not." );No trailing '.' for
error_setg()messages.Also, I'm not quite sure the NBD protocol allows an implementation that supports structured read but does not support the DF flag. Maybe that's an NBD spec bug that we should get clarified. (Ideally, the server always advertises the DF flag if structured replies are negotiated, because the common implementation of user-space handshake followed by kernel transmission phase works best when the already-existing
ioctl(NBD_SET_FLAGS)can then be used to tell the kernel to use/expect structure replies.)Quoted 31 lines
nbd/client.c +10
-7+ goto fail; + } }else if (magic ==NBD_CLIENT_MAGIC ) {uint32_t oldflags;@@ -790,20 +803,33 @@ ssize_t nbd_send_request(QIOChannel *ioc, NBDRequest *request)return 0 ; } -ssize_t nbd_receive_reply (QIOChannel *ioc,NBDReply *reply) +static inline int read_sync_check (QIOChannel *ioc,void *buffer,size_t size) { -uint8_t buf[NBD_REPLY_SIZE ]; -uint32_t magic;ssize_t ret; - ret =read_sync (ioc, buf,sizeof (buf)); + ret =read_sync (ioc, buffer, size);if (ret <0 ) {return ret; } - -if (ret !=sizeof (buf)) { +if (ret != size) {LOG ("read failed" ); -return -EINVAL ; +return -EIO ; + } + +return 0 ; +}nbd/client.c ssize_t nbd_send_request(QIOChannel *ioc, NBDRequest *request) · line 830
Quoted 10 lines
nbd/client.c +10
-0+ +/* nbd_receive_simple_reply +* Read simple reply except magic field (which should be already read) +*/ +static int nbd_receive_simple_reply (QIOChannel *ioc,NBDReply *reply) +{ +uint8_t buf[NBD_REPLY_SIZE -4 ]; +ssize_t ret; + + ret =read_sync_check (ioc, buf,sizeof (buf));This patch is fairly long; I might have split the creation and initial use of the
read_sync_check()function into its own patch.Quoted 12 lines
nbd/client.c +2
-3+if (ret <0 ) { +return ret; }/* Reply @@ -812,9 +838,124 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply)[ 7 .. 15] handle */ - magic =ldl_be_p (buf); - reply->error =ldl_be_p (buf +4 ); - reply->handle =ldq_be_p (buf +8 );nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 850
Quoted 10 lines
nbd/client.c +10
-0+ reply->error =ldl_be_p (buf); + reply->handle =ldq_be_p (buf +4 ); + +return 0 ; +} + +/* nbd_receive_structured_reply_chunk +* Read structured reply chunk except magic field (which should be already read) +* Data for NBD_REPLY_TYPE_OFFSET_DATA is not read too. +* Length field of reply out parameter corresponds to unread part of reply. Awkward wording; maybe:
Read the header portion of a structured reply chunk (except for magic, which should already be read). On success, the length field of reply corresponds to number of bytes in the reply that still need to be read.
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 858
Quoted 8 lines
nbd/client.c +8
-0+*/ +static int nbd_receive_structured_reply_chunk (QIOChannel *ioc,NBDReply *reply) +{ +NBDStructuredReplyChunk chunk; +ssize_t ret; +uint16_t message_size; + + ret = read_sync_check(ioc, (uint8_t *)&chunk + sizeof(chunk.magic),This looks like some ugly pointer math. I wonder if you'd be better served by having a separate packed struct without the magic field, particularly since you have to read the magic in first to decide whether to dispatch to simple/structured for the rest of the structure. In other words, note how patch 2/18 omits a
uint32_tmagic in NBDSimpleReply; maybe patch 3/18 could do the same with NBDStructuredReplyChunk. Then again, that would impact how you code things for subcasses like NBDStructuredRead, so I don't know how much churn to request here.Quoted 14 lines
nbd/client.c +14
-0+ sizeof(chunk) - sizeof(chunk.magic)); +if (ret <0 ) { +return ret; + } + + reply->flags =be16_to_cpu (chunk.flags); + reply->type =be16_to_cpu (chunk.type); + reply->handle =be64_to_cpu (chunk.handle); + reply->length =be32_to_cpu (chunk.length); + +switch (reply->type) { +case NBD_REPLY_TYPE_NONE : +break ; +case NBD_REPLY_TYPE_OFFSET_DATA :nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 882
Quoted 10 lines
nbd/client.c +10
-0+case NBD_REPLY_TYPE_OFFSET_HOLE : + ret =read_sync_check (ioc, &reply->offset,sizeof (reply->offset)); +if (ret <0 ) { +return ret; + } +be64_to_cpus (&reply->offset); + reply->length -=sizeof (reply->offset); +break ; +case NBD_REPLY_TYPE_ERROR : +case NBD_REPLY_TYPE_ERROR_OFFSET :Here, I think that you want:
default:
if (reply->type&0x8000) {rather than specific
NBD_REPLY_TYPE_ERRORlabels, so that you can gracefully handle ALL error packets sent by a server even if you haven't explicitly coded for them (a good server should probably not be sending an error packet you don't recognize, but the protocol also went to good efforts to describe a common behavior of all error packets)./me reads on...
Quoted 20 lines
nbd/client.c +20
-0+ ret =read_sync_check (ioc, &reply->error,sizeof (reply->error)); +if (ret <0 ) { +return ret; + } +be32_to_cpus (&reply->error); + + ret =read_sync_check (ioc, &message_size,sizeof (message_size)); +if (ret <0 ) { +return ret; + } +be16_to_cpus (&message_size); + +if (message_size >0 ) { +/* TODO: provide error message to user */ + ret =drop_sync (ioc, message_size); +if (ret <0 ) { +return ret; + } + } +nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 912
Quoted 10 lines
nbd/client.c +10
-0+if (reply->type ==NBD_REPLY_TYPE_ERROR_OFFSET ) { +/* drop 64bit offset */ + ret =drop_sync (ioc,8 ); +if (ret <0 ) { +return ret; + } + } +break ; +default : +if (reply->type & (1 <<15 )) {...oh, you did, mostly.
nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 914
Quoted 2 lines
nbd/client.c +2
-0+/* unknown error */ + ret =drop_sync (ioc, reply->length);Still, even if it is an unknown error, you should still be able to populate
reply->error, rather than ignoring it.nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 919
Quoted 5 lines
nbd/client.c +5
-0+if (ret <0 ) { +return ret; + } + + reply->error =NBD_EINVAL ;Meanwhile, you should also probably ensure that
reply->erroris non-zero even if the server was buggy and sent an error flag without telling you an error value (your choice of EINVAL seems reasonable).Quoted 24 lines
nbd/client.c +24
-0+ reply->length =0 ; + }else { +/* unknown non-error reply type */ +return -EINVAL ; + } + } + +return 0 ; +} + +int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply) +{ +uint32_t magic; +int ret; + + ret =read_sync_check (ioc, &magic,sizeof (magic)); +if (ret <0 ) { +return ret; + } + +be32_to_cpus (&magic); + +switch (magic) { +case NBD_SIMPLE_REPLY_MAGIC :nbd/client.c ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply) · line 953
Quoted 10 lines
nbd/client.c +10
-0+ reply->simple = true; + ret =nbd_receive_simple_reply (ioc, reply); +break ; +case NBD_STRUCTURED_REPLY_MAGIC : + reply->simple = false; + ret =nbd_receive_structured_reply_chunk (ioc, reply); +break ; +default : +LOG ("invalid magic (got 0x%" PRIx32")" , magic); +return -EINVAL ;This part looks good.
qemu-nbd.c
Quoted 20 lines
nbd/client.c +5
-5+ } + +if (ret <0 ) { +return ret; + } reply->error =nbd_errno_to_system_errno (reply->error);@@ -827,10 +968,5 @@ ssize_t nbd_receive_reply(QIOChannel *ioc, NBDReply *reply)", handle = %" PRIu64" }" , magic, reply->error, reply->handle); -if (magic !=NBD_SIMPLE_REPLY_MAGIC ) { -LOG ("invalid magic (got 0x%" PRIx32")" , magic); -return -EINVAL ; - }return 0 ; } -qemu-nbd.c +0
-0Why are you deleting a blank line from the end of the file? Is that rebase churn that should have been avoided in an earlier patch? </me goes and reads> - oh, it's been like that since commit
798bfe000created the file. Thanks for cleaning it up (and I'm surprised checkpatch doesn't flag it).qemu-nbd.c static void *nbd_client_thread(void *arg) · line 275
Quoted 9 lines
index
c734f627b4..de0099e333100644qemu-nbd.c +1
-1@@ -272,7 +272,7 @@ static void *nbd_client_thread(void *arg) ret =nbd_receive_negotiate (QIO_CHANNEL (sioc),NULL , &nbdflags,NULL ,NULL ,NULL , - &size, &local_error); + &size,NULL , &local_error);We could reasonably allow new-style structured reads when we're handling the entire client side; but I agree that when qemu-nbd is used as the handshaking phase before handing things over to the kernel that we can't request structured reads, at least not until a) the kernel nbd module implements structured read supports, and b) we have a way to tell that the kernel is willing to accept structured read negotiation (and that's where my earlier comments about the DF flag being a witness of structured reads comes into play).
Umm, how does this patch compile? You changed the signature of
nbd_receive_negotiate()to add a parameter, but did not modify the call inblock/nbd-client.cto pass the new parameter. (So far, I've been reviewing based just on email content; I also plan on applying and testing your patches before all is said and done, but I sometimes surprise myself with my ability to do a quality read-only review even without spending time compiling things).qemu-nbd.c static void *nbd_client_thread(void *arg) · line 278
Quoted 4 lines
qemu-nbd.c +0
-0if (ret <0 ) {if (local_error) {error_report_err (local_error);We'll definitely need a v2, but I'm grateful that you've tackled this work.
Re: [Qemu-devel] [PATCH 07/18] nbd: Minimal structured read for client
In reply to Eric BlakeQuoted 42 lines · Eric Blake
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation: always send DF flag, to not deal with fragmented replies.
This works well with your minimal server implementation, but I worry that it will cause us to fall over when talking to a fully-compliant server that chooses to send EOVERFLOW errors for any request larger than 64k when DF is set; it also makes it impossible to benefit from sparse reads. I guess that means we need to start thinking about followup patches to flush out our implementation. But maybe I can live with this patch as is, since the goal of your series was not so much the full power of structured reads, but getting to a point where we could use structured reply for block status, even if it means your client can only communicate with qemu-nbd as server for now, as long as we do get to the rest of the patches for a full-blown structured read.
Quoted 8 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd-client.c | 47 +++++++++++---- block/nbd-client.h | 2 + include/block/nbd.h | 15 +++-- nbd/client.c | 170 ++++++++++++++++++++++++++++++++++++++++++++++------ qemu-nbd.c | 2 +- 5 files changed, 203 insertions(+), 33 deletions(-)Hmm - no change to the testsuite. Structured reads seems like the sort of thing that it would be nice to test with some canned server replies, particularly with server behavior that is permitted by the NBD protocol but does not happen by default in qemu-nbd.
block/nbd-client.c static void nbd_co_receive_reply(NBDClientSession *s, · line 184
Quoted 10 lines
block/nbd-client.c +1
-0@@ -180,13 +180,20 @@ static void nbd_co_receive_reply(NBDClientSession *s, *reply = s->reply;if (reply->handle != request->handle || !s->ioc) { + reply->simple = true; reply->error =EIO ;I don't think this is quite right - by setting
reply->simpleto true, you are forcing the caller to treat this as the final packet related to thisrequest->handle, even though that might not be the case.
Quoted 6 lines
As it is, I wonder if this code is correct, even before your patch - the server is allowed to give responses out-of-order (if we request multiple reads without waiting for the first response) - I don't see how setting
reply->errorto EIO if therequest->handleindicates that we are receiving an out-of-order response to some other packet, but that our request is still awaiting traffic.
Hmm, looks like it should initiate disconnect instead of just reporting error to io operation caller.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH 07/18] nbd: Minimal structured read for client
Quoted 45 lines · Vladimir Sementsov-Ogievskiy
Quoted 44 lines · Eric Blake
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Minimal implementation: always send DF flag, to not deal with fragmented replies.
This works well with your minimal server implementation, but I worry that it will cause us to fall over when talking to a fully-compliant server that chooses to send EOVERFLOW errors for any request larger than 64k when DF is set; it also makes it impossible to benefit from sparse reads. I guess that means we need to start thinking about followup patches to flush out our implementation. But maybe I can live with this patch as is, since the goal of your series was not so much the full power of structured reads, but getting to a point where we could use structured reply for block status, even if it means your client can only communicate with qemu-nbd as server for now, as long as we do get to the rest of the patches for a full-blown structured read.
Quoted 9 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd-client.c | 47 +++++++++++---- block/nbd-client.h | 2 + include/block/nbd.h | 15 +++-- nbd/client.c | 170 ++++++++++++++++++++++++++++++++++++++++++++++------ qemu-nbd.c | 2 +- 5 files changed, 203 insertions(+), 33 deletions(-)Hmm - no change to the testsuite. Structured reads seems like the sort of thing that it would be nice to test with some canned server replies, particularly with server behavior that is permitted by the NBD protocol but does not happen by default in qemu-nbd.
block/nbd-client.c static void · line 185
Quoted 11 lines
block/nbd-client.c +1
-0@@ -180,13 +180,20 @@ static voidnbd_co_receive_reply (NBDClientSession *s, *reply = s->reply;if (reply->handle != request->handle || !s->ioc) { + reply->simple = true; reply->error =EIO ;I don't think this is quite right - by setting
reply->simpleto true, you are forcing the caller to treat this as the final packet related to thisrequest->handle, even though that might not be the case.
Quoted 10 lines
Quoted 6 lines
As it is, I wonder if this code is correct, even before your patch - the server is allowed to give responses out-of-order (if we request multiple reads without waiting for the first response) - I don't see how setting
reply->errorto EIO if therequest->handleindicates that we are receiving an out-of-order response to some other packet, but that our request is still awaiting traffic.Hmm, looks like it should initiate disconnect instead of just reporting error to io operation caller.
Also, nbd_co_send_request errors are not handled but just returned to the caller. Shouldn't the first
error on socket io initiate disconnect? I think, only the io errors, transferred from nbd-export block device
should be just returned to bdrv_{io} caller. NBD-protocol related errors (invalid handles, etc.) should initiate
disconnect, so that current and all future bdrv_{io}'s from client return -EIO.
Signature · 2 lines
Best regards, Vladimir
PATCH 8/18 hbitmap: add next_zero function
In reply to Vladimir Sementsov-OgievskiyThe function searches for next zero bit.
Also add interface for BdrvDirtyBitmap.
--- block/dirty-bitmap.c | 5 +++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ util/hbitmap.c | 26 ++++++++++++++++++++++++++ 4 files changed, 41 insertions(+)
block/dirty-bitmap.c +5 -0
@@ -533,3 +533,8 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap) {return hbitmap_count (bitmap->meta); } + +int64_t bdrv_dirty_bitmap_next_zero (BdrvDirtyBitmap *bitmap,uint64_t start) +{ +return hbitmap_next_zero (bitmap->bitmap, start); +}
include/block/dirty-bitmap.h +2 -0
@@ -72,4 +72,6 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap, bool finish);void bdrv_dirty_bitmap_deserialize_finish (BdrvDirtyBitmap *bitmap); +int64_t bdrv_dirty_bitmap_next_zero (BdrvDirtyBitmap *bitmap,uint64_t start); +#endif
include/qemu/hbitmap.h +8 -0
@@ -257,6 +257,14 @@ void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first);*/ unsigned long hbitmap_iter_skip_words (HBitmapIter *hbi); +/* hbitmap_next_zero: +* @hb: The HBitmap to operate on +* @start: The bit to start from. +* +* Find next not dirty bit. +*/ +int64_t hbitmap_next_zero (const HBitmap *hb,uint64_t start); +/* hbitmap_create_meta: * Create a "meta" hbitmap to track dirtiness of the bits in this HBitmap. * The caller owns the created bitmap and must call hbitmap_free_meta(hb) to
util/hbitmap.c +26 -0
@@ -166,6 +166,32 @@ void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first) } } +int64_t hbitmap_next_zero (const HBitmap *hb,uint64_t start) +{ +size_t pos = (start >> hb->granularity) >>BITS_PER_LEVEL ; +unsigned long *last_lev = hb->levels[HBITMAP_LEVELS -1 ]; +uint64_t sz = hb->sizes[HBITMAP_LEVELS -1 ]; +unsigned long cur = last_lev[pos]; +unsigned start_bit_offset = + (start >> hb->granularity) & (BITS_PER_LONG -1 ); + cur |= (1UL << start_bit_offset) -1 ; + +if (cur == (unsigned long )-1 ) { +do { + pos++; + }while (pos < sz && last_lev[pos] == (unsigned long )-1 ); + +if (pos >= sz) { +return -1 ; + } + + cur = last_lev[pos]; + } + +return MIN ((pos <<BITS_PER_LEVEL ) +ctol (cur), hb->size -1 ) + << hb->granularity; +} +bool hbitmap_empty (const HBitmap *hb) {return hb->count ==0 ;
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 08/18] hbitmap: add next_zero function
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
The function searches for next zero bit.
Also add interface for BdrvDirtyBitmap.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 5 +++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ util/hbitmap.c | 26 ++++++++++++++++++++++++++ 4 files changed, 41 insertions(+)
It would be nice to enhance the testsuite to cover hbitmap_next_zero().
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 08/18] hbitmap: add next_zero function
Quoted 13 lines · Eric Blake
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
The function searches for next zero bit.
Also add interface for BdrvDirtyBitmap.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 5 +++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ util/hbitmap.c | 26 ++++++++++++++++++++++++++ 4 files changed, 41 insertions(+)
Quoted 2 lines
It would be nice to enhance the testsuite to cover
hbitmap_next_zero().
Agreed.
Paolo
PATCH 9/18 block/dirty-bitmap: add bdrv_dirty_bitmap_next()
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 7 +++++++ include/block/dirty-bitmap.h | 3 +++ 2 files changed, 10 insertions(+)
block/dirty-bitmap.c +7 -0
@@ -538,3 +538,10 @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t start) {return hbitmap_next_zero (bitmap->bitmap, start); } + +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap) +{ +return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) : +QLIST_NEXT (bitmap, list); +}
include/block/dirty-bitmap.h +3 -0
@@ -74,4 +74,7 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);int64_t bdrv_dirty_bitmap_next_zero (BdrvDirtyBitmap *bitmap,uint64_t start); +BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs, +BdrvDirtyBitmap *bitmap); +#endif
Signature · 1 line
2.11.0
PATCH 10/18 block/dirty-bitmap: add bdrv_load_dirty_bitmap
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 53 ++++++++++++++++++++++++++++++++++++++++++++ include/block/block_int.h | 4 ++++ include/block/dirty-bitmap.h | 3 +++ 3 files changed, 60 insertions(+)
block/dirty-bitmap.c +53 -0
@@ -545,3 +545,56 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) :QLIST_NEXT (bitmap, list); } + +typedef struct BDRVLoadBitmapCo { +BlockDriverState *bs; +const char *name; +Error **errp; +BdrvDirtyBitmap *ret; +bool in_progress; +}BDRVLoadBitmapCo ; + +static void bdrv_load_dity_bitmap_co_entry (void *opaque) +{ +BDRVLoadBitmapCo *lbco = opaque; +BlockDriver *drv = lbco->bs->drv; + +if (!!drv && !!drv->bdrv_dirty_bitmap_load) { + lbco->ret = drv->bdrv_dirty_bitmap_load (lbco->bs, lbco->name, + lbco->errp); + }else if (lbco->bs->file) { +BlockDriverState *bs = lbco->bs; + lbco->bs = lbco->bs->file->bs; +bdrv_load_dity_bitmap_co_entry (lbco); +if (lbco->ret !=NULL ) { +QLIST_REMOVE (lbco->ret, list); +QLIST_INSERT_HEAD (&bs->dirty_bitmaps, lbco->ret, list); + } + }else { + lbco->ret =NULL ; + } + + lbco->in_progress = false; +} + +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp) +{ +Coroutine *co; +BDRVLoadBitmapCo lbco = { + .bs = bs, + .name = name, + .errp = errp, + .in_progress = true + }; + +if (qemu_in_coroutine ()) { +bdrv_load_dity_bitmap_co_entry (&lbco); + }else { + co =qemu_coroutine_create (bdrv_load_dity_bitmap_co_entry, &lbco); +qemu_coroutine_enter (co); +BDRV_POLL_WHILE (bs, lbco.in_progress); + } + +return lbco.ret; +}
include/block/block_int.h +4 -0
@@ -222,6 +222,10 @@ struct BlockDriver {int (*bdrv_get_info)(BlockDriverState *bs,BlockDriverInfo *bdi); ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs); +BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs, +const char *name, +Error **errp); + int coroutine_fn (*bdrv_save_vmstate)(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos);
include/block/dirty-bitmap.h +3 -0
@@ -77,4 +77,7 @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t start);BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs,BdrvDirtyBitmap *bitmap); +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp); +#endif
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 10/18] block/dirty-bitmap: add bdrv_load_dirty_bitmap
Quoted 15 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 53 ++++++++++++++++++++++++++++++++++++++++++++ include/block/block_int.h | 4 ++++ include/block/dirty-bitmap.h | 3 +++ 3 files changed, 60 insertions(+)block/dirty-bitmap.c +0
-0@@ -545,3 +545,56 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) :QLIST_NEXT (bitmap, list); }
block/dirty-bitmap.c BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs, · line 557
Quoted 10 lines
block/dirty-bitmap.c +10
-0+ +typedef struct BDRVLoadBitmapCo { +BlockDriverState *bs; +const char *name; +Error **errp; +BdrvDirtyBitmap *ret; +bool in_progress; +}BDRVLoadBitmapCo ; + +static void bdrv_load_dity_bitmap_co_entry (void *opaque)
Add coroutine_fn?
Quoted 73 lines
block/dirty-bitmap.c +43
-0+{ +BDRVLoadBitmapCo *lbco = opaque; +BlockDriver *drv = lbco->bs->drv; + +if (!!drv && !!drv->bdrv_dirty_bitmap_load) { + lbco->ret = drv->bdrv_dirty_bitmap_load (lbco->bs, lbco->name, + lbco->errp); + }else if (lbco->bs->file) { +BlockDriverState *bs = lbco->bs; + lbco->bs = lbco->bs->file->bs; +bdrv_load_dity_bitmap_co_entry (lbco); +if (lbco->ret !=NULL ) { +QLIST_REMOVE (lbco->ret, list); +QLIST_INSERT_HEAD (&bs->dirty_bitmaps, lbco->ret, list); + } + }else { + lbco->ret =NULL ; + } + + lbco->in_progress = false; +} + +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp) +{ +Coroutine *co; +BDRVLoadBitmapCo lbco = { + .bs = bs, + .name = name, + .errp = errp, + .in_progress = true + }; + +if (qemu_in_coroutine ()) { +bdrv_load_dity_bitmap_co_entry (&lbco); + }else { + co =qemu_coroutine_create (bdrv_load_dity_bitmap_co_entry, &lbco); +qemu_coroutine_enter (co); +BDRV_POLL_WHILE (bs, lbco.in_progress); + } + +return lbco.ret; +}include/block/block_int.h +4
-0@@ -222,6 +222,10 @@ struct BlockDriver {int (*bdrv_get_info)(BlockDriverState *bs,BlockDriverInfo *bdi); ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs); +BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs, +const char *name, +Error **errp); + int coroutine_fn (*bdrv_save_vmstate)(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos);include/block/dirty-bitmap.h +3
-0@@ -77,4 +77,7 @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t start);BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs,BdrvDirtyBitmap *bitmap); +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp); +#endif Signature · 1 line
2.11.0
Otherwise looks good!
Fam
Re: [Qemu-devel] [PATCH 10/18] block/dirty-bitmap: add bdrv_load_dirty_bitmap
In reply to Vladimir Sementsov-OgievskiyQuoted 40 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/dirty-bitmap.c | 53 ++++++++++++++++++++++++++++++++++++++++++++ include/block/block_int.h | 4 ++++ include/block/dirty-bitmap.h | 3 +++ 3 files changed, 60 insertions(+)block/dirty-bitmap.c +25
-0@@ -545,3 +545,56 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,return bitmap ==NULL ?QLIST_FIRST (&bs->dirty_bitmaps) :QLIST_NEXT (bitmap, list); } + +typedef struct BDRVLoadBitmapCo { +BlockDriverState *bs; +const char *name; +Error **errp; +BdrvDirtyBitmap *ret; +bool in_progress; +}BDRVLoadBitmapCo ; + +static void bdrv_load_dity_bitmap_co_entry (void *opaque) +{ +BDRVLoadBitmapCo *lbco = opaque; +BlockDriver *drv = lbco->bs->drv; + +if (!!drv && !!drv->bdrv_dirty_bitmap_load) { + lbco->ret = drv->bdrv_dirty_bitmap_load (lbco->bs, lbco->name, + lbco->errp); + }else if (lbco->bs->file) { +BlockDriverState *bs = lbco->bs; + lbco->bs = lbco->bs->file->bs; +bdrv_load_dity_bitmap_co_entry (lbco); +if (lbco->ret !=NULL ) { +QLIST_REMOVE (lbco->ret, list); +QLIST_INSERT_HEAD (&bs->dirty_bitmaps, lbco->ret, list); + }
block/dirty-bitmap.c BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs, · line 582
Quoted 10 lines
block/dirty-bitmap.c +10
-0+ }else { + lbco->ret =NULL ; + } + + lbco->in_progress = false; +} + +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp) +{
I think that you'd better check bs->drv here and not call coroutine in this case at all.
Quoted 45 lines
block/dirty-bitmap.c +18
-0+Coroutine *co; +BDRVLoadBitmapCo lbco = { + .bs = bs, + .name = name, + .errp = errp, + .in_progress = true + }; + +if (qemu_in_coroutine ()) { +bdrv_load_dity_bitmap_co_entry (&lbco); + }else { + co =qemu_coroutine_create (bdrv_load_dity_bitmap_co_entry, &lbco); +qemu_coroutine_enter (co); +BDRV_POLL_WHILE (bs, lbco.in_progress); + } + +return lbco.ret; +}include/block/block_int.h +4
-0@@ -222,6 +222,10 @@ struct BlockDriver {int (*bdrv_get_info)(BlockDriverState *bs,BlockDriverInfo *bdi); ImageInfoSpecific *(*bdrv_get_specific_info)(BlockDriverState *bs); +BdrvDirtyBitmap *(*bdrv_dirty_bitmap_load)(BlockDriverState *bs, +const char *name, +Error **errp); + int coroutine_fn (*bdrv_save_vmstate)(BlockDriverState *bs,QEMUIOVector *qiov,int64_t pos);include/block/dirty-bitmap.h +3
-0@@ -77,4 +77,7 @@ int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t start);BdrvDirtyBitmap *bdrv_dirty_bitmap_next (BlockDriverState *bs,BdrvDirtyBitmap *bitmap); +BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name, +Error **errp); +#endif
PATCH 11/18 nbd: BLOCK_STATUS for bitmap export: server part
In reply to Vladimir Sementsov-OgievskiyOnly one meta context type is defined: qemu-bitmap:<bitmap-name>. Maximum one query is allowed for NBD_OPT_{SET,LIST}_META_CONTEXT, NBD_REP_ERR_TOO_BIG is returned otherwise.
--- include/block/nbd.h | 15 ++ nbd/nbd-internal.h | 6 + nbd/server.c | 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 466 insertions(+)
include/block/nbd.h +15 -0
@@ -94,6 +94,16 @@ typedef struct NBDStructuredError {uint16_t message_length; }QEMU_PACKED NBDStructuredError; +typedef struct NBDStructuredMeta { +NBDStructuredReplyChunk h; +uint32_t context_id; +} QEMU_PACKED NBDStructuredMeta; + +typedef struct NBDExtent { +uint32_t length; +uint32_t flags; +} QEMU_PACKED NBDExtent; +/* Transmission (export) flags: sent from server to client during handshake, but describe what will happen during transmission */ #define NBD_FLAG_HAS_FLAGS (1 << 0)/* Flags are there */
@@ -120,6 +130,7 @@ typedef struct NBDStructuredError {#define NBD_REP_ACK (1)/* Data sending finished. */ #define NBD_REP_SERVER (2)/* Export description. */ +#define NBD_REP_META_CONTEXT (4)#define NBD_REP_ERR_UNSUP NBD_REP_ERR(1)/* Unknown option */ #define NBD_REP_ERR_POLICY NBD_REP_ERR(2)/* Server denied */
@@ -127,6 +138,8 @@ typedef struct NBDStructuredError {#define NBD_REP_ERR_PLATFORM NBD_REP_ERR(4)/* Not compiled in */ #define NBD_REP_ERR_TLS_REQD NBD_REP_ERR(5)/* TLS required */ #define NBD_REP_ERR_SHUTDOWN NBD_REP_ERR(7)/* Server shutting down */ +#define NBD_REP_ERR_TOO_BIG NBD_REP_ERR(9)/* The request or the reply is +too large to process */ /* Request flags, sent from client to server during transmission phase */ #define NBD_CMD_FLAG_FUA (1 << 0)/* 'force unit access' during write */
@@ -142,6 +155,7 @@ enum {NBD_CMD_TRIM =4 ,/* 5 reserved for failed experiment NBD_CMD_CACHE */ NBD_CMD_WRITE_ZEROES =6 , + NBD_CMD_BLOCK_STATUS = 7 };#define NBD_DEFAULT_PORT 10809
@@ -163,6 +177,7 @@ enum {#define NBD_REPLY_TYPE_NONE 0#define NBD_REPLY_TYPE_OFFSET_DATA 1#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_BLOCK_STATUS 5#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2)
nbd/nbd-internal.h +6 -0
@@ -83,6 +83,10 @@#define NBD_OPT_PEEK_EXPORT (4)#define NBD_OPT_STARTTLS (5)#define NBD_OPT_STRUCTURED_REPLY (8) +#define NBD_OPT_LIST_META_CONTEXT (9) +#define NBD_OPT_SET_META_CONTEXT (10) + +#define NBD_META_NS_BITMAPS "qemu-dirty-bitmap"/* NBD errors are based on errno numbers, so there is a 1:1 mapping, * but only a limited set of errno values is specified in the protocol.
@@ -105,6 +109,8 @@ static inline const char *nbd_opt_name(int opt)case NBD_OPT_PEEK_EXPORT :return "peek_export" ;case NBD_OPT_STARTTLS :return "tls" ;case NBD_OPT_STRUCTURED_REPLY :return "structured_reply" ; +case NBD_OPT_LIST_META_CONTEXT :return "list_meta_context" ; +case NBD_OPT_SET_META_CONTEXT :return "set_meta_context" ; }return "<unknown option>" ;
nbd/server.c +445 -0
482 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 11/18] nbd: BLOCK_STATUS for bitmap export: server part
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Only one meta context type is defined: qemu-bitmap:<bitmap-name>.
Why 'qemu-bitmap:' instead of 'qemu:' for our namespace? I guess it's okay, though.
Quoted 5 lines
Maximum one query is allowed for NBD_OPT_{SET,LIST}_META_CONTEXT,
NBD_REP_ERR_TOO_BIGis returned otherwise.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
This is only a high-level review; I may come back and look more closely at details (and compare against the proposed spec) when I have more time.
include/block/nbd.h enum { · line 159
Quoted 8 lines
/* Request flags, sent from client to server during transmission phase */
#defineNBD_CMD_FLAG_FUA(1 << 0) /* 'force unit access' during write */include/block/nbd.h +1
-0@@ -142,6 +155,7 @@ enum {NBD_CMD_TRIM =4 ,/* 5 reserved for failed experiment NBD_CMD_CACHE */ NBD_CMD_WRITE_ZEROES =6 , + NBD_CMD_BLOCK_STATUS = 7 };
Please keep the trailing comma (it makes it easier for later patches to add new values without having to modify existing lines).
include/block/nbd.h enum { · line 182
Quoted 9 lines
include/block/nbd.h +1
-0#define NBD_DEFAULT_PORT 10809@@ -163,6 +177,7 @@ enum {#define NBD_REPLY_TYPE_NONE 0#define NBD_REPLY_TYPE_OFFSET_DATA 1#define NBD_REPLY_TYPE_OFFSET_HOLE 2 +#define NBD_REPLY_TYPE_BLOCK_STATUS 5#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2)
Might be worth formatting these so that the definitions all start in the same column (if so, it affects an earlier patch in the series as well).
nbd/nbd-internal.h line 89
Quoted 13 lines
nbd/nbd-internal.h +4
-0@@ -83,6 +83,10 @@#define NBD_OPT_PEEK_EXPORT (4)#define NBD_OPT_STARTTLS (5)#define NBD_OPT_STRUCTURED_REPLY (8) +#define NBD_OPT_LIST_META_CONTEXT (9) +#define NBD_OPT_SET_META_CONTEXT (10) + +#define NBD_META_NS_BITMAPS "qemu-dirty-bitmap"
I still find it odd that we've split our #defines across two different headers, not your fault.
This name does not match your commit message. Why can't we use just 'qemu:' as our namespace?
nbd/nbd-internal.h static inline const char *nbd_opt_name(int opt) · line 113
Quoted 9 lines
nbd/nbd-internal.h +2
-0/* NBD errors are based on errno numbers, so there is a 1:1 mapping, * but only a limited set of errno values is specified in the protocol. @@ -105,6 +109,8 @@ static inline const char *nbd_opt_name(int opt)case NBD_OPT_PEEK_EXPORT :return "peek_export" ;case NBD_OPT_STARTTLS :return "tls" ;case NBD_OPT_STRUCTURED_REPLY :return "structured_reply" ; +case NBD_OPT_LIST_META_CONTEXT :return "list_meta_context" ; +case NBD_OPT_SET_META_CONTEXT :return "set_meta_context" ;
Same question as earlier as to whether this violates checkpatch formatting, and whether it belongs in a .c instead of inline in the header.
nbd/nbd-internal.h static inline const char *nbd_opt_name(int opt) · line 115
Quoted 4 lines
nbd/nbd-internal.h +2
-0+static int nbd_negotiate_read_size_string (QIOChannel *ioc,char **str, +uint32_t max_len)
I probably would have split the creation of this helper function into its own patch. Also, can it be utilized in any of the existing code, or is the new code the only client?
nbd/nbd-internal.h static inline const char *nbd_opt_name(int opt)
Quoted 8 lines
nbd/nbd-internal.h +8
-0+ +/* start handle LIST_META_CONTEXT and SET_META_CONTEXT requests +* @opt should be NBD_OPT_LIST_META_CONTEXT or NBD_OPT_SET_META_CONTEXT +* @length related option data to read +* @nb_queries out parameter, number of queries specified by client +* @bs out parameter, bs for export, selected by client +* will be zero if some not critical error occured and error reply +* was sent.
Wrong spelling of 'occurred', and awkward grammar. How about:
will be zero if an error reply is successfully sent
Quoted 3 lines
+ *
+ * Returns:
+ * Err. code < 0 on critical error
s/Err./Error/
Quoted 2 lines
+ * Number of bytes read otherwise (will be equal to length on non critical
+ * error or if there no queries in request)
will be equal to length if there were no errors, or no queries in the request
nbd/nbd-internal.h
Quoted 9 lines
+ */
nbd/nbd-internal.h +8
-0+static int nbd_negotiate_opt_meta_context_start (NBDClient *client,uint32_t opt, +uint32_t length, +uint32_t *nb_queries, +BlockDriverState **bs) +{ +int ret; +NBDExport *exp; +char *export_name;
Uninitialized...[1]
nbd/nbd-internal.h
Quoted 5 lines
nbd/nbd-internal.h +5
-0+int nb_read =0 ; + +if (!client->structured_reply) { +uint32_t tail = length - nb_read; +LOG ("Structured reply is not negotiated" );
This LOG() is redundant...[2]
nbd/nbd-internal.h
Quoted 6 lines
nbd/nbd-internal.h +6
-0+ +if (nbd_negotiate_drop_sync (client->ioc, tail) != tail) { +return -EIO ; + } + ret =nbd_negotiate_send_rep_err (client->ioc,NBD_REP_ERR_INVALID , opt, +"Structured reply is not negotiated" );
[2]...because this function also calls LOG().
Quoted 1 line
+
g_free(export_name);
[1]...so this crashes. Oops.
Quoted 31 lines
nbd/nbd-internal.h +29
-0+ +if (ret <0 ) { +return ret; + }else { + *bs =NULL ; + *nb_queries =0 ; +return length; + } + } + +static int nbd_negotiate_set_meta_context (NBDClient *client,uint32_t length) +{ +int ret; +BlockDriverState *bs; +uint32_t nb_queries; +int nb_read; + + nb_read =nbd_negotiate_opt_meta_context_start (client, +NBD_OPT_SET_META_CONTEXT , + length, &nb_queries, &bs); +if (nb_read <0 ) { +return nb_read; + } +if (bs ==NULL ) { +/* error reply was already sent by nbd_negotiate_opt_meta_context_start +* */ +return 0 ; + } +
nbd/nbd-internal.h
Quoted 10 lines
nbd/nbd-internal.h +10
-0+if (nb_queries ==0 ) { +return nbd_negotiate_send_rep (client->ioc,NBD_REP_ACK , +NBD_OPT_SET_META_CONTEXT ); + } + +if (nb_queries >1 ) { +return nbd_negotiate_send_rep_err (client->ioc,NBD_REP_ERR_TOO_BIG , +NBD_OPT_SET_META_CONTEXT , +"Only one exporting context is" +"supported" );
Why only one? What's wrong with supporting both 'qemu:xyz' (the dirty bitmap, with whatever prefix name we actually settle on) and 'base:allocation' at the same time?
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 11/18] nbd: BLOCK_STATUS for bitmap export: server part
In reply to Vladimir Sementsov-OgievskiyQuoted 5 lines · Vladimir Sementsov-Ogievskiy
Only one meta context type is defined: qemu-bitmap:<bitmap-name>. Maximum one query is allowed for NBD_OPT_{SET,LIST}_META_CONTEXT,
NBD_REP_ERR_TOO_BIGis returned otherwise.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>
...
Quoted 10 lines
nbd/server.c +10
-0+static int nbd_negotiate_opt_meta_context_start (NBDClient *client,uint32_t opt, +uint32_t length, +uint32_t *nb_queries, +BlockDriverState **bs) +{ +int ret; +NBDExport *exp; +char *export_name; +int nb_read =0 ; +
nbd/server.c static QIOChannel *nbd_negotiate_handle_starttls(NBDClient *client, · line 574
Quoted 10 lines
nbd/server.c +10
-0+if (!client->structured_reply) { +uint32_t tail = length - nb_read; +LOG ("Structured reply is not negotiated" ); + +if (nbd_negotiate_drop_sync (client->ioc, tail) != tail) { +return -EIO ; + } + ret =nbd_negotiate_send_rep_err (client->ioc,NBD_REP_ERR_INVALID , opt, +"Structured reply is not negotiated" ); +g_free (export_name);
export_name is not initialized here! for me there is no need to free anything here
Quoted 12 lines
nbd/server.c +12
-0+ +if (ret <0 ) { +return ret; + }else { + *bs =NULL ; + *nb_queries =0 ; +return length; + } + } + + nb_read =nbd_negotiate_read_size_string (client->ioc, &export_name, +NBD_MAX_NAME_SIZE );
nbd/server.c static QIOChannel *nbd_negotiate_handle_starttls(NBDClient *client, · line 596
Quoted 10 lines
nbd/server.c +10
-0+if (nb_read <0 ) { +return nb_read; + } + + exp =nbd_export_find (export_name); +if (exp ==NULL ) { +uint32_t tail = length - nb_read; +LOG ("export '%s' is not found" , export_name); + +if (nbd_negotiate_drop_sync (client->ioc, tail) != tail) {
export_name is leaked on this path
Quoted 7 lines
nbd/server.c +7
-0+return -EIO ; + } + ret =nbd_negotiate_send_rep_err (client->ioc,NBD_REP_ERR_INVALID , opt, +"export '%s' is not found" , + export_name); +g_free (export_name); +
PATCH 12/18 nbd: BLOCK_STATUS for bitmap export: client part
In reply to Vladimir Sementsov-Ogievskiy--- block/nbd-client.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/nbd-client.h | 6 +++ block/nbd.c | 9 +++- include/block/nbd.h | 6 ++- nbd/client.c | 103 +++++++++++++++++++++++++++++++++++- nbd/server.c | 2 - qapi/block-core.json | 5 +- qemu-nbd.c | 2 +- 8 files changed, 270 insertions(+), 9 deletions(-)
block/nbd-client.c +145 -1
@@ -388,6 +388,147 @@ int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) } +static inline ssize_t read_sync (QIOChannel *ioc,void *buffer,size_t size) +{ +struct iovec iov = { .iov_base = buffer, .iov_len = size }; +/* Sockets are kept in blocking mode in the negotiation phase. After +* that, a non-readable socket simply means that another thread stole +* our request/reply. Synchronization is done with recv_coroutine, so +* that this is coroutine-safe. +*/ +return nbd_wr_syncv (ioc, &iov,1 , size, true); +} + +static int nbd_client_co_cmd_block_status (BlockDriverState *bs,uint64_t offset, +uint64_t bytes,NBDExtent **pextents, +unsigned *nb_extents) +{ +int64_t ret; +NBDReply reply; +uint32_t context_id; +int64_t nb, i; +NBDExtent *extents =NULL ; +NBDClientSession *client =nbd_get_client_session (bs); +NBDRequest request = { + .type =NBD_CMD_BLOCK_STATUS , + .from = offset, + .len = bytes, + .flags =0 , + }; + +nbd_coroutine_start (client, &request); + + ret =nbd_co_send_request (bs, &request,NULL ); +if (ret <0 ) { + goto fail; + } + +nbd_co_receive_reply (client, &request, &reply,NULL ); +if (reply.error !=0 ) { + ret = -reply.error; + } +if (reply.simple) { + ret = -EINVAL ; + goto fail; + } +if (reply.error !=0 ) { + ret = -reply.error; + goto fail; + } +if (reply.type !=NBD_REPLY_TYPE_BLOCK_STATUS ) { + ret = -EINVAL ; + goto fail; + } + +read_sync (client->ioc, &context_id,sizeof (context_id)); +cpu_to_be32s (&context_id); +if (client->meta_data_context_id != context_id) { + ret = -EINVAL ; + goto fail; + } + + nb = (reply.length -sizeof (context_id)) /sizeof (NBDExtent); + extents =g_new (NBDExtent, nb); +if (read_sync (client->ioc, extents, nb *sizeof (NBDExtent)) != + nb *sizeof (NBDExtent)) + { + ret = -EIO ; + goto fail; + } + +if (!(reply.flags &&NBD_REPLY_FLAG_DONE )) { +nbd_co_receive_reply (client, &request, &reply,NULL ); +if (reply.simple) { + ret = -EINVAL ; + goto fail; + } +if (reply.error !=0 ) { + ret = -reply.error; + goto fail; + } +if (reply.type !=NBD_REPLY_TYPE_NONE || + !(reply.flags &&NBD_REPLY_FLAG_DONE )) { + ret = -EINVAL ; + goto fail; + } + } + +for (i =0 ; i < nb; ++i) { +cpu_to_be32s (&extents[i].length); +cpu_to_be32s (&extents[i].flags); + } + + *pextents = extents; + *nb_extents = nb; +nbd_coroutine_end (client, &request); +return 0 ; + +fail: +g_free (extents); +nbd_coroutine_end (client, &request); +return ret; +} + +/* nbd_client_co_load_bitmap_part() returns end of set area, i.e. first next +* byte of unknown status (may be >= disk size, which means that the bitmap was +* set up to the end). +*/ +int64_t nbd_client_co_load_bitmap_part (BlockDriverState *bs,uint64_t offset, +uint64_t bytes,BdrvDirtyBitmap *bitmap) +{ +int64_t ret; +uint64_t start_byte; +uint32_t nb_extents; +int64_t i, start_sector, last_sector, nr_sectors; +NBDExtent *extents =NULL ; + + ret =nbd_client_co_cmd_block_status (bs, offset, bytes, &extents, + &nb_extents); +if (ret <0 ) { +return ret; + } + + start_byte = offset; +for (i =0 ; i < nb_extents; ++i) { +if (extents[i].flags ==1 ) { + start_sector = start_byte >>BDRV_SECTOR_BITS ; + last_sector = + (start_byte + extents[i].length -1 ) >>BDRV_SECTOR_BITS ; + nr_sectors = last_sector - start_sector +1 ; + +bdrv_set_dirty_bitmap (bitmap, start_sector, nr_sectors); + } + + start_byte += extents[i].length; + } + +g_free (extents); + +return ROUND_UP ((uint64_t )start_byte, + (uint64_t )bdrv_dirty_bitmap_granularity (bitmap)); +} + +void nbd_client_detach_aio_context (BlockDriverState *bs) { aio_set_fd_handler(bdrv_get_aio_context(bs),
@@ -421,6 +562,7 @@ int nbd_client_init(BlockDriverState *bs,const char *export, QCryptoTLSCreds *tlscreds, const char *hostname, + const char *bitmap_name, Error **errp) { NBDClientSession *client = nbd_get_client_session(bs);
@@ -435,7 +577,9 @@ int nbd_client_init(BlockDriverState *bs, tlscreds, hostname, &client->ioc, &client->size, - &client->structured_reply, errp); + &client->structured_reply, + bitmap_name, + &client->bitmap_ok, errp);if (ret <0 ) {logout ("Failed to negotiate with the NBD server\n" );return ret;
block/nbd-client.h +6 -0
@@ -34,6 +34,8 @@ typedef struct NBDClientSession {bool is_unix;bool structured_reply; +bool bitmap_ok; +uint32_t meta_data_context_id; } NBDClientSession;NBDClientSession *nbd_get_client_session (BlockDriverState *bs);
@@ -43,6 +45,7 @@ int nbd_client_init(BlockDriverState *bs,const char *export_name, QCryptoTLSCreds *tlscreds, const char *hostname, + const char *bitmap_name, Error **errp);void nbd_client_close (BlockDriverState *bs);
@@ -54,9 +57,12 @@ int nbd_client_co_pwrite_zeroes(BlockDriverState *bs, int64_t offset, int count, BdrvRequestFlags flags);int nbd_client_co_preadv (BlockDriverState *bs,uint64_t offset,uint64_t bytes,QEMUIOVector *qiov,int flags); +int64_t nbd_client_co_load_bitmap_part (BlockDriverState *bs,uint64_t offset, +uint64_t bytes,BdrvDirtyBitmap *bitmap);void nbd_client_detach_aio_context (BlockDriverState *bs);void nbd_client_attach_aio_context (BlockDriverState *bs,AioContext *new_context); +#endif /* NBD_CLIENT_H */
block/nbd.c +7 -2
@@ -382,6 +382,11 @@ static QemuOptsList nbd_runtime_opts = { .type = QEMU_OPT_STRING, .help = "ID of the TLS credentials to use", }, + { + .name = "bitmap", + .type = QEMU_OPT_STRING, + .help = "Name of dirty bitmap to export", + }, }, };
@@ -440,8 +445,8 @@ static int nbd_open(BlockDriverState *bs, QDict *options, int flags, }/* NBD handshake */ - ret =nbd_client_init (bs, sioc, s->export, - tlscreds, hostname, errp); + ret =nbd_client_init (bs, sioc, s->export, tlscreds, hostname, +qemu_opt_get (opts,"bitmap" ), errp); error:if (sioc) {object_unref (OBJECT (sioc));
include/block/nbd.h +5 -1
@@ -181,6 +181,8 @@ enum {#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2) +#define NBD_MAX_BITMAP_EXTENTS (0x100000 / 8)/* 1 mb of extents data */ + ssize_t nbd_wr_syncv(QIOChannel *ioc, struct iovec *iov, size_t niov,
@@ -189,7 +191,9 @@ ssize_t nbd_wr_syncv(QIOChannel *ioc,int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname,QIOChannel **outioc, -off_t *size,bool *structured_reply,Error **errp); +off_t *size,bool *structured_reply, +const char *bitmap_name,bool *bitmap_ok, +Error **errp);int nbd_init (int fd,QIOChannelSocket *sioc,uint16_t flags,off_t size);ssize_t nbd_send_request (QIOChannel *ioc,NBDRequest *request);int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply);
nbd/client.c +102 -1
118 lines not on this page · raw message
nbd/server.c +0 -2
@@ -21,8 +21,6 @@#include "qapi/error.h" #include "nbd-internal.h" -#define NBD_MAX_BITMAP_EXTENTS (0x100000 / 8)/* 1 mb of extents data */ -static int system_errno_to_nbd_errno (int err) {switch (err) {
qapi/block-core.json +4 -1
@@ -2331,12 +2331,15 @@ # # @tls-creds: #optional TLS credentials ID # +# @bitmap: #optional Dirty bitmap name to export (vz-7.4) +# # Since: 2.8 ## { 'struct': 'BlockdevOptionsNbd', 'data': { 'server': 'SocketAddress', '*export': 'str', - '*tls-creds': 'str' } } + '*tls-creds': 'str', + '*bitmap': 'str'} } ## # @BlockdevOptionsRaw:
qemu-nbd.c +1 -1
@@ -272,7 +272,7 @@ static void *nbd_client_thread(void *arg) ret =nbd_receive_negotiate (QIO_CHANNEL (sioc),NULL , &nbdflags,NULL ,NULL ,NULL , - &size,NULL , &local_error); + &size,NULL ,NULL ,NULL , &local_error);if (ret <0 ) {if (local_error) {error_report_err (local_error);
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 12/18] nbd: BLOCK_STATUS for bitmap export: client part
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>
Rather light on the commit description. Might be worth pointing to the NBD protocol that it is implementing, particularly since that part of the protocol is still an extension and may be subject to change based on what we learn in our implementation attempt.
Quoted 10 lines
--- block/nbd-client.c | 146 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/nbd-client.h | 6 +++ block/nbd.c | 9 +++- include/block/nbd.h | 6 ++- nbd/client.c | 103 +++++++++++++++++++++++++++++++++++- nbd/server.c | 2 - qapi/block-core.json | 5 +- qemu-nbd.c | 2 +- 8 files changed, 270 insertions(+), 9 deletions(-)
Again, this is a high-level review where I haven't closely checked the spec.
Quoted 9 lines
block/nbd-client.c +0
-0@@ -388,6 +388,147 @@ int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) }
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 400
Quoted 10 lines
block/nbd-client.c +10
-0+static inline ssize_t read_sync (QIOChannel *ioc,void *buffer,size_t size) +{ +struct iovec iov = { .iov_base = buffer, .iov_len = size }; +/* Sockets are kept in blocking mode in the negotiation phase. After +* that, a non-readable socket simply means that another thread stole +* our request/reply. Synchronization is done with recv_coroutine, so +* that this is coroutine-safe. +*/ +return nbd_wr_syncv (ioc, &iov,1 , size, true); +}
Does this need a coroutine_fn tag? I wonder if part of this patch should be split, to introduce the new helper function and rework existing calls in one patch, then add new BLOCK_STATUS stuff in another.
Quoted 9 lines
block/nbd-client.c +8
-0+ +static int nbd_client_co_cmd_block_status (BlockDriverState *bs,uint64_t offset, +uint64_t bytes,NBDExtent **pextents, +unsigned *nb_extents) +{ + +nbd_co_receive_reply (client, &request, &reply,NULL ); +if (reply.error !=0 ) {
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 437
Quoted 10 lines
block/nbd-client.c +10
-0+ ret = -reply.error; + } +if (reply.simple) { + ret = -EINVAL ; + goto fail; + } +if (reply.error !=0 ) { + ret = -reply.error; + goto fail; + }
Duplicate check for reply.error.
Quoted 4 lines
block/nbd-client.c +4
-0+if (reply.type !=NBD_REPLY_TYPE_BLOCK_STATUS ) { + ret = -EINVAL ; + goto fail; + }
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 451
Quoted 10 lines
block/nbd-client.c +10
-0+ +read_sync (client->ioc, &context_id,sizeof (context_id)); +cpu_to_be32s (&context_id); +if (client->meta_data_context_id != context_id) { + ret = -EINVAL ; + goto fail; + } + + nb = (reply.length -sizeof (context_id)) /sizeof (NBDExtent); + extents =g_new (NBDExtent, nb);
reply.length is server-controlled. I didn't closely check whether we have any sanity checks that prevent a malicious server from sending a bogus length that would cause us to allocate far too much memory. We may already have such a check (we forbid incoming read packets larger than 32M), so this is more just making sure that those existing checks cover this code too.
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 459
Quoted 8 lines
block/nbd-client.c +8
-0+if (read_sync (client->ioc, extents, nb *sizeof (NBDExtent)) != + nb *sizeof (NBDExtent)) + { + ret = -EIO ; + goto fail; + } + +if (!(reply.flags &&NBD_REPLY_FLAG_DONE )) {
Wrong logic. s/&&/&/
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 472
Quoted 13 lines
block/nbd-client.c +13
-0+nbd_co_receive_reply (client, &request, &reply,NULL ); +if (reply.simple) { + ret = -EINVAL ; + goto fail; + } +if (reply.error !=0 ) { + ret = -reply.error; + goto fail; + } +if (reply.type !=NBD_REPLY_TYPE_NONE || + !(reply.flags &&NBD_REPLY_FLAG_DONE )) { + ret = -EINVAL ; + goto fail;
It looks like you are insisting that the server send at most one extent packet, and that either that packet, or a concluding TYPE_NONE packet, must end the transaction (and if not, you drop the connection). While that may happen to be the implementation you wrote for qemu as the server, I don't think it is a requirement in the NBD spec, and that a server could reasonably send multiple fragments that you must reassemble into the overall final reply. In fact, the final reply packet could be NBD_REPLY_TYPE_ERROR instead of NBD_REPLY_TYPE_NONE (if there was a late-breaking error detection that invalidates code sent earlier), which you should handle gracefully rather than aborting the connection.
block/nbd-client.c int nbd_client_co_pdiscard(BlockDriverState *bs, int64_t offset, int count) · line 478
Quoted 6 lines
block/nbd-client.c +6
-0+ } + } + +for (i =0 ; i < nb; ++i) { +cpu_to_be32s (&extents[i].length); +cpu_to_be32s (&extents[i].flags);
Is it worth sanity-checking that the server's reported extents actually fall within the range we requested information on (well, other than the last extent which may extend beyond the range)? When it comes to network communication, both the client and the server SHOULD assume that the other side may become malicious at any time, and verify that all data makes sense before relying on it.
Quoted 17 lines
block/nbd-client.c +14
-0+ } + + *pextents = extents; + *nb_extents = nb; +nbd_coroutine_end (client, &request); +return 0 ; + +fail: +g_free (extents); +nbd_coroutine_end (client, &request); +return ret; +} + +++ b/block/nbd-client.h
block/nbd-client.c typedef struct NBDClientSession {
Quoted 8 lines
block/nbd-client.c +1
-0@@ -34,6 +34,8 @@ typedef struct NBDClientSession {void nbd_client_detach_aio_context (BlockDriverState *bs);void nbd_client_attach_aio_context (BlockDriverState *bs,AioContext *new_context); +#endif /* NBD_CLIENT_H */
Why the added blank line here?
Quoted 5 lines
block/nbd.c +0
-0
block/nbd.c static QemuOptsList nbd_runtime_opts = { · line 391
Quoted 10 lines
block/nbd.c +5
-0@@ -382,6 +382,11 @@ static QemuOptsList nbd_runtime_opts = { .type = QEMU_OPT_STRING, .help = "ID of the TLS credentials to use", }, + { + .name = "bitmap", + .type = QEMU_OPT_STRING, + .help = "Name of dirty bitmap to export", + }, }, };
I'm confused here - it seems like the name of a dirty bitmap to export is a server detail, not a client detail. After all, it is the server that is using the bitmap to inform the client of what extents have a given property. So does this hunk belong in a different patch in the series?
block/nbd.c enum { · line 184
Quoted 6 lines
block/nbd.c +2
-0+++ b/include/block/nbd.h@@ -181,6 +181,8 @@ enum {#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2) +#define NBD_MAX_BITMAP_EXTENTS (0x100000 / 8)/* 1 mb of extents data */
The commit message might be a good place to document why you picked this arbitrary limit (1 million extents covers a LOT of file if each extent in turn represents a cluster of that file); meanwhile, since each extent occupies 8 bytes, this means that you are clamping the server to send at most 8 megabytes before you drop the connection (compared to accepting 32 megabytes for a read).
Quoted 11 lines
block/nbd.c +10
-0+static int nbd_receive_query_meta_context (QIOChannel *ioc,const char *export, +const char *context,bool *ok, +Error **errp) +{ +int ret; +nbd_opt_reply reply; +size_t export_len =strlen (export); +size_t context_len =strlen (context); +size_t data_len =4 + export_len +4 +4 + context_len; +
block/nbd.c enum {
Quoted 10 lines
block/nbd.c +10
-0+char *data =g_malloc (data_len); +char *p = data; +int nb_reps =0 ; + + *ok = false; +stl_be_p (p, export_len); +memcpy (p +=4 , export, export_len); +stl_be_p (p += export_len,1 ); +stl_be_p (p +=4 , context_len); +memcpy (p +=4 , context, context_len);
Is '4' better than 'sizeof(...)' here?
Quoted 12 lines
block/nbd.c +11
-0+ +TRACE ("Requesting set_meta_context option from server" ); + ret =nbd_send_option_request (ioc,NBD_OPT_SET_META_CONTEXT , data_len, data, + errp); +if (read_sync (ioc, &context_id,sizeof (context_id)) != +sizeof (context_id)) + { + ret = -EIO ; + goto out; + } +
block/nbd.c
Quoted 10 lines
block/nbd.c +10
-0+be32_to_cpus (&context_id); + + len = reply.length -sizeof (context_id); + context_name =g_malloc (len +1 ); +if (read_sync (ioc, context_name, len) != len) { + + ret = -EIO ; + goto out; + } + context_name[len] ='\0' ;
Shouldn't we validate that the context name returned by the server matches our expectations of the context name we requested?
block/nbd.c
Quoted 7 lines
block/nbd.c +7
-0+ +TRACE ("set meta: %u %s" , context_id, context_name); + + nb_reps++; + } + + *ok = nb_reps ==1 && reply.type ==NBD_REP_ACK ;
Insisting on exactly one context reply works only if you ensure that you only request exactly one context; but we may want to request 'base:allocation' in addition to 'qemu:bitmapname' at some point. Also, I think the spec allows for the server to send a different number of contexts than what we request (whether a context that it always provides even though we didn't request it, or omitting something we requested because it is unable to provide it).
block/nbd.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 684
Quoted 6 lines
block/nbd.c +2
-0@@ -589,6 +680,16 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags,nbd_receive_simple_option (ioc,NBD_OPT_STRUCTURED_REPLY , false,NULL ) ==0 ; } + +if (!!structured_reply && *structured_reply && !!bitmap_name) {
Why the use of !! to coerce pointers to bool when you already have && coercing to bool?
block/nbd.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 686
Quoted 2 lines
block/nbd.c +2
-0+int ret; +assert (!!bitmap_ok);
Another pointless use of !!
Quoted 11 lines
block/nbd.c +7
-0+ ret =nbd_receive_query_bitmap (ioc, name, bitmap_name, + bitmap_ok, errp) ==0 ; +if (ret <0 ) { + goto fail; + } + } }/* write the export name request */ +++ b/nbd/server.c
block/nbd.c line 25
Quoted 5 lines
block/nbd.c +0
-2@@ -21,8 +21,6 @@#include "qapi/error.h" #include "nbd-internal.h" -#define NBD_MAX_BITMAP_EXTENTS (0x100000 / 8)/* 1 mb of extents data */ -
This is needless churn, since you only barely introduced this in an earlier patch. Stick it in the final file in the first patch where you introduce it.
block/nbd.c line 2334
Quoted 6 lines
block/nbd.c +2
-0+++ b/qapi/block-core.json@@ -2331,12 +2331,15 @@ # # @tls-creds: #optional TLS credentials ID # +# @bitmap: #optional Dirty bitmap name to export (vz-7.4)
s/vz-7.4/2.9/
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 13/18 nbd: add nbd_dirty_bitmap_load
In reply to Vladimir Sementsov-OgievskiyRealize bdrv_dirty_bitmap_load interface.
--- block/nbd.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
block/nbd.c +32 -0
@@ -557,6 +557,35 @@ static void nbd_refresh_filename(BlockDriverState *bs, QDict *options) bs->full_open_options = opts; } +static BdrvDirtyBitmap *nbd_dirty_bitmap_load (BlockDriverState *bs, +const char *name,Error **errp) +{ +int64_t offset =0 , end; +uint32_t ma =-1 ; +BdrvDirtyBitmap *bitmap = +bdrv_create_dirty_bitmap (bs,bdrv_get_default_bitmap_granularity (bs), + name, errp); +if (bitmap ==NULL ) { +return NULL ; + } + + end =bdrv_dirty_bitmap_size (bitmap) <<BDRV_SECTOR_BITS ; + +while (offset < end) { + offset =nbd_client_co_load_bitmap_part (bs, offset, +MIN (end - offset, ma), bitmap); +if (offset <0 ) { + goto fail; + } + } + +return bitmap; + +fail: +bdrv_release_dirty_bitmap (bs, bitmap); +return NULL ; +} +static BlockDriver bdrv_nbd = { .format_name ="nbd" , .protocol_name ="nbd" ,
@@ -574,6 +603,7 @@ static BlockDriver bdrv_nbd = { .bdrv_detach_aio_context = nbd_detach_aio_context, .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, + .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, };static BlockDriver bdrv_nbd_tcp = {
@@ -593,6 +623,7 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_detach_aio_context = nbd_detach_aio_context, .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, + .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, };static BlockDriver bdrv_nbd_unix = {
@@ -612,6 +643,7 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_detach_aio_context = nbd_detach_aio_context, .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, + .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, };static void bdrv_nbd_init (void )
Signature · 1 line
2.11.0
PATCH 14/18 qmp: add x-debug-block-dirty-bitmap-sha256
In reply to Vladimir Sementsov-Ogievskiy--- block/dirty-bitmap.c | 5 +++++ blockdev.c | 29 +++++++++++++++++++++++++++++ include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 8 ++++++++ qapi/block-core.json | 27 +++++++++++++++++++++++++++ tests/Makefile.include | 2 +- util/hbitmap.c | 11 +++++++++++ 7 files changed, 83 insertions(+), 1 deletion(-)
block/dirty-bitmap.c +5 -0
@@ -598,3 +598,8 @@ BdrvDirtyBitmap *bdrv_load_dirty_bitmap(BlockDriverState *bs, const char *name,return lbco.ret; } + +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp) +{ +return hbitmap_sha256 (bitmap->bitmap, errp); +}
blockdev.c +29 -0
@@ -2790,6 +2790,35 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,aio_context_release (aio_context); } +BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256 (const char *node, +const char *name, +Error **errp) +{ +AioContext *aio_context; +BdrvDirtyBitmap *bitmap; +BlockDriverState *bs; +BlockDirtyBitmapSha256 *ret =NULL ; +char *sha256; + + bitmap =block_dirty_bitmap_lookup (node, name, &bs, &aio_context, errp); +if (!bitmap || !bs) { +return NULL ; + } + + sha256 =bdrv_dirty_bitmap_sha256 (bitmap, errp); +if (sha256 ==NULL ) { + goto out; + } + + ret =g_new (BlockDirtyBitmapSha256,1 ); + ret->sha256 = sha256; + +out: +aio_context_release (aio_context); + +return ret; +} +void hmp_drive_del (Monitor *mon,const QDict *qdict) {const char *id =qdict_get_str (qdict,"id" );
include/block/dirty-bitmap.h +2 -0
@@ -80,4 +80,6 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,BdrvDirtyBitmap *bdrv_load_dirty_bitmap (BlockDriverState *bs,const char *name,Error **errp); +char *bdrv_dirty_bitmap_sha256 (const BdrvDirtyBitmap *bitmap,Error **errp); +#endif
include/qemu/hbitmap.h +8 -0
@@ -225,6 +225,14 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,void hbitmap_deserialize_finish (HBitmap *hb);/** +* hbitmap_sha256: +* @bitmap: HBitmap to operate on. +* +* Returns SHA256 hash of the last level. +*/ +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp); + +/** * hbitmap_free: * @hb: HBitmap to operate on. *
qapi/block-core.json +27 -0
@@ -1280,6 +1280,33 @@ 'data': 'BlockDirtyBitmap' } ## +# @BlockDirtyBitmapSha256: +# +# SHA256 hash of dirty bitmap data +# +# @sha256: bitmap SHA256 hash +# +# Since: 2.9 +## + { 'struct': 'BlockDirtyBitmapSha256', + 'data': {'sha256': 'str'} } + +## +# @x-debug-block-dirty-bitmap-sha256: +# +# Get bitmap SHA256 +# +# Returns: BlockDirtyBitmapSha256 on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found or if hashing is failed, GenericError with an +# explanation +# +# Since: 2.9 +## + { 'command': 'x-debug-block-dirty-bitmap-sha256', + 'data': 'BlockDirtyBitmap', 'returns': 'BlockDirtyBitmapSha256' } + +## # @blockdev-mirror: # # Start mirroring a block device's writes to a new destination.
tests/Makefile.include +1 -1
@@ -497,7 +497,7 @@ tests/test-blockjob$(EXESUF): tests/test-blockjob.o $(test-block-obj-y) $(test-u tests/test-blockjob-txn$(EXESUF): tests/test-blockjob-txn.o $(test-block-obj-y) $(test-util-obj-y) tests/test-thread-pool$(EXESUF): tests/test-thread-pool.o $(test-block-obj-y) tests/test-iov$(EXESUF): tests/test-iov.o $(test-util-obj-y) -tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) +tests/test-hbitmap$(EXESUF): tests/test-hbitmap.o $(test-util-obj-y) $(test-crypto-obj-y) tests/test-x86-cpuid$(EXESUF): tests/test-x86-cpuid.o tests/test-xbzrle$(EXESUF): tests/test-xbzrle.o migration/xbzrle.o page_cache.o $(test-util-obj-y) tests/test-cutils$(EXESUF): tests/test-cutils.o util/cutils.o
util/hbitmap.c +11 -0
@@ -13,6 +13,7 @@#include "qemu/hbitmap.h" #include "qemu/host-utils.h" #include "trace.h" +#include "crypto/hash.h" /* HBitmaps provides an array of bits. The bits are stored as usual in an * array of unsigned longs, but HBitmap is also optimized to provide fast
@@ -699,3 +700,13 @@ void hbitmap_free_meta(HBitmap *hb)hbitmap_free (hb->meta); hb->meta =NULL ; } + +char *hbitmap_sha256 (const HBitmap *bitmap,Error **errp) +{ +size_t size = bitmap->sizes[HBITMAP_LEVELS -1 ] *sizeof (unsigned long ); +char *data = (char *)bitmap->levels[HBITMAP_LEVELS -1 ]; +char *hash =NULL ; +qcrypto_hash_digest (QCRYPTO_HASH_ALG_SHA256 , data, size, &hash, errp); + +return hash; +}
Signature · 1 line
2.11.0
PATCH 15/18 qmp: add block-dirty-bitmap-load
In reply to Vladimir Sementsov-OgievskiyFor loading dirty bitmap from nbd server. Or for underlying storages for other formats.
--- blockdev.c | 28 ++++++++++++++++++++++++++++ qapi/block-core.json | 14 ++++++++++++++ 2 files changed, 42 insertions(+)
blockdev.c +28 -0
@@ -2790,6 +2790,34 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,aio_context_release (aio_context); } +void qmp_block_dirty_bitmap_load (const char *node,const char *name, +Error **errp) +{ +AioContext *aio_context; +BlockDriverState *bs; + +if (!node) { +error_setg (errp,"Node cannot be NULL" ); +return ; + } +if (!name) { +error_setg (errp,"Bitmap name cannot be NULL" ); +return ; + } + bs =bdrv_lookup_bs (node, node,NULL ); +if (!bs) { +error_setg (errp,"Node '%s' not found" , node); +return ; + } + + aio_context =bdrv_get_aio_context (bs); +aio_context_acquire (aio_context); + +bdrv_load_dirty_bitmap (bs, name, errp); + +aio_context_release (aio_context); +} +BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256 (const char *node,const char *name,Error **errp)
qapi/block-core.json +14 -0
@@ -1280,6 +1280,20 @@ 'data': 'BlockDirtyBitmap' } ## +# @block-dirty-bitmap-load: +# +# Load a dirty bitmap from the storage (qcow2 file or nbd export) +# +# Returns: nothing on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found, GenericError with an explanation +# +# Since: vz-7.4 +## + { 'command': 'block-dirty-bitmap-load', + 'data': 'BlockDirtyBitmap' } + +## # @BlockDirtyBitmapSha256: # # SHA256 hash of dirty bitmap data
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 15/18] qmp: add block-dirty-bitmap-load
qapi/block-core.json line 1291
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
##
qapi/block-core.json +9
-0+# @block-dirty-bitmap-load: +# +# Load a dirty bitmap from the storage (qcow2 file or nbd export) +# +# Returns: nothing on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found, GenericError with an explanation +# +# Since: vz-7.4
Version number doesn't look very familiar :)
Fam
Quoted 11 lines
qapi/block-core.json +5
-0+## + { 'command': 'block-dirty-bitmap-load', + 'data': 'BlockDirtyBitmap' } + +## # @BlockDirtyBitmapSha256: # # SHA256 hash of dirty bitmap dataSignature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 15/18] qmp: add block-dirty-bitmap-load
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
For loading dirty bitmap from nbd server. Or for underlying storages for
s/storages/storage/
Quoted 20 lines
other formats.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 28 ++++++++++++++++++++++++++++ qapi/block-core.json | 14 ++++++++++++++ 2 files changed, 42 insertions(+)blockdev.c +4
-0@@ -2790,6 +2790,34 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,aio_context_release (aio_context); } +void qmp_block_dirty_bitmap_load (const char *node,const char *name, +Error **errp) +{ +AioContext *aio_context;
blockdev.c void qmp_block_dirty_bitmap_clear(const char *node, const char *name, · line 2806
Quoted 10 lines
blockdev.c +10
-0+BlockDriverState *bs; + +if (!node) { +error_setg (errp,"Node cannot be NULL" ); +return ; + } +if (!name) { +error_setg (errp,"Bitmap name cannot be NULL" ); +return ; + }
QAPI guarantees that node and name are non-null for non-optional arguments; these checks are dead code...
Quoted 7 lines
blockdev.c +3
-0+++ b/qapi/block-core.json@@ -1280,6 +1280,20 @@ 'data': 'BlockDirtyBitmap' } ## +# @block-dirty-bitmap-load: +#
blockdev.c line 1294
Quoted 10 lines
blockdev.c +10
-0+# Load a dirty bitmap from the storage (qcow2 file or nbd export) +# +# Returns : nothing on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found, GenericError with an explanation +# +# Since : vz-7.4 +## + { 'command': 'block-dirty-bitmap-load', + 'data': 'BlockDirtyBitmap' }
...since BlockDirtyBitmap defines them as required fields.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 15/18] qmp: add block-dirty-bitmap-load
In reply to Vladimir Sementsov-Ogievskiyqapi/block-core.json line 1285
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
##
qapi/block-core.json +3
-0+# @block-dirty-bitmap-load: +# +# Load a dirty bitmap from the storage (qcow2 file or nbd export)
NBD export only in upstream QEMU?
Paolo
Quoted 4 lines
qapi/block-core.json +4
-0+# Returns: nothing on success +# If @node is not a valid block device, DeviceNotFound +# If @name is not found, GenericError with an explanation +#
PATCH 16/18 iotests: add test for nbd dirty bitmap export
In reply to Vladimir Sementsov-Ogievskiy--- tests/qemu-iotests/180 | 133 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/180.out | 5 ++ tests/qemu-iotests/group | 1 + 3 files changed, 139 insertions(+) create mode 100755 tests/qemu-iotests/180 create mode 100644 tests/qemu-iotests/180.out
tests/qemu-iotests/180 new file +133 -0
@@ -0,0 +1,133 @@ +#!/usr/bin/env python +# +# Test case for NBD's bitmap export +# Copyright (C) 2017 Virtuozzo. +# +# derived from io test 147, original copyright: +# Copyright (C) 2016 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import socket +import stat +import time +import iotests +from iotests import cachemode, imgfmt, qemu_img, qemu_nbd + +NBD_PORT = 10811 + +test_img = os.path.join(iotests.test_dir, 'test.img') +unix_socket = os.path.join(iotests.test_dir, 'nbd.socket') + +class NBDBlockdevAddBase(iotests.QMPTestCase): + def blockdev_add_options(self, address, export=None): + options = { 'node-name': 'nbd-blockdev', + 'driver': 'raw', + 'file': { + 'driver': 'nbd', + 'server': address, + 'bitmap': 'mega' + } } + if export is not None: + options['file']['export'] = export + return options + + def client_test(self, filename, address, sha256, export=None): + bao = self.blockdev_add_options(address, export) + result = self.vm.qmp('blockdev-add', **bao) + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('query-named-block-nodes') + for node in result['return']: + if node['node-name'] == 'nbd-blockdev': + if isinstance(filename, str): + self.assert_qmp(node, 'image/filename', filename) + else: + self.assert_json_filename_equal(node['image']['filename'], + filename) + break + + result = self.vm.qmp('block-dirty-bitmap-load', + node='nbd-blockdev', name='mega') + self.assert_qmp(result, 'return', {}) + + result = self.vm.qmp('x-debug-block-dirty-bitmap-sha256', + node='nbd-blockdev', name='mega') + self.assert_qmp(result, 'return/sha256', sha256); + + def setUp(self): + qemu_img('create', '-f', iotests.imgfmt, test_img, '0x400000000') + self.vm = iotests.VM() + self.vm.launch() + + self.server = iotests.VM('.server') + self.server.add_drive_raw('if=none,id=nbd-export,' + + 'file=%s,' % test_img + + 'format=%s,' % imgfmt + + 'cache=%s' % cachemode) + self.server.launch() + + def tearDown(self): + self.vm.shutdown() + self.server.shutdown() + os.remove(test_img) + + def _server_up(self, address): + result = self.server.qmp('nbd-server-start', addr=address) + self.assert_qmp(result, 'return', {}) + + result = self.server.qmp('nbd-server-add', device='nbd-export') + self.assert_qmp(result, 'return', {}) + + def _server_down(self): + result = self.server.qmp('nbd-server-stop') + self.assert_qmp(result, 'return', {}) + + def test_export_bitmap(self): + address = { 'type': 'inet', + 'data': { + 'host': 'localhost', + 'port': str(NBD_PORT) + } } + + granularity = 65536 + regions = [ + { 'start': 0, 'count': 0x100000 }, + { 'start': 0x100000000, 'count': 0x200000 }, + { 'start': 0x399900000, 'count': 0x100000 } + ] + + result = self.server.qmp('block-dirty-bitmap-add', node='nbd-export', + name='mega', granularity=granularity) + self.assert_qmp(result, 'return', {}); + + for r in regions: + self.server.hmp_qemu_io('nbd-export', + 'write %d %d' % (r['start'], r['count'])) + + result = self.server.qmp('x-debug-block-dirty-bitmap-sha256', + node='nbd-export', name='mega') + sha256 = result['return']['sha256'] + + self._server_up(address) + self.client_test('nbd://localhost:%i/nbd-export' % NBD_PORT, + address, sha256, 'nbd-export') + #self._server_down() + +if __name__ == '__main__': + # Need to support image creation + iotests.main(supported_fmts=['vpc', 'parallels', 'qcow', 'vdi', 'qcow2', + 'vmdk', 'raw', 'vhdx', 'qed'])
tests/qemu-iotests/180.out new file +5 -0
@@ -0,0 +1,5 @@ +. +---------------------------------------------------------------------- +Ran 1 tests + +OK
tests/qemu-iotests/group +1 -0
@@ -165,3 +165,4 @@ 170 rw auto quick 171 rw auto quick 172 auto +180 auto
Signature · 1 line
2.11.0
PATCH 17/18 nbd: BLOCK_STATUS for standard get_block_status function: server part
In reply to Vladimir Sementsov-OgievskiyMinimal realization: only one extent in server answer is supported.
--- include/block/nbd.h | 3 ++ nbd/nbd-internal.h | 1 + nbd/server.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 77 insertions(+), 7 deletions(-)
include/block/nbd.h +3 -0
@@ -181,6 +181,9 @@ enum {#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2) +#define NBD_STATE_HOLE 1 +#define NBD_STATE_ZERO (1 << 1) +#define NBD_MAX_BITMAP_EXTENTS (0x100000 / 8)/* 1 mb of extents data */ ssize_t nbd_wr_syncv(QIOChannel *ioc,
nbd/nbd-internal.h +1 -0
@@ -86,6 +86,7 @@#define NBD_OPT_LIST_META_CONTEXT (9)#define NBD_OPT_SET_META_CONTEXT (10) +#define NBD_META_NS_BASE "base"#define NBD_META_NS_BITMAPS "qemu-dirty-bitmap"/* NBD errors are based on errno numbers, so there is a 1:1 mapping,
nbd/server.c +73 -7
132 lines not on this page · raw message
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 17/18] nbd: BLOCK_STATUS for standard get_block_status function: server part
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
Minimal realization: only one extent in server answer is supported.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/nbd.h | 3 ++ nbd/nbd-internal.h | 1 + nbd/server.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 77 insertions(+), 7 deletions(-)include/block/nbd.h +0
-0
include/block/nbd.h enum { · line 185
Quoted 5 lines
include/block/nbd.h +2
-0@@ -181,6 +181,9 @@ enum {#define NBD_REPLY_TYPE_ERROR ((1 << 15) + 1)#define NBD_REPLY_TYPE_ERROR_OFFSET ((1 << 15) + 2) +#define NBD_STATE_HOLE 1 +#define NBD_STATE_ZERO (1 << 1)
Why not (1 << 0) for NBD_STATE_HOLE?
I almost wonder if it is better to rebase the series to implement base:allocation first, rather than last, just for easier interoperability testing with other implementations that still need to implement structured replies; but I don't think it's a show-stopper.
include/block/nbd.h enum {
Quoted 7 lines
include/block/nbd.h +6
-0+static int blockstatus_to_extent (BlockDriverState *bs,uint64_t offset, +uint64_t length,NBDExtent *extent) +{ +BlockDriverState *file; +uint64_t start_sector = offset >>BDRV_SECTOR_BITS ; +uint64_t last_sector = (offset + length -1 ) >>BDRV_SECTOR_BITS ;
Converting from bytes to sectors by rounding...
Quoted 4 lines
include/block/nbd.h +4
-0+uint64_t begin = start_sector; +uint64_t end = last_sector +1 ; + +int nb =MIN (INT_MAX , end - begin);
include/block/nbd.h
Quoted 10 lines
include/block/nbd.h +10
-0+int64_t ret =bdrv_get_block_status_above (bs,NULL , begin, nb, &nb, &file); +if (ret <0 ) { +return ret; + } + + extent->flags = +cpu_to_be32 ((ret &BDRV_BLOCK_ALLOCATED ?0 :NBD_STATE_HOLE ) | + (ret &BDRV_BLOCK_ZERO ?NBD_STATE_ZERO :0 )); + extent->length =cpu_to_be32 ((nb <<BDRV_SECTOR_BITS ) - + (offset - (start_sector <<BDRV_SECTOR_BITS )));
...then computing the length by undoing the rounding. I really think we should consider fixing bdrv_get_block_status_above() to be byte-based, but that's a separate series. Your calculations look correct in the meantime, although '(offset & (BDRV_SECTOR_SIZE - 1))' may be a bit easier to read than '(offset - (start_sector << BDRV_SECTOR_BITS))'.
include/block/nbd.h static void nbd_trip(void *opaque) · line 1933
Quoted 6 lines
include/block/nbd.h +1
-1@@ -1869,13 +1930,18 @@ static void nbd_trip(void *opaque)break ;case NBD_CMD_BLOCK_STATUS :TRACE ("Request type is BLOCK_STATUS" ); -if (client->export_bitmap ==NULL ) { +if (!!client->export_bitmap) {
!! is not necessary here.
include/block/nbd.h static void nbd_trip(void *opaque) · line 1940
Quoted 7 lines
include/block/nbd.h +7
-0+ ret =nbd_co_send_bitmap (req->client, request.handle, + client->export_bitmap, request.from, + request.len,0 ); + }else if (client->export_block_status) { + ret =nbd_co_send_block_status (req->client, request.handle, +blk_bs (exp->blk), request.from, + request.len,0 );
Umm, why are we sending only one status? If the client requests two ids during NBD_OPT_SET_META_CONTEXT, we should be able to provide both pieces of information at once. For a minimal implementation, it works for proof of concept, but it is pretty restrictive to tell clients that they can only request a single status context. I'm fine if we add that functionality in a later patch, but we'd better have the implementation ready for the same release as this patch (I still think 2.9 is a reasonable goal).
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 17/18] nbd: BLOCK_STATUS for standard get_block_status function: server part
Quoted 19 lines · Eric Blake
nbd/server.c static unsigned add_extents(NBDExtent *extents, unsigned nb_extents, · line 1527
Quoted 6 lines
nbd/server.c +6
-0+static int blockstatus_to_extent (BlockDriverState *bs,uint64_t offset, +uint64_t length,NBDExtent *extent) +{ +BlockDriverState *file; +uint64_t start_sector = offset >>BDRV_SECTOR_BITS ; +uint64_t last_sector = (offset + length -1 ) >>BDRV_SECTOR_BITS ;Converting from bytes to sectors by rounding...
Quoted 11 lines
nbd/server.c +11
-0+uint64_t begin = start_sector; +uint64_t end = last_sector +1 ; + +int nb =MIN (INT_MAX , end - begin); +int64_t ret =bdrv_get_block_status_above (bs,NULL , begin, nb, &nb, &file); +if (ret <0 ) { +return ret; + } + + extent->flags = + cpu_to_be32((ret & BDRV_BLOCK_ALLOCATED ? 0 : NBD_STATE_HOLE) |
Quoted 8 lines
nbd/server.c static unsigned add_extents(NBDExtent *extents, unsigned nb_extents, · line 1541
Quoted 3 lines
nbd/server.c +3
-0+ (ret & BDRV_BLOCK_ZERO ? NBD_STATE_ZERO : 0)); + extent->length =cpu_to_be32 ((nb <<BDRV_SECTOR_BITS ) - + (offset - (start_sector <<BDRV_SECTOR_BITS )));...then computing the length by undoing the rounding. I really think we should consider fixing
bdrv_get_block_status_above()to be byte-based, but that's a separate series. Your calculations look correct in the meantime, although '(offset & (BDRV_SECTOR_SIZE- 1))' may be a bit easier to read than '(offset - (start_sector<<BDRV_SECTOR_BITS))'.
Agreed. And please make it a separate variable, i.e.
uint64_t length;
length = (nb << BDRV_SECTOR_BITS) - (offset & BDRV_SECTOR_SIZE - 1);
...
extent->length = cpu_to_be32(length);Paolo
Paolo
Re: [Qemu-devel] [PATCH 17/18] nbd: BLOCK_STATUS for standard get_block_status function: server part
In reply to Eric BlakeQuoted 8 lines · Eric Blake
Umm, why are we sending only one status? If the client requests two ids during
NBD_OPT_SET_META_CONTEXT, we should be able to provide both pieces of information at once. For a minimal implementation, it works for proof of concept, but it is pretty restrictive to tell clients that they can only request a single status context. I'm fine if we add that functionality in a later patch, but we'd better have the implementation ready for the same release as this patch (I still think 2.9 is a reasonable goal).
Agreed on this too.
Paolo
PATCH 18/18 nbd: BLOCK_STATUS for standard get_block_status function: client part
In reply to Vladimir Sementsov-OgievskiyMinimal realization: only first extent from the answer is used.
--- block/nbd-client.c | 41 ++++++++++++++++++++++++++++++++++++++++- block/nbd-client.h | 5 +++++ block/nbd.c | 3 +++ include/block/nbd.h | 2 +- nbd/client.c | 23 ++++++++++++++++------- qemu-nbd.c | 2 +- 6 files changed, 66 insertions(+), 10 deletions(-)
block/nbd-client.c +40 -1
@@ -528,6 +528,44 @@ int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, (uint64_t)bdrv_dirty_bitmap_granularity(bitmap)); } +int64_t coroutine_fn nbd_client_co_get_block_status(BlockDriverState *bs, +int64_t sector_num, +int nb_sectors,int *pnum, +BlockDriverState **file) +{ +int64_t ret; +uint32_t nb_extents; +NBDExtent *extents; +NBDClientSession *client =nbd_get_client_session (bs); + +if (!client->block_status_ok) { + *pnum = nb_sectors; + ret =BDRV_BLOCK_DATA |BDRV_BLOCK_ALLOCATED ; +if (bs->drv->protocol_name) { + ret |=BDRV_BLOCK_OFFSET_VALID | (sector_num *BDRV_SECTOR_SIZE ); + } +return ret; + } + + ret =nbd_client_co_cmd_block_status (bs, sector_num <<BDRV_SECTOR_BITS , + nb_sectors <<BDRV_SECTOR_BITS , + &extents, &nb_extents); +if (ret <0 ) { +return ret; + } + + *pnum = extents[0 ].length >>BDRV_SECTOR_BITS ; + ret = (extents[0 ].flags &NBD_STATE_HOLE ?0 :BDRV_BLOCK_ALLOCATED ) | + (extents[0 ].flags &NBD_STATE_ZERO ?BDRV_BLOCK_ZERO :0 ); + +if ((ret &BDRV_BLOCK_ALLOCATED ) && !(ret &BDRV_BLOCK_ZERO )) { + ret |=BDRV_BLOCK_DATA ; + } + +g_free (extents); + +return ret; +}void nbd_client_detach_aio_context (BlockDriverState *bs) {
@@ -579,7 +617,8 @@ int nbd_client_init(BlockDriverState *bs, &client->size, &client->structured_reply, bitmap_name, - &client->bitmap_ok, errp); + &client->bitmap_ok, + &client->block_status_ok, errp);if (ret <0 ) {logout ("Failed to negotiate with the NBD server\n" );return ret;
block/nbd-client.h +5 -0
@@ -34,6 +34,7 @@ typedef struct NBDClientSession {bool is_unix;bool structured_reply; +bool block_status_ok;bool bitmap_ok;uint32_t meta_data_context_id; } NBDClientSession;
@@ -59,6 +60,10 @@ int nbd_client_co_preadv(BlockDriverState *bs, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags);int64_t nbd_client_co_load_bitmap_part (BlockDriverState *bs,uint64_t offset,uint64_t bytes,BdrvDirtyBitmap *bitmap); +int64_t coroutine_fn nbd_client_co_get_block_status(BlockDriverState *bs, +int64_t sector_num, +int nb_sectors,int *pnum, +BlockDriverState **file);void nbd_client_detach_aio_context (BlockDriverState *bs); void nbd_client_attach_aio_context(BlockDriverState *bs,
block/nbd.c +3 -0
@@ -604,6 +604,7 @@ static BlockDriver bdrv_nbd = { .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, + .bdrv_co_get_block_status = nbd_client_co_get_block_status, };static BlockDriver bdrv_nbd_tcp = {
@@ -624,6 +625,7 @@ static BlockDriver bdrv_nbd_tcp = { .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, + .bdrv_co_get_block_status = nbd_client_co_get_block_status, };static BlockDriver bdrv_nbd_unix = {
@@ -644,6 +646,7 @@ static BlockDriver bdrv_nbd_unix = { .bdrv_attach_aio_context = nbd_attach_aio_context, .bdrv_refresh_filename = nbd_refresh_filename, .bdrv_dirty_bitmap_load = nbd_dirty_bitmap_load, + .bdrv_co_get_block_status = nbd_client_co_get_block_status, };static void bdrv_nbd_init (void )
include/block/nbd.h +1 -1
@@ -196,7 +196,7 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, QIOChannel **outioc, off_t *size, bool *structured_reply, const char *bitmap_name, bool *bitmap_ok, - Error **errp); + bool *block_status_ok, Error **errp);int nbd_init (int fd,QIOChannelSocket *sioc,uint16_t flags,off_t size);ssize_t nbd_send_request (QIOChannel *ioc,NBDRequest *request);int nbd_receive_reply (QIOChannel *ioc,NBDReply *reply);
nbd/client.c +16 -7
@@ -563,10 +563,10 @@ static int nbd_receive_query_bitmap(QIOChannel *ioc, const char *export,int nbd_receive_negotiate (QIOChannel *ioc,const char *name,uint16_t *flags,QCryptoTLSCreds *tlscreds,const char *hostname, -QIOChannel **outioc, -off_t *size,bool *structured_reply, +QIOChannel **outioc,off_t *size, +bool *structured_reply,const char *bitmap_name,bool *bitmap_ok, -Error **errp) +bool *block_status_ok,Error **errp) {char buf[256 ];uint64_t magic, s;
@@ -681,11 +681,19 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, false, NULL) == 0; } -if (!!structured_reply && *structured_reply && !!bitmap_name) { +if (!!structured_reply && *structured_reply) {int ret; -assert (!!bitmap_ok); - ret =nbd_receive_query_bitmap (ioc, name, bitmap_name, - bitmap_ok, errp) ==0 ; + +if (!!bitmap_name) { +assert (!!bitmap_ok); + ret =nbd_receive_query_bitmap (ioc, name, bitmap_name, + bitmap_ok, errp) ==0 ; + }else { + ret =nbd_receive_query_meta_context (ioc, name, +"base:allocation" , + block_status_ok, + errp); + }if (ret <0 ) { goto fail; }
@@ -969,6 +977,7 @@ static int nbd_receive_structured_reply_chunk(QIOChannel *ioc, NBDReply *reply)switch (reply->type) {case NBD_REPLY_TYPE_NONE : +case NBD_REPLY_TYPE_BLOCK_STATUS :break ;case NBD_REPLY_TYPE_OFFSET_DATA :case NBD_REPLY_TYPE_OFFSET_HOLE :
qemu-nbd.c +1 -1
@@ -272,7 +272,7 @@ static void *nbd_client_thread(void *arg) ret =nbd_receive_negotiate (QIO_CHANNEL (sioc),NULL , &nbdflags,NULL ,NULL ,NULL , - &size,NULL ,NULL ,NULL , &local_error); + &size,NULL ,NULL ,NULL ,NULL , &local_error);if (ret <0 ) {if (local_error) {error_report_err (local_error);
Signature · 1 line
2.11.0
Re: [Qemu-devel] [PATCH 18/18] nbd: BLOCK_STATUS for standard get_block_status function: client part
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Minimal realization: only first extent from the answer is used.
If you're only going to use one extent, should you implement NBD_CMD_FLAG_REQ_ONE support to save the server some effort?
Quoted 19 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/nbd-client.c | 41 ++++++++++++++++++++++++++++++++++++++++- block/nbd-client.h | 5 +++++ block/nbd.c | 3 +++ include/block/nbd.h | 2 +- nbd/client.c | 23 ++++++++++++++++------- qemu-nbd.c | 2 +- 6 files changed, 66 insertions(+), 10 deletions(-)block/nbd-client.c +8
-0+int64_t coroutine_fn nbd_client_co_get_block_status(BlockDriverState *bs, +int64_t sector_num, +int nb_sectors,int *pnum, +BlockDriverState **file) +{ +int64_t ret; +uint32_t nb_extents; +NBDExtent *extents;
block/nbd-client.c int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, · line 548
Quoted 10 lines
block/nbd-client.c +10
-0+NBDClientSession *client =nbd_get_client_session (bs); + +if (!client->block_status_ok) { + *pnum = nb_sectors; + ret =BDRV_BLOCK_DATA |BDRV_BLOCK_ALLOCATED ; +if (bs->drv->protocol_name) { + ret |=BDRV_BLOCK_OFFSET_VALID | (sector_num *BDRV_SECTOR_SIZE ); + } +return ret; + }
Looks like a sane fallback when we don't have anything more accurate.
Quoted 5 lines
block/nbd-client.c +5
-0+ + ret =nbd_client_co_cmd_block_status (bs, sector_num <<BDRV_SECTOR_BITS , + nb_sectors <<BDRV_SECTOR_BITS , + &extents, &nb_extents); +if (ret <0 ) {
block/nbd-client.c int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, · line 563
Quoted 10 lines
block/nbd-client.c +10
-0+return ret; + } + + *pnum = extents[0 ].length >>BDRV_SECTOR_BITS ; + ret = (extents[0 ].flags &NBD_STATE_HOLE ?0 :BDRV_BLOCK_ALLOCATED ) | + (extents[0 ].flags &NBD_STATE_ZERO ?BDRV_BLOCK_ZERO :0 ); + +if ((ret &BDRV_BLOCK_ALLOCATED ) && !(ret &BDRV_BLOCK_ZERO )) { + ret |=BDRV_BLOCK_DATA ; + }
And this conversion looks correct.
block/nbd-client.c int nbd_client_init(BlockDriverState *bs, · line 621
Quoted 7 lines
block/nbd-client.c +2
-1@@ -579,7 +617,8 @@ int nbd_client_init(BlockDriverState *bs, &client->size, &client->structured_reply, bitmap_name, - &client->bitmap_ok, errp); + &client->bitmap_ok, + &client->block_status_ok, errp);
That's a lot of parameters we're adding; I'm wondering if its smarter to start passing things through via a struct. In fact, I have posted patches previously (and need to rebase and repost them) that introduce such a struct, in order to make the INFO extension viable.
Quoted 11 lines
nbd/client.c +2
-4@@ -681,11 +681,19 @@ int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, false, NULL) == 0; } -if (!!structured_reply && *structured_reply && !!bitmap_name) { +if (!!structured_reply && *structured_reply) {int ret; -assert (!!bitmap_ok); - ret =nbd_receive_query_bitmap (ioc, name, bitmap_name, - bitmap_ok, errp) ==0 ; +
nbd/client.c int nbd_receive_negotiate(QIOChannel *ioc, const char *name, uint16_t *flags, · line 696
Quoted 10 lines
nbd/client.c +10
-0+if (!!bitmap_name) { +assert (!!bitmap_ok); + ret =nbd_receive_query_bitmap (ioc, name, bitmap_name, + bitmap_ok, errp) ==0 ; + }else { + ret =nbd_receive_query_meta_context (ioc, name, +"base:allocation" , + block_status_ok, + errp); + }
This looks weird trying to grab 'base:allocation' only if bitmap_name is not provided. The logic will probably have to be different if we want to allow a client to track both pieces of information at once.
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Re: [Qemu-devel] [PATCH 18/18] nbd: BLOCK_STATUS for standard get_block_status function: client part
Quoted 4 lines · Eric Blake
block/nbd-client.c int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, · line 544
Quoted 4 lines
block/nbd-client.c +4
-0+if (!client->block_status_ok) { + *pnum = nb_sectors; + ret =BDRV_BLOCK_DATA |BDRV_BLOCK_ALLOCATED ; +if (bs->drv->protocol_name) {
This condition is always true, I think?
Quoted 6 lines
block/nbd-client.c int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, · line 548
Quoted 4 lines
block/nbd-client.c +4
-0+ ret |=BDRV_BLOCK_OFFSET_VALID | (sector_num *BDRV_SECTOR_SIZE ); + } +return ret; + }Looks like a sane fallback when we don't have anything more accurate.
Quoted 15 lines
Quoted 5 lines
block/nbd-client.c +5
-0+ + ret =nbd_client_co_cmd_block_status (bs, sector_num <<BDRV_SECTOR_BITS , + nb_sectors <<BDRV_SECTOR_BITS , + &extents, &nb_extents); +if (ret <0 ) {block/nbd-client.c int64_t nbd_client_co_load_bitmap_part(BlockDriverState *bs, uint64_t offset, · line 563
Quoted 10 lines
block/nbd-client.c +10
-0+return ret; + } + + *pnum = extents[0 ].length >>BDRV_SECTOR_BITS ; + ret = (extents[0 ].flags &NBD_STATE_HOLE ?0 :BDRV_BLOCK_ALLOCATED ) | + (extents[0 ].flags &NBD_STATE_ZERO ?BDRV_BLOCK_ZERO :0 ); + +if ((ret &BDRV_BLOCK_ALLOCATED ) && !(ret &BDRV_BLOCK_ZERO )) { + ret |=BDRV_BLOCK_DATA ; + }
You can always return BDRV_BLOCK_OFFSET_VALID here, too.
Paolo
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Hi all!
We really need exporting dirty bitmaps feature as well as remote
get_block_statusfor nbd devices. So, here is minimalistic and restricted realization of 'structured reply' and 'block status' nbd protocol extension (as second is developed over the first the combined spec may be found here:
https://github.com/NetworkBlockDevice/nbd/blob/extension-blockstatus/doc/proto.md)What is done:
server:
- can send not fragmented structured replies forCMD_READ(this was done only because the spec doesn't allow structure reply feature without maintaining structured read)
- can export dirty bitmap throughBLOCK_STATUS. Only one bitmap can be exported, negotiation query should be 'qemu-dirty-bitmap:<bitmap name>'
- cab export block status throughBLOCK_STATUS. Client can negotiate only one entity for exporting throughBLOCK_STATUS- bitmap _or_ block status. negotiation query should be 'base:allocation', as defined in the spec. server sends only one extent on eachBLOCK_STATUSquery.client:
- can receive not fragmented structured replies forCMD_READ
- can get load dirty bitmap through nbd. Be careful: bitmap for export is is selected during nbd negotiation - actually inopen(). So, name argument for qmp block-dirty-bitmap-load is just a _new_ name for loaded bitmap. (any way, for us block-dirty-bitmap-load is just a way to test the feature, really we need only server part)
-get_block_statusworks now through nbdCMD_BLOCK_STATUS, if base:allocation is negotiated for export.
Quoted 1 line
It should be minimal but fully compatible realization of the spec.
This will require v2, but it seems pretty close already.
Paolo
Quoted 47 lines
web: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=nbd-block-status-v1
git: https://src.openvz.org/scm/~vsementsov/qemu.git (tag nbd-block-status-v1)Vladimir Sementsov-Ogievskiy (18):
nbd: renameNBD_REPLY_MAGICtoNBD_SIMPLE_REPLY_MAGIC
nbd-server: refactor simple reply sending
nbd: Minimal structured read for server
nbd/client: refactornbd_receive_starttls
nbd/client: fixdrop_sync
nbd/client: refactordrop_sync
nbd: Minimal structured read for client
hbitmap: addnext_zerofunction
block/dirty-bitmap: addbdrv_dirty_bitmap_next()block/dirty-bitmap: addbdrv_load_dirty_bitmap
nbd:BLOCK_STATUSfor bitmap export: server part
nbd:BLOCK_STATUSfor bitmap export: client part
nbd: addnbd_dirty_bitmap_load
qmp: add x-debug-block-dirty-bitmap-sha256
qmp: add block-dirty-bitmap-load
iotests: add test for nbd dirty bitmap export
nbd:BLOCK_STATUSfor standardget_block_statusfunction: server part
nbd:BLOCK_STATUSfor standardget_block_statusfunction: client partblock/dirty-bitmap.c | 70 ++++ block/nbd-client.c | 230 ++++++++++- block/nbd-client.h | 13 + block/nbd.c | 44 +- blockdev.c | 57 +++ include/block/block_int.h | 4 + include/block/dirty-bitmap.h | 10 + include/block/nbd.h | 73 +++- include/qemu/hbitmap.h | 16 + nbd/client.c | 373 ++++++++++++++--- nbd/nbd-internal.h | 25 +- nbd/server.c | 661 ++++++++++++++++++++++++++++++- qapi/block-core.json | 46 ++- qemu-nbd.c | 2 +- tests/Makefile.include | 2 +- tests/qemu-iotests/180 | 133 +++++++ tests/qemu-iotests/180.out | 5 + tests/qemu-iotests/group | 1 + tests/qemu-iotests/nbd-fault-injector.py | 4 +- util/hbitmap.c | 37 ++ 20 files changed, 1713 insertions(+), 93 deletions(-) create mode 100755 tests/qemu-iotests/180 create mode 100644 tests/qemu-iotests/180.out
On 02/15/2017 11:05 AM, Paolo Bonzini wrote:
[now several months later...]
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
Quoted 10 lines
Hi all!
We really need exporting dirty bitmaps feature as well as remote
get_block_statusfor nbd devices. So, here is minimalistic and restricted realization of 'structured reply' and 'block status' nbd protocol extension (as second is developed over the first the combined spec may be found here:
https://github.com/NetworkBlockDevice/nbd/blob/extension-blockstatus/doc/proto.md)
Quoted 3 lines
Quoted 1 line
It should be minimal but fully compatible realization of the spec.
This will require v2, but it seems pretty close already.
This series now needs a major rebase on top of the cleanups already present for qemu 2.10. Realistically, I'm aiming to get my series [1] for NBD_OPT_GO into 2.10 (we have less than a week before soft freeze, so any reviews offered on that series will be appreciated), then the work on adding structured read and BLOCK_STATUS will probably be big enough that it will have to be 2.11 material, but getting it posted sooner rather than later will make it easier to fine-tune all the details.
[1] https://lists.gnu.org/archive/html/qemu-devel/2017-07/msg01971.html
And remember that in open source projects, review can often be the bottleneck - it does not scale to have reviews done solely by the people listed in MAINTAINERS. My personal rule of thumb is that I try to review 2 patches for every 1 that I submit, to make sure I'm not contributing to a black hole of unreviewed patch backlog (although no one will ever turn down a patch from a contributor unable to follow that advice). Other benefits of offering reviews: you quickly become more familiar with the code base, and you earn some name-recognition in the community (where it is more likely that your own patches get reviewed quickly because someone recognizes that you are a regular contributor). It's okay to state on a review that you are not familiar/comfortable with the code, and want a second reviewer to double-check your work.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org