nbd: accept a large NBD_CMD_CACHE, cover the transmission commands
v2 cover nbd: accept a large NBD_CMD_CACHE, cover the transmission commands
NBD_CMD_CACHE carries no payload in either direction, yet the server rejects any effect length above the advertised maximum payload size. The specification explicitly allows the oversize request, and we already accept it for NBD_CMD_TRIM and NBD_CMD_WRITE_ZEROES. Patch 2 drops the check.
The rest of the series adds tests/qemu-iotests/tests/nbd-commands, a small client which speaks the NBD transmission phase itself. Nothing in the tree does that today, so the protocol surface of the server - reply modes, command flags, error codes, sparse and extended replies - is covered only as far as our own client happens to exercise it. It is useful both as coverage and as a regression tool.
Changes since v1:
- the libnbd handle now lives in setUp() and tearDown() instead of a context manager entered by every test case, which drops a level of indent everywhere (Vladimir)
- one block_status() helper returning a context to extents map, in place of the three near identical extent collecting callbacks (Vladimir)
- the unused callback arguments are named with a leading underscore, so that iotest 297 stays pylint clean; v1 added eight unused-argument warnings, which would have failed 297 wherever pylint is installed
- Vladimir's Reviewed-by is kept on patch 2, which is unchanged, and dropped from the four test patches, since the two points above rewrote most of the test
v1: https://lore.kernel.org/qemu-devel/[email protected]/
Denis V. Lunev (5):
iotests: add coverage for NBD transmission commands nbd/server: accept NBD_CMD_CACHE above the maximum payload size iotests/nbd-commands: exercise the simple and structured reply modes iotests/nbd-commands: cover NBD_CMD_BLOCK_STATUS with a payload iotests/nbd-commands: cover the command flags and sparse replies
nbd/server.c | 4 +- tests/qemu-iotests/tests/nbd-commands | 244 ++++++++++++++++++++++ tests/qemu-iotests/tests/nbd-commands.out | 5 + 3 files changed, 250 insertions(+), 3 deletions(-) create mode 100755 tests/qemu-iotests/tests/nbd-commands create mode 100644 tests/qemu-iotests/tests/nbd-commands.out
base-commit: bde2492aace2b5acb755a5b057013e915163a77f
Signature · 1 line
2.53.0
PATCH v2 1/5 iotests: add coverage for NBD transmission commands
From: Denis V. Lunev <[email protected]>
NBD_CMD_CACHE has no coverage anywhere in the tree. Nothing ever sends it: our own NBD client does not implement the command, and neither qemu-io nor 'qemu-nbd --list' can issue one, so the only clients reaching this server path are external ones.
Add a test driven by libnbd, gated the way nbd-multiconn already is, and start it with the case the command exists for. The export is a qcow2 image over a fully written backing file, so a prefetch has visible work to do.
--- tests/qemu-iotests/tests/nbd-commands | 140 ++++++++++++++++++++++ tests/qemu-iotests/tests/nbd-commands.out | 5 + 2 files changed, 145 insertions(+) create mode 100755 tests/qemu-iotests/tests/nbd-commands create mode 100644 tests/qemu-iotests/tests/nbd-commands.out
tests/qemu-iotests/tests/nbd-commands new file +140 -0
@@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +# group: rw auto quick +# +# Test NBD transmission commands against a qemu NBD export +# +# Copyright (C) 2026 Virtuozzo International GmbH +# +# SPDX-License-Identifier: GPL-2.0-or-later + +import os +from types import ModuleType + +import iotests +from iotests import qemu_img_create, qemu_img_map, qemu_io + + +base = os.path.join(iotests.test_dir, 'base') +top = os.path.join(iotests.test_dir, 'top') +# Larger than the maximum payload size an export can advertise +size = 64 * 1024 * 1024 +pattern = 0xa5 +nbd_sock = os.path.join(iotests.sock_dir, 'nbd_sock') +nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock +nbd: ModuleType + +DEPTH_LOCAL = 1 +DEPTH_BACKING = 2 + + +class TestNbdCommands(iotests.QMPTestCase): + def setUp(self): + qemu_img_create('-f', iotests.imgfmt, base, str(size)) + qemu_io('-c', f'write -P {pattern} 0 {size}', base) + qemu_img_create('-f', iotests.imgfmt, '-b', base, + '-F', iotests.imgfmt, top, str(size)) + + self.vm = iotests.VM() + self.vm.launch() + self.vm.cmd('blockdev-add', { + 'driver': iotests.imgfmt, + 'node-name': 'n', + 'file': {'driver': 'file', 'filename': top}, + 'backing': { + 'driver': iotests.imgfmt, + 'node-name': 'base', + 'file': {'driver': 'file', 'filename': base}, + }, + }) + self.vm.cmd('nbd-server-start', { + 'addr': {'type': 'unix', 'data': {'path': nbd_sock}} + }) + self.vm.cmd('block-export-add', { + 'type': 'nbd', + 'id': 'exp', + 'node-name': 'n', + 'name': 'exp', + 'writable': True, + 'allocation-depth': True, + }) + + self.h = None + self.connect() + + def tearDown(self): + self.disconnect() + self.vm.shutdown() + for f in (top, base, nbd_sock): + try: + os.remove(f) + except OSError: + pass + + def connect(self, structured=True, extended=True): + self.disconnect() + h = nbd.NBD() + h.set_request_structured_replies(structured) + h.set_request_extended_headers(extended) + h.add_meta_context('base:allocation') + h.add_meta_context('qemu:allocation-depth') + # Let the server, not libnbd, reject the out of range requests below + h.set_strict_mode(h.get_strict_mode() & + ~(nbd.STRICT_BOUNDS | nbd.STRICT_PAYLOAD)) + h.connect_uri(nbd_uri) + self.assertEqual(h.get_structured_replies_negotiated(), structured) + self.assertEqual(h.get_extended_headers_negotiated(), extended) + self.h = h + + def disconnect(self): + if self.h is not None: + self.h.shutdown() + self.h = None + + def block_status(self, count=size): + """Map each meta context in the reply to its list of extents.""" + reply = {} + + def cb(meta, _offset, entries, _err): + reply.setdefault(meta, []).extend(zip(entries[0::2], + entries[1::2])) + + self.h.block_status(count, 0, cb) + return reply + + def top_extents(self): + """Which parts of the top image are local, once qemu has let go.""" + self.disconnect() + self.vm.shutdown() + return [(e['start'], e['length'], e['depth']) + for e in qemu_img_map(top)] + + def test_cache_copies_on_read(self): + maximum = self.h.get_block_size(nbd.SIZE_MAXIMUM) + self.assertLess(maximum, size) + self.assertEqual(self.block_status()['qemu:allocation-depth'], + [(size, DEPTH_BACKING)]) + + self.h.cache(maximum, 0) + + self.assertEqual(self.top_extents(), + [(0, maximum, 0), (maximum, size - maximum, 1)]) + qemu_io('-c', f'read -P {pattern} 0 {size}', top) + + def test_cache_past_end_of_export(self): + self.assertRaises(nbd.Error, self.h.cache, size + 1, 0) + + def test_read_bound_by_max_payload(self): + maximum = self.h.get_block_size(nbd.SIZE_MAXIMUM) + self.assertRaises(nbd.Error, self.h.pread, maximum + 65536, 0) + + +if __name__ == '__main__': + try: + # Easier to use libnbd than to try and set up parallel + # 'qemu-nbd --list' or 'qemu-io' processes, but not all systems + # have libnbd installed. + import nbd # type: ignore + + iotests.main(supported_fmts=['qcow2']) + except ImportError: + iotests.notrun('Python bindings to libnbd are not installed')
tests/qemu-iotests/tests/nbd-commands.out new file +5 -0
@@ -0,0 +1,5 @@ +... +---------------------------------------------------------------------- +Ran 3 tests + +OK
Signature · 1 line
2.53.0
PATCH v2 2/5 nbd/server: accept NBD_CMD_CACHE above the maximum payload size
In reply to Denis V. LunevFrom: Denis V. Lunev <[email protected]>
NBD_CMD_CACHE carries no payload in either direction. The request is a header only, nbd_do_cmd_cache() passes a NULL qiov to blk_co_preadv(), and the reply is a bare status. Still the server rejects any effect length above NBD_MAX_BUFFER_SIZE with EINVAL, which forces a client to split a large prefetch into 32 MiB pieces.
The specification does not ask for this. The constraint was renamed from "maximum block size" to "maximum payload size" precisely to separate payload length from effect length, and it says:
For commands that do not require a payload in either direction (such as NBD_CMD_TRIM or NBD_CMD_WRITE_ZEROES), the client MAY request an effect length larger than the maximum payload size; the server SHOULD NOT disconnect, but MAY reply with an NBD_EOVERFLOW or NBD_EINVAL error if the oversize request would require too many server resources when compared to the same command with an effect length limited to the maximum payload size (such as an implementation of NBD_CMD_WRITE_ZEROES that utilizes a scratch buffer).
We already follow that for NBD_CMD_TRIM and NBD_CMD_WRITE_ZEROES, which carry no length check at all, and our client assumes a server supporting extended headers takes unlimited zero and trim lengths. Handle NBD_CMD_CACHE in the same way.
--- nbd/server.c | 4 +--- tests/qemu-iotests/tests/nbd-commands | 5 ++--- 2 files changed, 3 insertions(+), 6 deletions(-)
nbd/server.c +1 -3
@@ -2718,7 +2718,6 @@ static int coroutine_fn nbd_co_receive_request(NBDRequestData *req,break ;case NBD_CMD_CACHE : - check_length = true;break ;case NBD_CMD_WRITE_ZEROES :
@@ -2752,7 +2751,7 @@ static int coroutine_fn nbd_co_receive_request(NBDRequestData *req, req->complete = true; }if (check_length && request->len >NBD_MAX_BUFFER_SIZE ) { -/* READ, WRITE, CACHE */ +/* READ, WRITE */ error_setg (errp,"len (%" PRIu64") is larger than max len (%u)" , request->len,NBD_MAX_BUFFER_SIZE );return -EINVAL ;
@@ -2908,7 +2907,6 @@ static coroutine_fn int nbd_do_cmd_cache(NBDClient *client, NBDRequest *request,NBDExport *exp = client->exp;assert (request->type ==NBD_CMD_CACHE ); -assert (request->len <=NBD_MAX_BUFFER_SIZE ); ret =blk_co_preadv (exp->common.blk, request->from, request->len,NULL ,BDRV_REQ_COPY_ON_READ |BDRV_REQ_PREFETCH );
tests/qemu-iotests/tests/nbd-commands +2 -3
@@ -114,10 +114,9 @@ class TestNbdCommands(iotests.QMPTestCase): self.assertEqual(self.block_status()['qemu:allocation-depth'], [(size, DEPTH_BACKING)]) - self.h.cache(maximum, 0) + self.h.cache(size, 0) - self.assertEqual(self.top_extents(), - [(0, maximum, 0), (maximum, size - maximum, 1)]) + self.assertEqual(self.top_extents(), [(0, size, 0)]) qemu_io('-c', f'read -P {pattern} 0 {size}', top) def test_cache_past_end_of_export(self):
Signature · 1 line
2.53.0
PATCH v2 3/5 iotests/nbd-commands: exercise the simple and structured reply modes
In reply to Denis V. LunevFrom: Denis V. Lunev <[email protected]>
Every NBD request the iotests send arrives in NBD_MODE_EXTENDED, so the server paths kept for older clients are never taken. Instrumenting nbd_co_receive_request() over the whole auto group and every test that touches NBD gives mode 4 for all of them, which leaves nbd_co_send_simple_reply() and the compact header handling dead under test. A regression there would only show against a third party client.
libnbd can negotiate down, so run the same command set three times, once per mode, and assert the mode that was actually reached. Block status is limited to the two modes that can negotiate a meta context.
--- tests/qemu-iotests/tests/nbd-commands | 29 +++++++++++++++++++++++ tests/qemu-iotests/tests/nbd-commands.out | 4 ++-- 2 files changed, 31 insertions(+), 2 deletions(-)
tests/qemu-iotests/tests/nbd-commands +29 -0
@@ -119,6 +119,35 @@ class TestNbdCommands(iotests.QMPTestCase): self.assertEqual(self.top_extents(), [(0, size, 0)]) qemu_io('-c', f'read -P {pattern} 0 {size}', top) + def check_commands(self, structured, extended): + self.connect(structured, extended) + + self.assertEqual(self.h.pread(4096, 4096), bytes([pattern]) * 4096) + + self.h.cache(size, 0) + + self.h.pwrite(b'x' * 4096, 4096) + self.h.flush() + self.assertEqual(self.h.pread(4096, 4096), b'x' * 4096) + + self.h.zero(4096, 8192) + self.assertEqual(self.h.pread(4096, 8192), bytes(4096)) + + self.h.trim(4096, 16384) + + if structured: + self.assertEqual(self.block_status()['qemu:allocation-depth'], + [(size, DEPTH_LOCAL)]) + + def test_commands_simple_replies(self): + self.check_commands(structured=False, extended=False) + + def test_commands_structured_replies(self): + self.check_commands(structured=True, extended=False) + + def test_commands_extended_headers(self): + self.check_commands(structured=True, extended=True) + def test_cache_past_end_of_export(self): self.assertRaises(nbd.Error, self.h.cache, size + 1, 0)
tests/qemu-iotests/tests/nbd-commands.out +2 -2
@@ -1,5 +1,5 @@ -... +...... ---------------------------------------------------------------------- -Ran 3 tests +Ran 6 tests OK
Signature · 1 line
2.53.0
PATCH v2 4/5 iotests/nbd-commands: cover NBD_CMD_BLOCK_STATUS with a payload
In reply to Denis V. LunevFrom: Denis V. Lunev <[email protected]>
With extended headers a client may name the meta contexts it wants in a request payload. Our own client never sends one, so the server side of it, nbd_co_block_status_payload_read(), is not exercised anywhere.
Export two meta contexts and ask libnbd for each of them in turn, which is only answerable through that path.
--- tests/qemu-iotests/tests/nbd-commands | 14 ++++++++++++-- tests/qemu-iotests/tests/nbd-commands.out | 4 ++-- 2 files changed, 14 insertions(+), 4 deletions(-)
tests/qemu-iotests/tests/nbd-commands +12 -2
@@ -90,7 +90,7 @@ class TestNbdCommands(iotests.QMPTestCase): self.h.shutdown() self.h = None - def block_status(self, count=size): + def block_status(self, count=size, wanted=None): """Map each meta context in the reply to its list of extents.""" reply = {}
@@ -98,7 +98,10 @@ class TestNbdCommands(iotests.QMPTestCase): reply.setdefault(meta, []).extend(zip(entries[0::2], entries[1::2])) - self.h.block_status(count, 0, cb) + if wanted is None: + self.h.block_status(count, 0, cb) + else: + self.h.block_status_filter(count, 0, wanted, cb) return reply def top_extents(self):
@@ -148,6 +151,13 @@ class TestNbdCommands(iotests.QMPTestCase): def test_commands_extended_headers(self): self.check_commands(structured=True, extended=True) + def test_block_status_payload_filter(self): + self.assertEqual(sorted(self.block_status()), + ['base:allocation', 'qemu:allocation-depth']) + + for wanted in (['base:allocation'], ['qemu:allocation-depth']): + self.assertEqual(sorted(self.block_status(wanted=wanted)), wanted) + def test_cache_past_end_of_export(self): self.assertRaises(nbd.Error, self.h.cache, size + 1, 0)
tests/qemu-iotests/tests/nbd-commands.out +2 -2
@@ -1,5 +1,5 @@ -...... +....... ---------------------------------------------------------------------- -Ran 6 tests +Ran 7 tests OK
Signature · 1 line
2.53.0
PATCH v2 5/5 iotests/nbd-commands: cover the command flags and sparse replies
In reply to Denis V. LunevFrom: Denis V. Lunev <[email protected]>
No test sends the NBD command flags, so the server paths behind them go unexercised. Send them from the new client.
A structured read answers a hole with an offset and a length rather than a cluster of zeroes. Lay out data, a hole and data again, and read the three arrangements a hole can appear in, since a leading hole and a hole between two extents take different turns through nbd_co_send_sparse_read(). NBD_CMD_FLAG_DF asks for one chunk instead, which is the same layout sent as real zeroes. Check the chunk boundaries in both forms, and that the data still reads back.
NBD_CMD_FLAG_REQ_ONE caps the extent array at one entry, so the block status reply covers only the first cluster of the three asked about. That a reply may describe less than was requested is also how the server keeps NBD_MAX_BLOCK_STATUS_EXTENTS from being exceeded on a long fragmented range, and a client assuming full coverage believes stale status.
NBD_CMD_FLAG_FAST_ZERO becomes BDRV_REQ_NO_FALLBACK. A cluster aligned zero can be done by marking the cluster, while zeroing part of a cluster over a backing file needs the read modify write the flag forbids, so check both the success and the ENOTSUP.
NBD_CMD_FLAG_FUA is added to a write and a trim.
--- tests/qemu-iotests/tests/nbd-commands | 72 ++++++++++++++++++++++- tests/qemu-iotests/tests/nbd-commands.out | 4 +- 2 files changed, 71 insertions(+), 5 deletions(-)
tests/qemu-iotests/tests/nbd-commands +69 -3
@@ -25,6 +25,7 @@ nbd: ModuleType DEPTH_LOCAL = 1 DEPTH_BACKING = 2 +CLUSTER = 65536 class TestNbdCommands(iotests.QMPTestCase):
@@ -90,7 +91,7 @@ class TestNbdCommands(iotests.QMPTestCase): self.h.shutdown() self.h = None - def block_status(self, count=size, wanted=None): + def block_status(self, count=size, wanted=None, flags=0): """Map each meta context in the reply to its list of extents.""" reply = {}
@@ -99,9 +100,9 @@ class TestNbdCommands(iotests.QMPTestCase): entries[1::2])) if wanted is None: - self.h.block_status(count, 0, cb) + self.h.block_status(count, 0, cb, flags) else: - self.h.block_status_filter(count, 0, wanted, cb) + self.h.block_status_filter(count, 0, wanted, cb, flags) return reply def top_extents(self):
@@ -158,6 +159,71 @@ class TestNbdCommands(iotests.QMPTestCase): for wanted in (['base:allocation'], ['qemu:allocation-depth']): self.assertEqual(sorted(self.block_status(wanted=wanted)), wanted) + def read_chunks(self, count, offset, flags=0): + chunks = [] + + def cb(subbuf, off, status, _err): + chunks.append((off, len(subbuf), status)) + + self.h.pread_structured(count, offset, cb, flags) + return chunks + + def make_sparse(self): + """Lay out data, a hole and data again, one cluster each.""" + self.h.pwrite(b'z' * CLUSTER, 0) + self.h.zero(CLUSTER, CLUSTER) + self.h.pwrite(b'z' * CLUSTER, 2 * CLUSTER) + + def test_read_sparse_chunks(self): + self.make_sparse() + + first = (0, CLUSTER, nbd.READ_DATA) + hole = (CLUSTER, CLUSTER, nbd.READ_HOLE) + second = (2 * CLUSTER, CLUSTER, nbd.READ_DATA) + + # A hole is an offset and a length, not a cluster of zeroes, + # wherever it falls in the reply + self.assertEqual(self.read_chunks(2 * CLUSTER, 0), [first, hole]) + self.assertEqual(self.read_chunks(2 * CLUSTER, CLUSTER), + [hole, second]) + self.assertEqual(self.read_chunks(3 * CLUSTER, 0), + [first, hole, second]) + + self.assertEqual(self.h.pread(CLUSTER, CLUSTER), bytes(CLUSTER)) + self.assertEqual(self.h.pread(CLUSTER, 0), b'z' * CLUSTER) + + def test_read_dont_fragment(self): + self.make_sparse() + + self.assertEqual(self.read_chunks(3 * CLUSTER, 0, nbd.CMD_FLAG_DF), + [(0, 3 * CLUSTER, nbd.READ_DATA)]) + + def test_block_status_req_one(self): + hole = nbd.STATE_HOLE | nbd.STATE_ZERO + self.make_sparse() + + alloc = self.block_status(3 * CLUSTER)['base:allocation'] + self.assertEqual(alloc, [(CLUSTER, 0), (CLUSTER, hole), (CLUSTER, 0)]) + + # One extent, so the reply covers less than was asked for and the + # client has to come back for the rest + alloc = self.block_status(3 * CLUSTER, flags=nbd.CMD_FLAG_REQ_ONE) + self.assertEqual(alloc['base:allocation'], [(CLUSTER, 0)]) + + def test_write_and_trim_fua(self): + self.h.pwrite(b'y' * 4096, 4096, nbd.CMD_FLAG_FUA) + self.assertEqual(self.h.pread(4096, 4096), b'y' * 4096) + self.h.trim(4096, 4096, nbd.CMD_FLAG_FUA) + + def test_fast_zero(self): + self.h.zero(CLUSTER, CLUSTER, nbd.CMD_FLAG_FAST_ZERO) + self.assertEqual(self.h.pread(CLUSTER, CLUSTER), bytes(CLUSTER)) + + # Zeroing part of a cluster needs the fallback the flag forbids + with self.assertRaises(nbd.Error) as caught: + self.h.zero(4096, 4096, nbd.CMD_FLAG_FAST_ZERO) + self.assertEqual(caught.exception.errno, 'ENOTSUP') + def test_cache_past_end_of_export(self): self.assertRaises(nbd.Error, self.h.cache, size + 1, 0)
tests/qemu-iotests/tests/nbd-commands.out +2 -2
@@ -1,5 +1,5 @@ -....... +............ ---------------------------------------------------------------------- -Ran 7 tests +Ran 12 tests OK
Signature · 1 line
2.53.0
Quoted 28 lines · Denis V. Lunev
NBD_CMD_CACHEcarries no payload in either direction, yet the server rejects any effect length above the advertised maximum payload size. The specification explicitly allows the oversize request, and we already accept it forNBD_CMD_TRIMandNBD_CMD_WRITE_ZEROES. Patch 2 drops the check.The rest of the series adds tests/qemu-iotests/tests/nbd-commands, a small client which speaks the NBD transmission phase itself. Nothing in the tree does that today, so the protocol surface of the server - reply modes, command flags, error codes, sparse and extended replies - is covered only as far as our own client happens to exercise it. It is useful both as coverage and as a regression tool.
Changes since v1:
- the libnbd handle now lives insetUp()andtearDown()instead of a context manager entered by every test case, which drops a level of indent everywhere (Vladimir)
- oneblock_status()helper returning a context to extents map, in place of the three near identical extent collecting callbacks (Vladimir)
- the unused callback arguments are named with a leading underscore, so that iotest 297 stays pylint clean; v1 added eight unused-argument warnings, which would have failed 297 wherever pylint is installed
- Vladimir's Reviewed-by is kept on patch 2, which is unchanged, and dropped from the four test patches, since the two points above rewrote most of the test
Quoted 5 lines
v1:https://lore.kernel.org/qemu-devel/[email protected]/
Signed-off-by Denis V. Lunev<[email protected]>CC Eric Blake<[email protected]>CC Vladimir Sementsov-Ogievskiy<[email protected]>
Thanks,
and applied to my block branch.
Signature · 2 lines
Best regards, Vladimir