Fix crash due to NBD export leak
cover Fix crash due to NBD export leak
Hi all!
We've faced crash bug, which is reproducing on master branch as well. The case is described in 03, where fix is suggested.
New iotest in 04 crashes without that fix.
Vladimir Sementsov-Ogievskiy (4):
iotests: QemuIoInteractive: use qemu_io_args_no_fmt iotests.py: QemuIoInteractive: print output on failure
nbd: make client_close synchronous
iotests: test shutdown when bitmap is exported through NBD
nbd/server.c | 3 ++ tests/qemu-iotests/299 | 65 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/299.out | 10 ++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 15 +++++--- 5 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 tests/qemu-iotests/299 create mode 100644 tests/qemu-iotests/299.out
Signature · 1 line
2.21.0
PATCH 1/4 iotests: QemuIoInteractive: use qemu_io_args_no_fmt
All users of QemuIoInteractive provides -f argument, so it's incorrect to use qemu_io_args, which contains -f too. Let's use qemu_io_args_no_fmt, which also makes possible to use --image-opts with QemuIoInteractive in the following patch.
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -211,7 +211,7 @@ def get_virtio_scsi_device(): class QemuIoInteractive: def __init__(self, *args): - self.args = qemu_io_args + list(args) + self.args = qemu_io_args_no_fmt + list(args) self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
Signature · 1 line
2.21.0
Re: [PATCH 1/4] iotests: QemuIoInteractive: use qemu_io_args_no_fmt
Quoted 1 line · Vladimir Sementsov-Ogievskiy
All users of QemuIoInteractive provides -f argument, so it's incorrect
So far, all users is just test 205; although your series adds test 209 that does likewise. "incorrect" is a bit harsh since you can specify -f more than once (last one wins); maybe "inefficient" is better wording.
Quoted 3 lines
to use
qemu_io_args, which contains -f too. Let's useqemu_io_args_no_fmt, which also makes possible to use --image-opts with QemuIoInteractive in the following patch.
Quoted 20 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/iotests.py +1
-1@@ -211,7 +211,7 @@ def get_virtio_scsi_device(): class QemuIoInteractive: def __init__(self, *args): - self.args = qemu_io_args + list(args) + self.args = qemu_io_args_no_fmt + list(args) self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 2/4 iotests.py: QemuIoInteractive: print output on failure
In reply to Vladimir Sementsov-OgievskiyMake it simpler to debug when qemu-io failes due to wrong arguments or environment.
--- tests/qemu-iotests/iotests.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +7 -1
@@ -216,7 +216,13 @@ class QemuIoInteractive: stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) - assert self._p.stdout.read(9) == 'qemu-io> ' + out = self._p.stdout.read(9) + if out != 'qemu-io> ': + # Most probably qemu-io just failed to start. + # Let's collect the whole output and exit. + out += self._p.stdout.read() + self._p.wait(timeout=1) + raise ValueError(out) def close(self): self._p.communicate('q\n')
Signature · 1 line
2.21.0
Re: [PATCH 2/4] iotests.py: QemuIoInteractive: print output on failure
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Make it simpler to debug when qemu-io failes due to wrong arguments or
fails
Quoted 14 lines
environment.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/iotests.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)tests/qemu-iotests/iotests.py +0
-0@@ -216,7 +216,13 @@ class QemuIoInteractive: stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
tests/qemu-iotests/iotests.py class QemuIoInteractive: · line 226
Quoted 10 lines
tests/qemu-iotests/iotests.py +7
-1universal_newlines=True) - assert self._p.stdout.read(9) == 'qemu-io> ' + out = self._p.stdout.read(9) + if out != 'qemu-io> ': + # Most probably qemu-io just failed to start. + # Let's collect the whole output and exit. + out += self._p.stdout.read() + self._p.wait(timeout=1) + raise ValueError(out)
Looks sensible.
Quoted 3 lines
tests/qemu-iotests/iotests.py +0
-0def close(self): self._p.communicate('q\n')
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 3/4 nbd: make client_close synchronous
In reply to Vladimir Sementsov-Ogievskiyclient_close doesn't guarantee that client is closed: nbd_trip() keeps reference to it. Let's wait for nbd_trip to finish.
Without this fix, the following crash is possible:
- export bitmap through unternal Qemu NBD server
- connect a client
- shutdown Qemu
On shutdown nbd_export_close_all is called, but it actually don't wait for nbd_trip() to finish and to release its references. So, export is not release, and exported bitmap remains busy, and on try to remove the bitmap (which is part of bdrv_close()) the assertion fairs:
bdrv_release_dirty_bitmap_locked: Assertion `!bdrv_dirty_bitmap_busy(bitmap)' failed
--- nbd/server.c | 3 +++ 1 file changed, 3 insertions(+)
nbd/server.c +3 -0
@@ -1419,6 +1419,8 @@ static void client_close(NBDClient *client, bool negotiated)qio_channel_shutdown (client->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +AIO_WAIT_WHILE (client->exp->ctx, client->recv_coroutine); +/* Also tell the client, so that they release their reference. */ if (client->close_fn) { client->close_fn (client, negotiated);
@@ -2450,6 +2452,7 @@ static coroutine_fn void nbd_trip(void *opaque)trace_nbd_trip ();if (client->closing) { + client->recv_coroutine =NULL ;nbd_client_put (client);return ; }
Signature · 1 line
2.21.0
Re: [PATCH 3/4] nbd: make client_close synchronous
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
client_closedoesn't guarantee that client is closed:nbd_trip()keeps reference to it. Let's wait fornbd_tripto finish.Without this fix, the following crash is possible:
- export bitmap through unternal Qemu NBD server
internal
Quoted 7 lines
- connect a client
- shutdown QemuOn shutdown
nbd_export_close_allis called, but it actually don't wait fornbd_trip()to finish and to release its references. So, export is not release, and exported bitmap remains busy, and on try to remove the bitmap (which is part ofbdrv_close()) the assertion fairs:
fails
Quoted 23 lines
bdrv_release_dirty_bitmap_locked: Assertion `!bdrv_dirty_bitmap_busy(bitmap)' failedSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/server.c | 3 +++ 1 file changed, 3 insertions(+)nbd/server.c +2
-0@@ -1419,6 +1419,8 @@ static void client_close(NBDClient *client, bool negotiated)qio_channel_shutdown (client->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +AIO_WAIT_WHILE (client->exp->ctx, client->recv_coroutine); +/* Also tell the client, so that they release their reference. */ if (client->close_fn) { client->close_fn (client, negotiated);
nbd/server.c static coroutine_fn void nbd_trip(void *opaque) · line 2458
Quoted 7 lines
nbd/server.c +1
-0@@ -2450,6 +2452,7 @@ static coroutine_fn void nbd_trip(void *opaque)trace_nbd_trip ();if (client->closing) { + client->recv_coroutine =NULL ;nbd_client_put (client);return ; }
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 3/4] nbd: make client_close synchronous
In reply to Vladimir Sementsov-OgievskiyQuoted 36 lines · Vladimir Sementsov-Ogievskiy
client_closedoesn't guarantee that client is closed:nbd_trip()keeps reference to it. Let's wait fornbd_tripto finish.Without this fix, the following crash is possible:
- export bitmap through unternal Qemu NBD server
- connect a client
- shutdown QemuOn shutdown
nbd_export_close_allis called, but it actually don't wait fornbd_trip()to finish and to release its references. So, export is not release, and exported bitmap remains busy, and on try to remove the bitmap (which is part ofbdrv_close()) the assertion fairs:
bdrv_release_dirty_bitmap_locked: Assertion `!bdrv_dirty_bitmap_busy(bitmap)' failedSigned-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- nbd/server.c | 3 +++ 1 file changed, 3 insertions(+)nbd/server.c +2
-0@@ -1419,6 +1419,8 @@ static void client_close(NBDClient *client, bool negotiated)qio_channel_shutdown (client->ioc,QIO_CHANNEL_SHUTDOWN_BOTH ,NULL ); +AIO_WAIT_WHILE (client->exp->ctx, client->recv_coroutine); +/* Also tell the client, so that they release their reference. */ if (client->close_fn) { client->close_fn (client, negotiated);
nbd/server.c static coroutine_fn void nbd_trip(void *opaque) · line 2458
Quoted 7 lines
nbd/server.c +1
-0@@ -2450,6 +2452,7 @@ static coroutine_fn void nbd_trip(void *opaque)trace_nbd_trip ();if (client->closing) { + client->recv_coroutine =NULL ;nbd_client_put (client);return ; }
I have a doubt, isn't it possible to hang in AIO_WAIT_WHILE() after this patch?
Do we need aio_wait_kick() in the nbd_trip()? Or may be, something like aio_wait_kick(), but to wake exactly client->exp->ctx aio context?
Also, why in block/io.c we kick the main context, but not bs->aio_context?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 3/4] nbd: make client_close synchronous
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Also, why in
block/io.cwe kick the main context, but notbs->aio_context?
From AIO_WAIT_WHILE():
* The caller's thread must be the IOThread that owns @ctx or the main loop
* thread (with @ctx acquired exactly once). This function cannot be used to
* wait on conditions between two IOThreads since that could lead to deadlock,
* go via the main loop instead.
Case 1: IOThread
while ((cond)) { \
aio_poll(ctx_, true); \
waited_ = true; \
} \In this case aio_poll() returns after every event loop iteration and we will re-evaluate cond. Therefore we don't need to be kicked.
Case 2: Main loop thread
In this case we need the kick since we're waiting on the main loop AioContext, not the IOThread AioContext that is doing the actual work. aio_wait_kick() schedules a dummy BH to wake up the main loop thread.
There is no harm in scheduling the dummy BH in the main loop thread when AIO_WAIT_WHILE() is called from an IOThread.
Hope this helps,
Stefan
Re: [PATCH 3/4] nbd: make client_close synchronous
Quoted 22 lines · Stefan Hajnoczi
Quoted 1 line · Vladimir Sementsov-Ogievskiy
Also, why in
block/io.cwe kick the main context, but notbs->aio_context?From
AIO_WAIT_WHILE():* The caller's thread must be the IOThread that owns @ctx or the main loop
* thread (with @ctx acquired exactly once). This function cannot be used to
* wait on conditions between two IOThreads since that could lead to deadlock,
* go via the main loop instead.Case 1: IOThread
while ((cond)) { \ aio_poll(ctx_, true); \ waited_ = true; \ } \In this case
aio_poll()returns after every event loop iteration and we will re-evaluate cond. Therefore we don't need to be kicked.Case 2: Main loop thread
Quoted 8 lines
In this case we need the kick since we're waiting on the main loop AioContext, not the IOThread AioContext that is doing the actual work.
aio_wait_kick()schedules a dummy BH to wake up the main loop thread.There is no harm in scheduling the dummy BH in the main loop thread when
AIO_WAIT_WHILE()is called from an IOThread.Hope this helps,
Thanks!
Looking at this all again, I think that client->recv_coroutine == NULL is a bad marker of finish, as it's not directly bound to _put. I'll try another approach, to make nbd_export_close_all() be synchronous instead by waiting for all export to be actually freed.
Signature · 2 lines
Best regards, Vladimir
PATCH 4/4 iotests: test shutdown when bitmap is exported through NBD
In reply to Vladimir Sementsov-OgievskiyTest shutdown when bitmap is exported through NBD and active client exists. The previous patch fixes a crash, provoked by this scenario.
--- tests/qemu-iotests/299 | 65 +++++++++++++++++++++++++++++++++++ tests/qemu-iotests/299.out | 10 ++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 5 +-- 4 files changed, 79 insertions(+), 2 deletions(-) create mode 100644 tests/qemu-iotests/299 create mode 100644 tests/qemu-iotests/299.out
tests/qemu-iotests/299 new file +65 -0
@@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +# +# Test shutdown when bitmap is exported through NBD server +# +# Copyright (c) 2020 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import iotests + +# The test is unrelated to formats, restrict it to qcow2 to avoid extra runs +iotests.script_initialize( + supported_fmts=['qcow2'], +) + +nbd_sock = iotests.file_path('nbd.sock', base_dir=iotests.sock_dir) +nbd_uri = 'nbd+unix:///disk?socket=' + nbd_sock +size = 1024 * 1024 + +vm = iotests.VM() +vm.launch() + +vm.qmp_log('blockdev-add', **{ + 'node-name': 'disk', + 'driver': 'null-co', + 'size': 1024 * 1024, +}) + +vm.qmp_log('block-dirty-bitmap-add', **{ + 'node': 'disk', + 'name': 'bitmap0' +}) + +vm.qmp_log('nbd-server-start', **{ + 'addr': { + 'type': 'unix', + 'data': {'path': nbd_sock} + } +}, filters=[iotests.filter_qmp_testfiles]) + +vm.qmp_log('nbd-server-add', **{ + 'device': 'disk', + 'writable': True, + 'bitmap': 'bitmap0' +}) + +p = iotests.QemuIoInteractive('-f', 'raw', nbd_uri) +# wait for connection and check it: +iotests.log(p.cmd('read 0 512').rstrip(), filters=[iotests.filter_qemu_io]) + +vm.shutdown() + +p.close()
tests/qemu-iotests/299.out new file +10 -0
@@ -0,0 +1,10 @@ +{"execute": "blockdev-add", "arguments": {"driver": "null-co", "node-name": "disk", "size": 1048576}} +{"return": {}} +{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap0", "node": "disk"}} +{"return": {}} +{"execute": "nbd-server-start", "arguments": {"addr": {"data": {"path": "SOCK_DIR/PID-nbd.sock"}, "type": "unix"}}} +{"return": {}} +{"execute": "nbd-server-add", "arguments": {"bitmap": "bitmap0", "device": "disk", "writable": true}} +{"return": {}} +read 512/512 bytes at offset 0 +512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/qemu-iotests/group +1 -0
@@ -302,3 +302,4 @@ 291 rw quick 292 rw auto quick 297 meta +299 auto quick
tests/qemu-iotests/iotests.py +3 -2
@@ -345,8 +345,9 @@ def filter_qmp(qmsg, filter_fn): return qmsg def filter_testfiles(msg): - prefix = os.path.join(test_dir, "%s-" % (os.getpid())) - return msg.replace(prefix, 'TEST_DIR/PID-') + pref1 = os.path.join(test_dir, "%s-" % (os.getpid())) + pref2 = os.path.join(sock_dir, "%s-" % (os.getpid())) + return msg.replace(pref1, 'TEST_DIR/PID-').replace(pref2, 'SOCK_DIR/PID-') def filter_qmp_testfiles(qmsg): def _filter(_key, value):
Signature · 1 line
2.21.0
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
Not run: 259
Failures: 140 143
Failed 2 of 120 iotests
make: *** [check-tests/check-block.sh] Error 1
make: *** Waiting for unfinished jobs....
TEST check-qtest-aarch64: tests/qtest/qos-test
Traceback (most recent call last):
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=0c91d8fcbb86459aa0e4fb7fbaad61d0', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-xdxld493/src/docker-src.2020-06-25-11.01.00.15808:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=0c91d8fcbb86459aa0e4fb7fbaad61d0
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-xdxld493/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 14m24.488s
user 0m9.266s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]