iotests: Let 233 run concurrently
cover iotests: Let 233 run concurrently
Currently, 233 cannot reliably run concurrently to other NBD TCP tests. When it starts, it looks for a free port and then attempts to use that for the whole duration of the test run. This is a TOCTTOU race
condition: It does not reserve that port, so another NBD TCP test that runs in parallel can grab it.
To fix this, we must not use the same port all the time, but always choose a new one when qemu-nbd is started. We cannot check whether it is free, but must let qemu-nbd do so and take it atomically. We can achieve this by using the existing --fork option.
There are two problems with --fork, however. First, it does not give us a chance to reliably get the server’s PID, which we need. We can change that by letting qemu-nbd (optionally) write a PID file, though. (Which makes sense if we have a daemon mode.)
Second, it currently discards all output after the server has been started. That looks like an accident to me, because we clearly try to restore the old stderr channel after having redirected all startup messages to the parent process. If it is a bug, we can fix it.
Max Reitz (5):
qemu-nbd: Add --pid-file option
iotests.py: Add qemu_nbd_early_pipe()
qemu-nbd: Do not close stderr
iotests: Use qemu-nbd's --pid-file
iotests: Let 233 run concurrently
qemu-nbd.c | 32 +++++++++++- qemu-nbd.texi | 2 + tests/qemu-iotests/147 | 4 +- tests/qemu-iotests/233 | 1 - tests/qemu-iotests/common.nbd | 93 ++++++++++++++++------------------- tests/qemu-iotests/common.rc | 4 +- tests/qemu-iotests/iotests.py | 9 ++-- 7 files changed, 85 insertions(+), 60 deletions(-)
Signature · 1 line
2.20.1
PATCH 1/5 qemu-nbd: Add --pid-file option
--fork is a bit boring if there is no way to get the child's PID. This option helps.
--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)
qemu-nbd.c +29 -0
@@ -59,6 +59,7 @@#define QEMU_NBD_OPT_IMAGE_OPTS 262#define QEMU_NBD_OPT_FORK 263#define QEMU_NBD_OPT_TLSAUTHZ 264 +#define QEMU_NBD_OPT_PID_FILE 265#define MBR_SIZE 512
@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" #if HAVE_NBD_DEVICE"\n" "Kernel NBD client support:\n"
@@ -651,6 +653,7 @@ int main(int argc, char **argv) { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS }, { "trace", required_argument, NULL, 'T' }, { "fork", no_argument, NULL, QEMU_NBD_OPT_FORK }, + { "pid-file", required_argument, NULL, QEMU_NBD_OPT_PID_FILE }, { NULL, 0, NULL, 0 } };int ch;
@@ -677,6 +680,8 @@ int main(int argc, char **argv)bool list = false;int old_stderr =-1 ;unsigned socket_activation; +const char *pid_path =NULL ; +FILE *pid_file;/* The client thread uses SIGTERM to interrupt the server. A signal * handler ensures that "qemu-nbd -v -c" exits with a nice status code.
@@ -876,6 +881,9 @@ int main(int argc, char **argv)case 'L' : list = true;break ; +case QEMU_NBD_OPT_PID_FILE : + pid_path = optarg; +break ; } }
@@ -1196,6 +1204,27 @@ int main(int argc, char **argv)nbd_update_server_watch (); +if (pid_path) { + pid_file =fopen (pid_path,"w" ); +if (!pid_file) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (errno)); +exit (EXIT_FAILURE ); + } + + ret =fprintf (pid_file,"%ld" , (long )getpid ()); +if (ret <0 ) { + ret = -errno; + } +fclose (pid_file); + +if (ret <0 ) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (-ret)); +exit (EXIT_FAILURE ); + } + } +/* now when the initialization is (almost) complete, chdir("/") * to free any busy filesystems */ if (chdir ("/" ) <0 ) {
qemu-nbd.texi +2 -0
@@ -117,6 +117,8 @@ option; or provide the credentials needed for connecting as a client in list mode. @item --fork Fork off the server process and exit the parent once the server is running. +@item --pid-file=PATH +Store the server's process ID in the given file. @item --tls-authz=ID Specify the ID of a qauthz object previously created with the --object option. This will be used to authorize connecting users
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
Quoted 11 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)
qemu-nbd.c static void usage(const char *name) · line 115
Quoted 4 lines
qemu-nbd.c +1
-0@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n"
Should --pid-file imply --fork, or be an error if --fork was not supplied? As coded, it writes a pid file regardless of --fork, even though it is less obvious that it is useful in that case. I don't have a strong preference (there doesn't seem to be a useful consensus on what forking daemons should do), but it would at least be worth documenting the intended action (even if that implies a tweak to the patch to match the intent).
Quoted 12 lines
qemu-nbd.c +1
-0#if HAVE_NBD_DEVICE "\n" "Kernel NBD client support:\n" @@ -651,6 +653,7 @@ int main(int argc, char **argv) { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS }, { "trace", required_argument, NULL, 'T' }, { "fork", no_argument, NULL, QEMU_NBD_OPT_FORK }, + { "pid-file", required_argument, NULL, QEMU_NBD_OPT_PID_FILE }, { NULL, 0, NULL, 0 } };int ch;
qemu-nbd.c int main(int argc, char **argv) · line 683
Quoted 4 lines
qemu-nbd.c +1
-0@@ -677,6 +680,8 @@ int main(int argc, char **argv)bool list = false;int old_stderr =-1 ;unsigned socket_activation; +const char *pid_path =NULL ;
Bikeshedding: pid_name is nicer (path makes me think of $PATH and other colon-separated lists, which this is not).
Otherwise, I agree that this is long overdue. Thanks! If you can justify the behavior without --fork,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
Quoted 11 lines · Eric Blake
Quoted 9 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)
Quoted 13 lines
qemu-nbd.c static void usage(const char *name) · line 115
Quoted 5 lines
qemu-nbd.c +1
-0@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" Should --pid-file imply --fork, or be an error if --fork was not supplied? As coded, it writes a pid file regardless of --fork, even though it is less obvious that it is useful in that case. I don't have a strong preference (there doesn't seem to be a useful consensus on what forking daemons should do), but it would at least be worth documenting the intended action (even if that implies a tweak to the patch to match the intent).
I think the documentation is pretty clear. It stores the server's PID, whether it has been forked or not.
I don't think we would gain anything from forbidding --pid-file without --fork, would we?
Quoted 11 lines
Quoted 11 lines
qemu-nbd.c +1
-0#if HAVE_NBD_DEVICE "\n" "Kernel NBD client support:\n" @@ -651,6 +653,7 @@ int main(int argc, char **argv) { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS }, { "trace", required_argument, NULL, 'T' }, { "fork", no_argument, NULL, QEMU_NBD_OPT_FORK }, + { "pid-file", required_argument, NULL, QEMU_NBD_OPT_PID_FILE }, { NULL, 0, NULL, 0 } };int ch;
Quoted 8 lines
qemu-nbd.c int main(int argc, char **argv) · line 683
Quoted 5 lines
qemu-nbd.c +1
-0@@ -677,6 +680,8 @@ int main(int argc, char **argv)bool list = false;int old_stderr =-1 ;unsigned socket_activation; +const char *pid_path =NULL ;Bikeshedding:
pid_nameis nicer (path makes me think of $PATH and other colon-separated lists, which this is not).
I'd prefer pid_filename myself, then, because pid_name sounds like a weird way to say "process name". O:-)
Quoted 2 lines
Otherwise, I agree that this is long overdue. Thanks! If you can justify the behavior without --fork,
I just can’t think of a reason not to allow it without --fork. Maybe a user doesn’t need --fork because they just start the server in the background and that’s good enough, but they still want a PID file. So basically like common.rc’s _qemu_nbd_wrapper() before this series.
Max
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
Quoted 26 lines · Max Reitz
Quoted 11 lines · Eric Blake
Quoted 9 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)Quoted 13 lines
qemu-nbd.c static void usage(const char *name) · line 115
Quoted 5 lines
qemu-nbd.c +1
-0@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" Should --pid-file imply --fork, or be an error if --fork was not supplied? As coded, it writes a pid file regardless of --fork, even though it is less obvious that it is useful in that case. I don't have a strong preference (there doesn't seem to be a useful consensus on what forking daemons should do), but it would at least be worth documenting the intended action (even if that implies a tweak to the patch to match the intent).
Quoted 5 lines
I think the documentation is pretty clear. It stores the server's PID, whether it has been forked or not.
I don't think we would gain anything from forbidding --pid-file without --fork, would we?
I can't think of any reason to forbid it. So it sounds like we are intentional, this writes the pid into --pid-file regardless of whether that pid can be learned by other means as well.
Quoted 7 lines
Quoted 4 lines
qemu-nbd.c int main(int argc, char **argv) · line 683
Quoted 1 line
qemu-nbd.c +1
-0+const char *pid_path =NULL ;Bikeshedding:
pid_nameis nicer (path makes me think of $PATH and other colon-separated lists, which this is not).I'd prefer
pid_filenamemyself, then, becausepid_namesounds like a weird way to say "process name". O:-)
Works for me, even if it is longer. Do you want to respin, or just have me touch it up when folding it into my NBD tree?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
Quoted 40 lines · Eric Blake
Quoted 26 lines · Max Reitz
Quoted 11 lines · Eric Blake
Quoted 9 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)Quoted 13 lines
qemu-nbd.c static void usage(const char *name) · line 115
Quoted 5 lines
qemu-nbd.c +1
-0@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" Should --pid-file imply --fork, or be an error if --fork was not supplied? As coded, it writes a pid file regardless of --fork, even though it is less obvious that it is useful in that case. I don't have a strong preference (there doesn't seem to be a useful consensus on what forking daemons should do), but it would at least be worth documenting the intended action (even if that implies a tweak to the patch to match the intent).
Quoted 5 lines
I think the documentation is pretty clear. It stores the server's PID, whether it has been forked or not.
I don't think we would gain anything from forbidding --pid-file without --fork, would we?
I can't think of any reason to forbid it. So it sounds like we are intentional, this writes the pid into --pid-file regardless of whether that pid can be learned by other means as well.
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
qemu-nbd.c +1
-0+const char *pid_path =NULL ;
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Bikeshedding:
pid_nameis nicer (path makes me think of $PATH and other colon-separated lists, which this is not).I'd prefer
pid_filenamemyself, then, becausepid_namesounds like a weird way to say "process name". O:-)Works for me, even if it is longer. Do you want to respin, or just have me touch it up when folding it into my NBD tree?
I suppose I’d prefer a respin, independently of what you make of patches 4 and 5.
Max
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
In reply to Max ReitzQuoted 26 lines · Max Reitz
Quoted 11 lines · Eric Blake
Quoted 9 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)Quoted 13 lines
qemu-nbd.c static void usage(const char *name) · line 115
Quoted 5 lines
qemu-nbd.c +1
-0@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" Should --pid-file imply --fork, or be an error if --fork was not supplied? As coded, it writes a pid file regardless of --fork, even though it is less obvious that it is useful in that case. I don't have a strong preference (there doesn't seem to be a useful consensus on what forking daemons should do), but it would at least be worth documenting the intended action (even if that implies a tweak to the patch to match the intent).
Quoted 5 lines
I think the documentation is pretty clear. It stores the server's PID, whether it has been forked or not.
I don't think we would gain anything from forbidding --pid-file without --fork, would we?
Indeed, use of --pid-file should be independant of --fork, as a mgmt app may have already forked it into the background, and merely want to get the pidfile
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
In reply to Max ReitzQuoted 70 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)qemu-nbd.c +18
-0@@ -59,6 +59,7 @@#define QEMU_NBD_OPT_IMAGE_OPTS 262#define QEMU_NBD_OPT_FORK 263#define QEMU_NBD_OPT_TLSAUTHZ 264 +#define QEMU_NBD_OPT_PID_FILE 265#define MBR_SIZE 512@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" #if HAVE_NBD_DEVICE"\n" "Kernel NBD client support:\n" @@ -651,6 +653,7 @@ int main(int argc, char **argv) { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS }, { "trace", required_argument, NULL, 'T' }, { "fork", no_argument, NULL, QEMU_NBD_OPT_FORK }, + { "pid-file", required_argument, NULL, QEMU_NBD_OPT_PID_FILE }, { NULL, 0, NULL, 0 } };int ch;@@ -677,6 +680,8 @@ int main(int argc, char **argv)bool list = false;int old_stderr =-1 ;unsigned socket_activation; +const char *pid_path =NULL ; +FILE *pid_file;/* The client thread uses SIGTERM to interrupt the server. A signal * handler ensures that "qemu-nbd -v -c" exits with a nice status code. @@ -876,6 +881,9 @@ int main(int argc, char **argv)case 'L' : list = true;break ; +case QEMU_NBD_OPT_PID_FILE : + pid_path = optarg; +break ; } }@@ -1196,6 +1204,27 @@ int main(int argc, char **argv)nbd_update_server_watch (); +if (pid_path) { + pid_file =fopen (pid_path,"w" ); +if (!pid_file) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (errno)); +exit (EXIT_FAILURE ); + } + + ret =fprintf (pid_file,"%ld" , (long )getpid ()); +if (ret <0 ) {
qemu-nbd.c int main(int argc, char **argv) · line 1226
Quoted 10 lines
qemu-nbd.c +10
-0+ ret = -errno; + } +fclose (pid_file); + +if (ret <0 ) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (-ret)); +exit (EXIT_FAILURE ); + } + }
This is racy because multiple qemu-nbd's can be started pointing to the same pidfile and one will blindly overwrite the other.
Please use qemu_write_pidfile instead which acquires locks on the pidfile in a race free manner.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH 1/5] qemu-nbd: Add --pid-file option
Quoted 85 lines · Daniel P. Berrangé
Quoted 70 lines · Max Reitz
--fork is a bit boring if there is no way to get the child's PID. This option helps.
Signed-off-by Max Reitz <[email protected]>--- qemu-nbd.c | 29 +++++++++++++++++++++++++++++ qemu-nbd.texi | 2 ++ 2 files changed, 31 insertions(+)qemu-nbd.c +18
-0@@ -59,6 +59,7 @@#define QEMU_NBD_OPT_IMAGE_OPTS 262#define QEMU_NBD_OPT_FORK 263#define QEMU_NBD_OPT_TLSAUTHZ 264 +#define QEMU_NBD_OPT_PID_FILE 265#define MBR_SIZE 512@@ -111,6 +112,7 @@ static void usage(const char *name)" specify tracing options\n" " --fork fork off the server process and exit the parent\n" " once the server is running\n" +" --pid-file=PATH store the server's process ID in the given file\n" #if HAVE_NBD_DEVICE"\n" "Kernel NBD client support:\n" @@ -651,6 +653,7 @@ int main(int argc, char **argv) { "image-opts", no_argument, NULL, QEMU_NBD_OPT_IMAGE_OPTS }, { "trace", required_argument, NULL, 'T' }, { "fork", no_argument, NULL, QEMU_NBD_OPT_FORK }, + { "pid-file", required_argument, NULL, QEMU_NBD_OPT_PID_FILE }, { NULL, 0, NULL, 0 } };int ch;@@ -677,6 +680,8 @@ int main(int argc, char **argv)bool list = false;int old_stderr =-1 ;unsigned socket_activation; +const char *pid_path =NULL ; +FILE *pid_file;/* The client thread uses SIGTERM to interrupt the server. A signal * handler ensures that "qemu-nbd -v -c" exits with a nice status code. @@ -876,6 +881,9 @@ int main(int argc, char **argv)case 'L' : list = true;break ; +case QEMU_NBD_OPT_PID_FILE : + pid_path = optarg; +break ; } }@@ -1196,6 +1204,27 @@ int main(int argc, char **argv)nbd_update_server_watch (); +if (pid_path) { + pid_file =fopen (pid_path,"w" ); +if (!pid_file) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (errno)); +exit (EXIT_FAILURE ); + } + + ret =fprintf (pid_file,"%ld" , (long )getpid ()); +if (ret <0 ) {qemu-nbd.c int main(int argc, char **argv) · line 1226
Quoted 10 lines
qemu-nbd.c +10
-0+ ret = -errno; + } +fclose (pid_file); + +if (ret <0 ) { +error_report ("Failed to store PID in %s: %s" , + pid_path,strerror (-ret)); +exit (EXIT_FAILURE ); + } + }This is racy because multiple qemu-nbd's can be started pointing to the same pidfile and one will blindly overwrite the other.
Quoted 2 lines
Please use
qemu_write_pidfileinstead which acquires locks on the pidfile in a race free manner.
Ah, nice, that makes things better and easier. :-)
Max
PATCH 2/5 iotests.py: Add qemu_nbd_early_pipe()
In reply to Max Reitzqemu_nbd_pipe() currently unconditionally reads qemu-nbd's output. That is not ideal because qemu-nbd may keep stderr open after the parent process has exited.
Currently, the only user of qemu_nbd_pipe() is 147, which discards the whole output if the parent process returned success and only evaluates it on error. Therefore, we can replace qemu_nbd_pipe() by qemu_nbd_early_pipe() that does the same: Discard the output on success, and return it on error.
--- tests/qemu-iotests/147 | 4 ++-- tests/qemu-iotests/iotests.py | 9 ++++++--- 2 files changed, 8 insertions(+), 5 deletions(-)
tests/qemu-iotests/147 +2 -2
@@ -24,7 +24,7 @@ import socket import stat import time import iotests -from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_pipe +from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_early_pipe NBD_PORT_START = 32768 NBD_PORT_END = NBD_PORT_START + 1024
@@ -93,7 +93,7 @@ class QemuNBD(NBDBlockdevAddBase): pass def _try_server_up(self, *args): - status, msg = qemu_nbd_pipe('-f', imgfmt, test_img, *args) + status, msg = qemu_nbd_early_pipe('-f', imgfmt, test_img, *args) if status == 0: return True if 'Address already in use' in msg:
tests/qemu-iotests/iotests.py +6 -3
@@ -204,9 +204,9 @@ def qemu_nbd(*args): '''Run qemu-nbd in daemon mode and return the parent's exit code''' return subprocess.call(qemu_nbd_args + ['--fork'] + list(args)) -def qemu_nbd_pipe(*args): +def qemu_nbd_early_pipe(*args): '''Run qemu-nbd in daemon mode and return both the parent's exit code - and its output''' + and its output in case of an error''' subp = subprocess.Popen(qemu_nbd_args + ['--fork'] + list(args), stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
@@ -216,7 +216,10 @@ def qemu_nbd_pipe(*args): sys.stderr.write('qemu-nbd received signal %i: %s\n' % (-exitcode, ' '.join(qemu_nbd_args + ['--fork'] + list(args)))) - return exitcode, subp.communicate()[0] + if exitcode == 0: + return exitcode, '' + else: + return exitcode, subp.communicate()[0] def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt): '''Return True if two image files are identical'''
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 2/5] iotests.py: Add qemu_nbd_early_pipe()
Quoted 18 lines · Max Reitz
qemu_nbd_pipe()currently unconditionally reads qemu-nbd's output. That is not ideal because qemu-nbd may keep stderr open after the parent process has exited.Currently, the only user of
qemu_nbd_pipe()is 147, which discards the whole output if the parent process returned success and only evaluates it on error. Therefore, we can replaceqemu_nbd_pipe()byqemu_nbd_early_pipe()that does the same: Discard the output on success, and return it on error.Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/147 | 4 ++-- tests/qemu-iotests/iotests.py | 9 ++++++--- 2 files changed, 8 insertions(+), 5 deletions(-)
tests/qemu-iotests/iotests.py def qemu_nbd_pipe(*args): · line 223
Quoted 9 lines
tests/qemu-iotests/iotests.py +4
-1@@ -216,7 +216,10 @@ def qemu_nbd_pipe(*args): sys.stderr.write('qemu-nbd received signal %i: %s\n' % (-exitcode, ' '.join(qemu_nbd_args + ['--fork'] + list(args)))) - return exitcode, subp.communicate()[0] + if exitcode == 0: + return exitcode, '' + else: + return exitcode, subp.communicate()[0]
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 3/5 qemu-nbd: Do not close stderr
In reply to Max ReitzWe kept old_stderr specifically so we could keep emitting error message on stderr. However, qemu_daemon() closes stderr. Therefore, we need to dup() stderr to old_stderr before invoking qemu_daemon().
---
As I hinted at in the cover letter, I am not sure whether this is truly a bug or whether the current behavior is intentional. So if you disagree with me on this patch, you are welcome to suggest an alternative.
I personally can see two:
(1) Add a --log option for a file to store the server's messages in. Seems a bit cumbersome to me.
(2) Add a --keep-stderr option, which specifically enables this behavior here. Without this option we keep the old behavior.
--- qemu-nbd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
qemu-nbd.c +2 -1
@@ -1015,10 +1015,11 @@ int main(int argc, char **argv)exit (EXIT_FAILURE ); }else if (pid ==0 ) {close (stderr_fd[0 ]); + + old_stderr =dup (STDERR_FILENO ); ret =qemu_daemon (1 ,0 );/* Temporarily redirect stderr to the parent's pipe... */ - old_stderr =dup (STDERR_FILENO );dup2 (stderr_fd[1 ],STDERR_FILENO );if (ret <0 ) {error_report ("Failed to daemonize: %s" ,strerror (errno));
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 3/5] qemu-nbd: Do not close stderr
Quoted 10 lines · Max Reitz
We kept
old_stderrspecifically so we could keep emitting error message on stderr. However,qemu_daemon()closes stderr. Therefore, we need todup()stderr toold_stderrbefore invokingqemu_daemon().Signed-off-by Max Reitz <[email protected]>---
As I hinted at in the cover letter, I am not sure whether this is truly a bug or whether the current behavior is intentional. So if you disagree with me on this patch, you are welcome to suggest an alternative.
Looks to me like a bug fix, for a problem present since commit c1f8fdc3 added old_stderr in 2011.
Quoted 7 lines
I personally can see two:
(1) Add a --log option for a file to store the server's messages in. Seems a bit cumbersome to me.(2) Add a --keep-stderr option, which specifically enables this behavior here. Without this option we keep the old behavior.
The approach done here is simplest, I wouldn't worry about your two alternatives.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH 4/5 iotests: Use qemu-nbd's --pid-file
In reply to Max Reitz--- tests/qemu-iotests/common.rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tests/qemu-iotests/common.rc +2 -2
@@ -106,8 +106,8 @@ _qemu_io_wrapper() _qemu_nbd_wrapper() { ( - echo $BASHPID > "${QEMU_TEST_DIR}/qemu-nbd.pid" - exec "$QEMU_NBD_PROG" $QEMU_NBD_OPTIONS "$@" + exec "$QEMU_NBD_PROG" --pid-file="${QEMU_TEST_DIR}/qemu-nbd.pid" \ + $QEMU_NBD_OPTIONS "$@" ) }
Signature · 1 line
2.20.1
Re: [Qemu-devel] [PATCH 4/5] iotests: Use qemu-nbd's --pid-file
Quoted 10 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/common.rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tests/qemu-iotests/common.rc +0
-0
tests/qemu-iotests/common.rc _qemu_io_wrapper() · line 111
Quoted 8 lines
tests/qemu-iotests/common.rc +2
-2@@ -106,8 +106,8 @@ _qemu_io_wrapper() _qemu_nbd_wrapper() { ( - echo $BASHPID > "${QEMU_TEST_DIR}/qemu-nbd.pid" - exec "$QEMU_NBD_PROG" $QEMU_NBD_OPTIONS "$@" + exec "$QEMU_NBD_PROG" --pid-file="${QEMU_TEST_DIR}/qemu-nbd.pid" \ + $QEMU_NBD_OPTIONS "$@" )
Beforehand, we needed the subshell + exec to guarantee that the pid we were writing was that of the subshell. Now, we don't need either; this could be simplified to:
_qemu_nbd_wrapper()
{
"$QEMU_NBD_PROG" --pid-file... "$@"
}Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH 4/5] iotests: Use qemu-nbd's --pid-file
Quoted 24 lines · Eric Blake
Quoted 10 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/common.rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tests/qemu-iotests/common.rc +0
-0tests/qemu-iotests/common.rc _qemu_io_wrapper() · line 111
Quoted 8 lines
tests/qemu-iotests/common.rc +2
-2@@ -106,8 +106,8 @@ _qemu_io_wrapper() _qemu_nbd_wrapper() { ( - echo $BASHPID > "${QEMU_TEST_DIR}/qemu-nbd.pid" - exec "$QEMU_NBD_PROG" $QEMU_NBD_OPTIONS "$@" + exec "$QEMU_NBD_PROG" --pid-file="${QEMU_TEST_DIR}/qemu-nbd.pid" \ + $QEMU_NBD_OPTIONS "$@" )Beforehand, we needed the subshell + exec to guarantee that the pid we were writing was that of the subshell. Now, we don't need either; this could be simplified to:
Quoted 4 lines
_qemu_nbd_wrapper(){ "$QEMU_NBD_PROG" --pid-file... "$@" }
True, but I just followed _qemu_img_wrapper()’s example. I could change both, of course...
Max
PATCH 5/5 iotests: Let 233 run concurrently
In reply to Max Reitzcommon.nbd's nbd_server_set_tcp_port() tries to find a free port, and then uses it for the whole test run. However, this is racey because even if the port was free at the beginning, there is no guarantee it will continue to be available. Therefore, 233 currently cannot reliably be run concurrently with other NBD TCP tests.
This patch addresses the problem by dropping nbd_server_set_tcp_port(), and instead finding a new port every time nbd_server_start_tcp_socket() is invoked. For this, we run qemu-nbd with --fork and on error evaluate the output to see whether it contains "Address already in use". If so, we try the next port.
On success, we still want to continually redirect the output from qemu-nbd to stderr. To achieve both, we redirect qemu-nbd's stderr to a FIFO that we then open in bash. If the parent process exits with status 0 (which means that the server has started successfully), we launch a background cat process that copies the FIFO to stderr. On failure, we read the whole content into a variable and then evaluate it.
While at it, use --fork in nbd_server_start_unix_socket(), too. Doing so allows us to drop nbd_server_wait_for_*_socket().
Note that the reason common.nbd did not use --fork before is that qemu-nbd did not have --pid-file.
--- tests/qemu-iotests/233 | 1 - tests/qemu-iotests/common.nbd | 93 ++++++++++++++++------------------- 2 files changed, 42 insertions(+), 52 deletions(-)
tests/qemu-iotests/233 +0 -1
@@ -50,7 +50,6 @@ _supported_proto file _supported_os Linux _require_command QEMU_NBD -nbd_server_set_tcp_port tls_x509_init echo
tests/qemu-iotests/common.nbd +42 -51
124 lines not on this page · raw message
Signature · 1 line
2.20.1
Quoted 2 lines · Max Reitz
common.nbd's
nbd_server_set_tcp_port()tries to find a free port, and then uses it for the whole test run. However, this is racey because
racy
Quoted 38 lines
even if the port was free at the beginning, there is no guarantee it will continue to be available. Therefore, 233 currently cannot reliably be run concurrently with other NBD TCP tests.
This patch addresses the problem by dropping
nbd_server_set_tcp_port(), and instead finding a new port every timenbd_server_start_tcp_socket()is invoked. For this, we run qemu-nbd with --fork and on error evaluate the output to see whether it contains "Address already in use". If so, we try the next port.On success, we still want to continually redirect the output from qemu-nbd to stderr. To achieve both, we redirect qemu-nbd's stderr to a FIFO that we then open in bash. If the parent process exits with status 0 (which means that the server has started successfully), we launch a background cat process that copies the FIFO to stderr. On failure, we read the whole content into a variable and then evaluate it.
While at it, use --fork in
nbd_server_start_unix_socket(), too. Doing so allows us to drop nbd_server_wait_for_*_socket().Note that the reason common.nbd did not use --fork before is that qemu-nbd did not have --pid-file.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/233 | 1 - tests/qemu-iotests/common.nbd | 93 ++++++++++++++++------------------- 2 files changed, 42 insertions(+), 52 deletions(-)tests/qemu-iotests/common.nbd +0
-4@@ -34,76 +39,62 @@ nbd_server_stop() fi fi rm -f "$nbd_unix_socket" -} - -nbd_server_wait_for_unix_socket() -{
...
tests/qemu-iotests/common.nbd nbd_server_stop() · line 42
Quoted 3 lines
tests/qemu-iotests/common.nbd +1
-2- echo "Failed in check of unix socket created by qemu-nbd" - exit 1 + rm -f "$nbd_stderr_fifo"
You could use a single 'rm -f "$nbd_unix_socket" "$nbd_stderr_fifo"'. That's cosmetic, though.
Are we sure that even on failure, our fifo will not fill up and cause deadlock? If the failing qemu-nbd has so much output as to be non-atomic so that it blocks waiting for a reader, but we don't read anything until after qemu-nbd exits after forking the daemon, then we have deadlock. But in the common case, I don't think qemu-nbd ever spits out that much in errors, even when it fails to start whether due to a socket in use or for other reasons. And even if it does hang, it is our testsuite (and our CI tools will probably notice it), rather than our main code.
Otherwise, it's a lot of shell code with quite a few bash-isms, but we already require bash, and I didn't spot anything blatantly wrong.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Quoted 49 lines · Eric Blake
Quoted 2 lines · Max Reitz
common.nbd's
nbd_server_set_tcp_port()tries to find a free port, and then uses it for the whole test run. However, this is racey becauseracy
Quoted 38 lines
even if the port was free at the beginning, there is no guarantee it will continue to be available. Therefore, 233 currently cannot reliably be run concurrently with other NBD TCP tests.
This patch addresses the problem by dropping
nbd_server_set_tcp_port(), and instead finding a new port every timenbd_server_start_tcp_socket()is invoked. For this, we run qemu-nbd with --fork and on error evaluate the output to see whether it contains "Address already in use". If so, we try the next port.On success, we still want to continually redirect the output from qemu-nbd to stderr. To achieve both, we redirect qemu-nbd's stderr to a FIFO that we then open in bash. If the parent process exits with status 0 (which means that the server has started successfully), we launch a background cat process that copies the FIFO to stderr. On failure, we read the whole content into a variable and then evaluate it.
While at it, use --fork in
nbd_server_start_unix_socket(), too. Doing so allows us to drop nbd_server_wait_for_*_socket().Note that the reason common.nbd did not use --fork before is that qemu-nbd did not have --pid-file.
Signed-off-by Max Reitz <[email protected]>--- tests/qemu-iotests/233 | 1 - tests/qemu-iotests/common.nbd | 93 ++++++++++++++++------------------- 2 files changed, 42 insertions(+), 52 deletions(-)tests/qemu-iotests/common.nbd +0
-4@@ -34,76 +39,62 @@ nbd_server_stop() fi fi rm -f "$nbd_unix_socket" -} - -nbd_server_wait_for_unix_socket() -{...
tests/qemu-iotests/common.nbd nbd_server_stop() · line 42
Quoted 3 lines
tests/qemu-iotests/common.nbd +1
-2- echo "Failed in check of unix socket created by qemu-nbd" - exit 1 + rm -f "$nbd_stderr_fifo"
Quoted 7 lines
You could use a single 'rm -f "$nbd_unix_socket" "$nbd_stderr_fifo"'. That's cosmetic, though.
Are we sure that even on failure, our fifo will not fill up and cause deadlock? If the failing qemu-nbd has so much output as to be non-atomic so that it blocks waiting for a reader, but we don't read anything until after qemu-nbd exits after forking the daemon, then we have deadlock.
Hm, right. I don’t think it will happen, but if it does, it won’t be because of an “Address already in use”. So if it did happen, the test should fail anyway.
Of course, a hang is not the nicest way to fail a test, but I think as long as we don’t think it will be a problem, it should be fine.
(The alternative I can think of would be to start a background cat that copies data over to a log file, and then kill it after the qemu-nbd parent process has exited. On error, we read the log; on success, we print it to stderr and then start the cat from nbd_stderr_fifo to stderr.)
Quoted 9 lines
But in the common case, I don't think qemu-nbd ever spits out that much in errors, even when it fails to start whether due to a socket in use or for other reasons. And even if it does hang, it is our testsuite (and our CI tools will probably notice it), rather than our main code.
Otherwise, it's a lot of shell code with quite a few bash-isms, but we already require bash, and I didn't spot anything blatantly wrong.
Reviewed-by Eric Blake <[email protected]>
Thanks again!
I’ll prepare the v2.
Max