Skip to contentSkip to conversation

qemu-nbd: Document benefit of --pid-file

PATCH In review Eric Blake 4 participants 9 messages Oct 7, 2019 – Nov 16, 2019

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

PATCH 1/1 qemu-nbd: Document benefit of --pid-file

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

---
 qemu-nbd.texi | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qemu-nbd.texi +2 -1
index 7f55657722bd..d495bbe8a0ed 100644
@@ -118,7 +118,8 @@ in list mode.
 @item --fork
 Fork off the server process and exit the parent once the server is running.
 @item --pid-file=PATH
-Store the server's process ID in the given file.
+Store the server's process ID in the given file.  The pid file is not
+created until after the server socket is open.
 @item --tls-authz=ID
 Specify the ID of a qauthz object previously created with the
 --object option. This will be used to authorize connecting users
Signature · 1 line
2.21.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 15 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

---
 qemu-nbd.texi | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qemu-nbd.texi +0 -0
index 7f55657722bd..d495bbe8a0ed 100644

qemu-nbd.texi in list mode. · line 125

Quoted 9 lines
qemu-nbd.texi +2 -1
@@ -118,7 +118,8 @@ in list mode.
 @item --fork
 Fork off the server process and exit the parent once the server is running.
 @item --pid-file=PATH
-Store the server's process ID in the given file.
+Store the server's process ID in the given file.  The pid file is not
+created until after the server socket is open.
 @item --tls-authz=ID
 Specify the ID of a qauthz object previously created with the
 --object option. This will be used to authorize connecting users

Well, not wrong, but at least most iotests do this by --fork and seeing when the parent exits. But I suppose:

#2 lore raw
In reply to Eric Blake
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

I'm not seeing how checking the pid file is better than checking the socket directly ? I think it is probably actually worse.

The main problem with the socket is that while we unlink on clean shutdown, it may still exist in disk if the process has exitted abnormally.

With the pidfile though we don't ever unlink it, even on clean shutdown, as we don't use the pid files existance as a mutual exclusion check. We instead acquire fcntl locks on it.

IOW the pidfile could exist already when qemu-nbd starts up and will still exist when it quits.

Quoted 23 lines
---
 qemu-nbd.texi | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
qemu-nbd.texi +2 -1
index 7f55657722bd..d495bbe8a0ed 100644
@@ -118,7 +118,8 @@ in list mode.
 @item --fork
 Fork off the server process and exit the parent once the server is running.
 @item --pid-file=PATH
-Store the server's process ID in the given file.
+Store the server's process ID in the given file.  The pid file is not
+created until after the server socket is open.
 @item --tls-authz=ID
 Specify the ID of a qauthz object previously created with the
 --object option. This will be used to authorize connecting users
Signature · 1 line
2.21.0

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw
Quoted 13 lines · Daniel P. Berrangé
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

I'm not seeing how checking the pid file is better than checking the socket directly ? I think it is probably actually worse.

The main problem with the socket is that while we unlink on clean shutdown, it may still exist in disk if the process has exitted abnormally.

Quoted 6 lines

With the pidfile though we don't ever unlink it, even on clean shutdown, as we don't use the pid files existance as a mutual exclusion check. We instead acquire fcntl locks on it.

IOW the pidfile could exist already when qemu-nbd starts up and will still exist when it quits.

Good point.

I was just a bit confused, because pid file is something unrelated to socket, and why use one thing to check the existence of another, if we can directly try to connect.

Quoted 28 lines
Quoted 24 lines
---
  qemu-nbd.texi | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
qemu-nbd.texi +2 -1
index 7f55657722bd..d495bbe8a0ed 100644
@@ -118,7 +118,8 @@ in list mode.
  @item --fork
  Fork off the server process and exit the parent once the server is running.
  @item --pid-file=PATH
-Store the server's process ID in the given file.
+Store the server's process ID in the given file.  The pid file is not
+created until after the server socket is open.
  @item --tls-authz=ID
  Specify the ID of a qauthz object previously created with the
  --object option. This will be used to authorize connecting users
Signature · 1 line
2.21.0

Regards,
Daniel

Signature · 2 lines
Best regards,
Vladimir
author
#4 lore raw
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 17 lines · Daniel P. Berrangé
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

I'm not seeing how checking the pid file is better than checking the socket directly ? I think it is probably actually worse.

The main problem with the socket is that while we unlink on clean shutdown, it may still exist in disk if the process has exitted abnormally.

With the pidfile though we don't ever unlink it, even on clean shutdown, as we don't use the pid files existance as a mutual exclusion check. We instead acquire fcntl locks on it.

Quoted 8 lines
Quoted 2 lines

IOW the pidfile could exist already when qemu-nbd starts up and will still exist when it quits.

Good point.

I was just a bit confused, because pid file is something unrelated to socket, and why use one thing to check the existence of another, if we can directly try to connect.

Consider the case of writing a testsuite that involves an nbd client, where you want to fire up qemu-nbd as the server. Checking for a pid file in shell is easy, and can be done prior to the point of spawning a client. Checking for a successful connect is harder - the shell is not the point that would actually connect, so checking if connect works involves the shell actually spawning off the child process that attempts the connect. If the client itself has a retry builtin, then you don't need to do anything in shell - just spawn the client with retry (at which point, the client retrying on the connection is smarter than the client retrying on the pid file). But pid files are immensely useful when you have a client that does not have builtin retry, and when writing a testing framework where you use shell to learn whether it is safe to spawn the client: rather than having to modify the client or write a shell loop that respawns child attempts, you merely need a shell loop probing for the pid file to exist.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#5 lore raw
Quoted 28 lines · Eric Blake
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Quoted 17 lines · Daniel P. Berrangé
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

I'm not seeing how checking the pid file is better than checking the socket directly ? I think it is probably actually worse.

The main problem with the socket is that while we unlink on clean shutdown, it may still exist in disk if the process has exitted abnormally.

With the pidfile though we don't ever unlink it, even on clean shutdown, as we don't use the pid files existance as a mutual exclusion check. We instead acquire fcntl locks on it.

Quoted 8 lines
Quoted 2 lines

IOW the pidfile could exist already when qemu-nbd starts up and will still exist when it quits.

Good point.

I was just a bit confused, because pid file is something unrelated to socket, and why use one thing to check the existence of another, if we can directly try to connect.

Quoted 14 lines

Consider the case of writing a testsuite that involves an nbd client, where you want to fire up qemu-nbd as the server. Checking for a pid file in shell is easy, and can be done prior to the point of spawning a client. Checking for a successful connect is harder - the shell is not the point that would actually connect, so checking if connect works involves the shell actually spawning off the child process that attempts the connect. If the client itself has a retry builtin, then you don't need to do anything in shell - just spawn the client with retry (at which point, the client retrying on the connection is smarter than the client retrying on the pid file). But pid files are immensely useful when you have a client that does not have builtin retry, and when writing a testing framework where you use shell to learn whether it is safe to spawn the client: rather than having to modify the client or write a shell loop that respawns child attempts, you merely need a shell loop probing for the pid file to exist.

We shouldn't need todo any of those tricks IIUC. The --fork argument is supposed to only let the parent process exit once the server is running.

IOW, if you run qemu-nbd --fork, in the foreground, then when execution continues the sockets should be present & accepting connections. No need to check for existance of any files or check connecting, etc.

Except that AFAICT, --fork isn't actually implemented with this semantics in qemu-nbd. It looks like we only listen on the sockets after the parent has already exited :-( Can we fix that to synchronize wrt socket listeners ?

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#6 lore raw
Quoted 29 lines · Daniel P. Berrangé
Quoted 28 lines · Eric Blake
Quoted 26 lines · Vladimir Sementsov-Ogievskiy
Quoted 13 lines · Daniel P. Berrangé
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

I'm not seeing how checking the pid file is better than checking the socket directly ? I think it is probably actually worse.

The main problem with the socket is that while we unlink on clean shutdown, it may still exist in disk if the process has exitted abnormally.

Quoted 6 lines

With the pidfile though we don't ever unlink it, even on clean shutdown, as we don't use the pid files existance as a mutual exclusion check. We instead acquire fcntl locks on it.

IOW the pidfile could exist already when qemu-nbd starts up and will still exist when it quits.

Good point.

I was just a bit confused, because pid file is something unrelated to socket, and why use one thing to check the existence of another, if we can directly try to connect.

Quoted 14 lines
Quoted 6 lines

Consider the case of writing a testsuite that involves an nbd client, where you want to fire up qemu-nbd as the server. Checking for a pid file in shell is easy, and can be done prior to the point of spawning a client. Checking for a successful connect is harder - the shell is not the point that would actually connect, so checking if connect works involves the shell actually spawning off the child process that attempts the connect. If the

Quoted 8 lines

client itself has a retry builtin, then you don't need to do anything in shell - just spawn the client with retry (at which point, the client retrying on the connection is smarter than the client retrying on the pid file). But pid files are immensely useful when you have a client that does not have builtin retry, and when writing a testing framework where you use shell to learn whether it is safe to spawn the client: rather than having to modify the client or write a shell loop that respawns child attempts, you merely need a shell loop probing for the pid file to exist.

I've already implemented loop of attempting to connect in my series (patch 4/3). It's a bit more difficult to implement, but it's done. And it's a bit better, as it test exactly what we want to test. Can we proceed with it?

Quoted 16 lines

We shouldn't need todo any of those tricks IIUC. The --fork argument is supposed to only let the parent process exit once the server is running.

IOW, if you run qemu-nbd --fork, in the foreground, then when execution continues the sockets should be present & accepting connections. No need to check for existance of any files or check connecting, etc.

Except that AFAICT, --fork isn't actually implemented with this semantics in qemu-nbd. It looks like we only listen on the sockets after the parent has already exited :-( Can we fix that to synchronize wrt socket listeners ?

Regards,
Daniel

Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw
Quoted 5 lines · Vladimir Sementsov-Ogievskiy

I've already implemented loop of attempting to connect in my series (patch 4/3). It's a bit more difficult to implement, but it's done. And it's a bit better, as it test exactly what we want to test. Can we proceed with it?

For test purposes, yes, that's fine (a test doesn't have to be clean, just work).

Quoted 4 lines
Quoted 4 lines

We shouldn't need todo any of those tricks IIUC. The --fork argument is supposed to only let the parent process exit once the server is running.

Quoted 8 lines
Quoted 8 lines

IOW, if you run qemu-nbd --fork, in the foreground, then when execution continues the sockets should be present & accepting connections. No need to check for existance of any files or check connecting, etc.

Except that AFAICT, --fork isn't actually implemented with this semantics in qemu-nbd. It looks like we only listen on the sockets after the parent has already exited :-( Can we fix that to synchronize wrt socket listeners ?

Yes, sounds like something good to have. I'll take a look at doing that.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#8 lore raw
In reply to Daniel P. Berrangé
Quoted 8 lines · Daniel P. Berrangé
Quoted 7 lines · Eric Blake
Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 5 lines · Daniel P. Berrangé
Quoted 4 lines · Eric Blake

One benefit of --pid-file is that it is easier to probe the file system to see if a pid file has been created than it is to probe if a socket is available for connection. Document that this is an intentional feature.

Revisiting an older thread,

Quoted 22 lines
Quoted 4 lines
Quoted 3 lines

I was just a bit confused, because pid file is something unrelated to socket, and why use one thing to check the existence of another, if we can directly try to connect.

Quoted 14 lines

Consider the case of writing a testsuite that involves an nbd client, where you want to fire up qemu-nbd as the server. Checking for a pid file in shell is easy, and can be done prior to the point of spawning a client. Checking for a successful connect is harder - the shell is not the point that would actually connect, so checking if connect works involves the shell actually spawning off the child process that attempts the connect. If the client itself has a retry builtin, then you don't need to do anything in shell - just spawn the client with retry (at which point, the client retrying on the connection is smarter than the client retrying on the pid file). But pid files are immensely useful when you have a client that does not have builtin retry, and when writing a testing framework where you use shell to learn whether it is safe to spawn the client: rather than having to modify the client or write a shell loop that respawns child attempts, you merely need a shell loop probing for the pid file to exist.

We shouldn't need todo any of those tricks IIUC. The --fork argument is supposed to only let the parent process exit once the server is running.

Quoted 8 lines

IOW, if you run qemu-nbd --fork, in the foreground, then when execution continues the sockets should be present & accepting connections. No need to check for existance of any files or check connecting, etc.

Except that AFAICT, --fork isn't actually implemented with this semantics in qemu-nbd. It looks like we only listen on the sockets after the parent has already exited :-( Can we fix that to synchronize wrt socket listeners ?

Actually, after re-reading the code, it IS implemented this way. The parent process when --fork is used does not exit until after the child process closes its temporary stderr pipe to the parent, which is after the socket has been set up and only right before beginning the main event loop. So when --fork is in use, waiting for the process to exit is sufficient.

But when --fork is not in use (because you want to keep the qemu-nbd process in the foreground for whatever reason), then --pid-file can indeed serve as a witness when the single process has progressed far enough to have the socket open.

At any rate, I'm less certain if it is worth trying to revive this patch. A doc-only change is still okay for 4.2-rc2, except that we have to agree on what such a doc change should be and if it adds anything useful to the reader.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org