util: retry open() when it gets interrupted by a signal
PATCH 1/1 util: retry open() when it gets interrupted by a signal
As with many syscalls, open() might be interrupted by a signal.
The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '/dev/drbd1003': Interrupted system call
Retry it until it is not interrupted by a signal.
FYI, dd has the same kind of loop aroud open().
https://github.com/coreutils/coreutils/blob/1ae98dbda7322427e8226356fd110d2553f5fac9/src/dd.c#L1294-L1299
--- util/osdep.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)
util/osdep.c +8 -5
@@ -294,14 +294,17 @@ bool qemu_has_direct_io(void)static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret; +do {#ifdef O_CLOEXEC - ret =open (name, flags |O_CLOEXEC , mode); + ret =open (name, flags |O_CLOEXEC , mode);#else - ret =open (name, flags, mode); -if (ret >=0 ) { -qemu_set_cloexec (ret); - } + ret =open (name, flags, mode); +if (ret >=0 ) { +qemu_set_cloexec (ret); + }#endif + }while (ret ==-1 && errno ==EINTR ); +return ret; }
Signature · 1 line
2.45.2
Quoted 32 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '
/dev/drbd1003': Interrupted system callRetry it until it is not interrupted by a signal.
FYI, dd has the same kind of loop aroudopen().
https://github.com/coreutils/coreutils/blob/1ae98dbda7322427e8226356fd110d2553f5fac9/src/dd.c#L1294-L1299Signed-off-by Philipp Reisner <[email protected]>--- util/osdep.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)util/osdep.c +2
-5@@ -294,14 +294,17 @@ bool qemu_has_direct_io(void)static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret; + do { #ifdef O_CLOEXEC - ret =open (name, flags |O_CLOEXEC , mode); + ret = open(name, flags | O_CLOEXEC, mode); #else - ret =open (name, flags, mode); -if (ret >=0 ) { -qemu_set_cloexec (ret); - }
util/osdep.c bool qemu_has_direct_io(void) · line 310
Quoted 10 lines
util/osdep.c +6
-0+ ret =open (name, flags, mode); +if (ret >=0 ) { +qemu_set_cloexec (ret); + }#endif + }while (ret ==-1 && errno ==EINTR ); +return ret; }
Signature · 3 lines
Cheers, David / dhildenb
Quoted 7 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '
/dev/drbd1003': Interrupted system callRetry it until it is not interrupted by a signal.
As you say, many syscalls can be interruptted by signals, so special casing open() isn't really a solution - its just addressing one specific instance you happened to see.
If there are certain signals that we don't want to have a fatal interruption for, it'd be better to set SA_RESTART with sigaction, which will auto-restart a large set of syscalls, while allowing other signals to be fatal.
Quoted 39 lines
FYI, dd has the same kind of loop aroud
open().
https://github.com/coreutils/coreutils/blob/1ae98dbda7322427e8226356fd110d2553f5fac9/src/dd.c#L1294-L1299Signed-off-by Philipp Reisner <[email protected]>--- util/osdep.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-)util/osdep.c +8
-5@@ -294,14 +294,17 @@ bool qemu_has_direct_io(void)static int qemu_open_cloexec (const char *name,int flags,mode_t mode) {int ret; +do {#ifdef O_CLOEXEC - ret =open (name, flags |O_CLOEXEC , mode); + ret =open (name, flags |O_CLOEXEC , mode);#else - ret =open (name, flags, mode); -if (ret >=0 ) { -qemu_set_cloexec (ret); - } + ret =open (name, flags, mode); +if (ret >=0 ) { +qemu_set_cloexec (ret); + }#endif + }while (ret ==-1 && errno ==EINTR ); +return ret; }Signature · 1 line
2.45.2
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 10 lines · Daniel P. Berrangé
Quoted 7 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '
/dev/drbd1003': Interrupted system callRetry it until it is not interrupted by a signal.
Quoted 8 lines
As you say, many syscalls can be interruptted by signals, so special casing
open()isn't really a solution - its just addressing one specific instance you happened to see.If there are certain signals that we don't want to have a fatal interruption for, it'd be better to set
SA_RESTARTwith sigaction, which will auto-restart a large set of syscalls, while allowing other signals to be fatal.
This is why we have the RETRY_ON_EINTR() macro, right?
Currently we have some places that call qemu_open_old() inside RETRY_ON_EINTR -- we should decide whether we want to handle EINTR inside the qemu_open family of functions, or make the caller deal with it, and put the macro uses in the right place consistently.
I agree that it would be nicer if we could use SA_RESTART, but presumably there's a reason why we don't. (At any rate code that's shared with the user-mode emulation has to be EINTR-resistant, because we can't force the user-mode guest code to avoid registering signal handlers that aren't SA_RESTART.)
thanks
-- PMM
PATCH 1/1 util: retry open() when it gets interrupted by a signal
As with many syscalls, open() might be interrupted by a signal.
The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '/dev/drbd1003': Interrupted system call
Retry it until it is not interrupted by a signal.
FYI, dd has the same kind of loop aroud open().
https://github.com/coreutils/coreutils/blob/1ae98dbda7322427e8226356fd110d2553f5fac9/src/dd.c#L1294-L1299
--- util/osdep.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
util/osdep.c +2 -2
@@ -295,9 +295,9 @@ static int qemu_open_cloexec(const char *name, int flags, mode_t mode) {int ret;#ifdef O_CLOEXEC - ret =open (name, flags |O_CLOEXEC , mode); + ret =RETRY_ON_EINTR (open (name, flags |O_CLOEXEC , mode));#else - ret =open (name, flags, mode); + ret =RETRY_ON_EINTR (open (name, flags, mode));if (ret >=0 ) {qemu_set_cloexec (ret); }
Signature · 1 line
2.45.2
PATCH 1/1 util: use RETRY_ON_EINTR() on open() more consistently
In reply to Peter MaydellAs with many syscalls, open() might be interrupted by a signal.
The call traceimg_open_file()
blk_new_open()
raw_open()
raw_open_common()
qemu_open()
qemu_open_internal()
qemu_open_cloexec()Ended up in calling open() without a retry loop around it.
The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '/dev/drbd1003': Interrupted system call
Add the RETRY_ON_EINTR() in qemu_open_cloexec() and remove it on call-sites using qemu_open_old().
--- chardev/char-fd.c | 2 +- chardev/char-pipe.c | 4 ++-- os-posix.c | 2 +- util/osdep.c | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)
chardev/char-fd.c +1 -1
@@ -198,7 +198,7 @@ int qmp_chardev_open_file_source(char *src, int flags, Error **errp) {int fd =-1 ; - fd =RETRY_ON_EINTR (qemu_open_old (src, flags,0666 )); + fd =qemu_open_old (src, flags,0666 );if (fd ==-1 ) {error_setg_file_open (errp, errno, src); }
chardev/char-pipe.c +2 -2
@@ -131,8 +131,8 @@ static void qemu_chr_open_pipe(Chardev *chr, filename_in =g_strdup_printf ("%s.in" , filename); filename_out =g_strdup_printf ("%s.out" , filename); - fd_in =RETRY_ON_EINTR (qemu_open_old (filename_in,O_RDWR |O_BINARY )); - fd_out =RETRY_ON_EINTR (qemu_open_old (filename_out,O_RDWR |O_BINARY )); + fd_in =qemu_open_old (filename_in,O_RDWR |O_BINARY ); + fd_out =qemu_open_old (filename_out,O_RDWR |O_BINARY );g_free (filename_in);g_free (filename_out);if (fd_in <0 || fd_out <0 ) {
os-posix.c +1 -1
@@ -291,7 +291,7 @@ void os_setup_post(void)error_report ("not able to chdir to /: %s" ,strerror (errno));exit (1 ); } - fd =RETRY_ON_EINTR (qemu_open_old ("/dev/null" ,O_RDWR )); + fd =qemu_open_old ("/dev/null" ,O_RDWR );if (fd ==-1 ) {exit (1 ); }
util/osdep.c +2 -2
@@ -295,9 +295,9 @@ static int qemu_open_cloexec(const char *name, int flags, mode_t mode) {int ret;#ifdef O_CLOEXEC - ret =open (name, flags |O_CLOEXEC , mode); + ret =RETRY_ON_EINTR (open (name, flags |O_CLOEXEC , mode));#else - ret =open (name, flags, mode); + ret =RETRY_ON_EINTR (open (name, flags, mode));if (ret >=0 ) {qemu_set_cloexec (ret); }
Signature · 1 line
2.45.2
Re: [PATCH v3] util: use RETRY_ON_EINTR() on open() more consistently
Hi Philipp,
Quoted 19 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The call trace
img_open_file()
blk_new_open()raw_open() raw_open_common() qemu_open() qemu_open_internal() qemu_open_cloexec()Ended up in calling
open()without a retry loop around it.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '/dev/drbd1003': Interrupted system callAdd the
RETRY_ON_EINTR()inqemu_open_cloexec()and remove it on call-sites usingqemu_open_old().
Quoted 7 lines
Signed-off-by Philipp Reisner <[email protected]>--- chardev/char-fd.c | 2 +- chardev/char-pipe.c | 4 ++-- os-posix.c | 2 +- util/osdep.c | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)
If you need to repost, please set the next version in your patch (which will be v4). Also avoid replying to previous version / discussions and your patch could be missed, see
https://www.qemu.org/docs/master/devel/submitting-a-patch.html#when-resending-patches-add-a-version-tag:
Send each new revision as a new top-level thread, rather than burying it in-reply-to an earlier revision, as many reviewers are not looking inside deep threads for new patches.
Also please Cc [email protected] since the block layer is involved via the blk_new_open() call.
Regards,
Phil.
Re: [PATCH v3] util: use RETRY_ON_EINTR() on open() more consistently
Hi Philippe and everyone else on the thread,
In the meantime, I learned that we can cause the kernel to restart the open() syscall by returning ERESTARTSYS from the kernel-level block device driver side. It was our mistake from the DRBD side that we returned EINTR instead.
The proposed patch to qemu is okay but not necessary. I will no further pursuit the inclusion of this patch.
Sorry for the noise.
Best regards,
Philipp
Quoted 46 lines · Philippe Mathieu-Daudé
Hi Philipp,
Quoted 19 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The call trace
img_open_file()
blk_new_open()raw_open() raw_open_common() qemu_open() qemu_open_internal() qemu_open_cloexec()Ended up in calling
open()without a retry loop around it.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '/dev/drbd1003': Interrupted system callAdd the
RETRY_ON_EINTR()inqemu_open_cloexec()and remove it on call-sites usingqemu_open_old().Quoted 7 lines
Signed-off-by Philipp Reisner <[email protected]>--- chardev/char-fd.c | 2 +- chardev/char-pipe.c | 4 ++-- os-posix.c | 2 +- util/osdep.c | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-)If you need to repost, please set the next version in your patch (which will be v4). Also avoid replying to previous version / discussions and your patch could be missed, see
https://www.qemu.org/docs/master/devel/submitting-a-patch.html#when-resending-patches-add-a-version-tag:Send each new revision as a new top-level thread, rather than burying it in-reply-to an earlier revision, as many reviewers are not looking inside deep threads for new patches.
Also please Cc [email protected] since the block layer is involved via the
blk_new_open()call.Regards,
Phil.
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Daniel P. Berrangé
Quoted 5 lines · Philipp Reisner
As with many syscalls,
open()might be interrupted by a signal.The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '
/dev/drbd1003': Interrupted system call
What is the actual signal you are seeing that impacts QEMU in this way ?
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Retry it until it is not interrupted by a signal.
As you say, many syscalls can be interruptted by signals, so special casing
open()isn't really a solution - its just addressing one specific instance you happened to see.If there are certain signals that we don't want to have a fatal interruption for, it'd be better to set
SA_RESTARTwith sigaction, which will auto-restart a large set of syscalls, while allowing other signals to be fatal.
Quoted 7 lines
This is why we have the
RETRY_ON_EINTR()macro, right?Currently we have some places that call
qemu_open_old()insideRETRY_ON_EINTR-- we should decide whether we want to handle EINTR inside theqemu_openfamily of functions, or make the caller deal with it, and put the macro uses in the right place consistently.
It is incredibly arbitrary where we use RETRY_ON_EINTR, which I think points towards it being a sub-optimal solution to the general problem.
Quoted 7 lines
I agree that it would be nicer if we could use
SA_RESTART, but presumably there's a reason why we don't. (At any rate code that's shared with the user-mode emulation has to be EINTR-resistant, because we can't force the user-mode guest code to avoid registering signal handlers that aren'tSA_RESTART.)
For user mode emulation isn't it valid to just propagage the EINTR back up to the application, since EINTR is a valid errno they have to be willing to handle unless the app has itself use SA_RESTART.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 4 lines · Daniel P. Berrangé
Quoted 2 lines · Peter Maydell
This is why we have the
RETRY_ON_EINTR()macro, right?
Quoted 8 lines
Quoted 5 lines
Currently we have some places that call
qemu_open_old()insideRETRY_ON_EINTR-- we should decide whether we want to handle EINTR inside theqemu_openfamily of functions, or make the caller deal with it, and put the macro uses in the right place consistently.It is incredibly arbitrary where we use
RETRY_ON_EINTR, which I think points towards it being a sub-optimal solution to the general problem.
Agreed (and agreed that SA_RESTART is the usual approach to avoid this mess). Partly I just vaguely recall discussions about this back when we added/improved the RETRY_ON_EINTR macro in the first place: maybe there's a reason we have it still...
Quoted 11 lines
Quoted 6 lines
I agree that it would be nicer if we could use
SA_RESTART, but presumably there's a reason why we don't. (At any rate code that's shared with the user-mode emulation has to be EINTR-resistant, because we can't force the user-mode guest code to avoid registering signal handlers that aren'tSA_RESTART.)For user mode emulation isn't it valid to just propagage the EINTR back up to the application, since EINTR is a valid errno they have to be willing to handle unless the app has itself use
SA_RESTART.
Yes, that's what we must do for cases where we are doing some syscall on behalf of the guest. But for cases where we're doing a syscall because of something QEMU itself needs to do, we may need to retry, because we might not be in a position to be able to back out of what we're doing (or we might not even be inside the "handle a guest syscall" codepath at all).
-- PMM
Quoted 32 lines · Peter Maydell
Quoted 4 lines · Daniel P. Berrangé
Quoted 2 lines · Peter Maydell
This is why we have the
RETRY_ON_EINTR()macro, right?Quoted 8 lines
Quoted 5 lines
Currently we have some places that call
qemu_open_old()insideRETRY_ON_EINTR-- we should decide whether we want to handle EINTR inside theqemu_openfamily of functions, or make the caller deal with it, and put the macro uses in the right place consistently.It is incredibly arbitrary where we use
RETRY_ON_EINTR, which I think points towards it being a sub-optimal solution to the general problem.Agreed (and agreed that
SA_RESTARTis the usual approach to avoid this mess). Partly I just vaguely recall discussions about this back when we added/improved theRETRY_ON_EINTRmacro in the first place: maybe there's a reason we have it still...Quoted 11 lines
Quoted 6 lines
I agree that it would be nicer if we could use
SA_RESTART, but presumably there's a reason why we don't. (At any rate code that's shared with the user-mode emulation has to be EINTR-resistant, because we can't force the user-mode guest code to avoid registering signal handlers that aren'tSA_RESTART.)For user mode emulation isn't it valid to just propagage the EINTR back up to the application, since EINTR is a valid errno they have to be willing to handle unless the app has itself use
SA_RESTART.
Quoted 6 lines
Yes, that's what we must do for cases where we are doing some syscall on behalf of the guest. But for cases where we're doing a syscall because of something QEMU itself needs to do, we may need to retry, because we might not be in a position to be able to back out of what we're doing (or we might not even be inside the "handle a guest syscall" codepath at all).
Ah ok, so RETRY_ON_EINTR conceivably makes sense in the linux-user / bsd-user code in certain scenarios......but it seems almost every single use today is in system emulator code !
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi Daniel,
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The experienced logfile entry is:
qemu-system-x86_64: -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=libvirt-2-format,id=virtio-disk0,bootindex=2,write-cache=on,serial=1b990c4d13b74a4e90ea: Could not open '
/dev/drbd1003': Interrupted system callWhat is the actual signal you are seeing that impacts QEMU in this way ?
I do not know at this point. This only reproduces on a customer's system we do not have access to. We do not see it in our in-house lab. And qemu is called through libvirt through ApacheCloudStack. And it affects only about 10%-20% of the VM start operations.
I will wrap my head around bpftrace and see if I can instruct the customer to run that on his systems. So, maybe I can answer the question regarding the signal in a few days. Maybe next week.
The backing device we use (drbd) does an "auto promote" action in its open implementation. That involves exchanging some packets with some peers on the local network. I guess that takes between 1ms to 10ms. So, it exposes a larger time window than other backing block devices, which probably have a shorter running open implementation.
So this is why we see it sometimes.
with regards,
Philipp