xen/xenbus: make xs_talkv() interruptible
Juergen Gross
maintainerPATCH v2 1/1 xen/xenbus: make xs_talkv() interruptible
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
---
V2:
- don't special case SIGKILL as libxenstore is handling -EINTR fine
--- drivers/xen/xenbus/xenbus_xs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/xen/xenbus/xenbus_xs.c +8 -1
@@ -205,8 +205,15 @@ static bool test_reply(struct xb_req_data *req)static void *read_reply (struct xb_req_data *req) { +int ret; + do { - wait_event(req->wq, test_reply(req)); + ret = wait_event_interruptible(req->wq, test_reply(req)); + + if (ret == -ERESTARTSYS && signal_pending(current)) { + req->msg.type = XS_ERROR; + return ERR_PTR(-EINTR); + } if (!xenbus_ok()) /*
Signature · 1 line
2.26.2
Quoted 7 lines · Juergen Gross
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
Signed-off-by Juergen Gross <[email protected]>---
V2:
- don't special case SIGKILL as libxenstore is handling-EINTRfine
Quoted 19 lines · Juergen Gross
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
Signed-off-by Juergen Gross <[email protected]>---
V2:
- don't special case SIGKILL as libxenstore is handling-EINTRfine--- drivers/xen/xenbus/xenbus_xs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/xen/xenbus/xenbus_xs.c +0
-0@@ -205,8 +205,15 @@ static bool test_reply(struct xb_req_data *req)static void *read_reply (struct xb_req_data *req) {
drivers/xen/xenbus/xenbus_xs.c static bool test_reply(struct xb_req_data *req) · line 216
Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +8
-1+int ret; + do { - wait_event(req->wq, test_reply(req)); + ret = wait_event_interruptible(req->wq, test_reply(req)); + + if (ret == -ERESTARTSYS && signal_pending(current)) { + req->msg.type = XS_ERROR; + return ERR_PTR(-EINTR); + }
So now I can talk fully about the situations which lead to this, I think there is a bit more complexity.
It turns out there are a number of issues related to running a Xen system with no xenstored.
1) If a xenstore-write occurs during startup before init-xenstore-domain runs, the former blocks on /dev/xen/xenbus waiting for xenstored to reply, while the latter blocks on /dev/xen/xenbus_backend when trying to tell the dom0 kernel that xenstored is in dom1. This effectively deadlocks the system.
2) If xenstore-watch is running when xenstored dies, it spins at 100% cpu usage making no system calls at all. This is caused by bad error handling from xs_watch(), and attempting to debug found:
3) (this issue). If anyone starts xenstore-watch with no xenstored running at all, it blocks in D in the kernel.
The cause is the special handling for watch/unwatch commands which, instead of just queuing up the data for xenstore, explicitly waits for an OK for registering the watch. This causes a write() system call to block waiting for a non-existent entity to reply.
So while this patch does resolve the major usability issue I found (I can't even SIGINT and get my terminal back), I think there are issues.
The reason why XS_WATCH/XS_UNWATCH are special cased is because they do require special handling. The main kernel thread for processing incoming data from xenstored does need to know how to associate each async XS_WATCH_EVENT to the caller who watched the path.
Therefore, depending on when this cancellation hits, we might be in any of the following states:
1) the watch is queued in the kernel, but not even sent to xenstored yet
2) the watch is queued in the xenstored ring, but not acted upon
3) the watch is queued in the xenstored ring, and the xenstored has seen it but not replied yet
4) the watch has been processed, but the XS_WATCH reply hasn't been received yet
5) the watch has been processed, and the XS_WATCH reply received
State 5 (and a little bit) is the normal success path when xenstored has acted upon the request, and the internal kernel infrastructure is set up appropriately to handle XS_WATCH_EVENTs.
States 1 and 2 can be very common if there is no xenstored (or at least, it hasn't started up yet). In reality, there is either no xenstored, or it is up and running (and for a period of time during system startup, these cases occur in sequence).
As soon as the XS_WATCH event has been written into the xenstored ring, it is not safe to cancel. You've committed to xenstored processing the request (if it is up).
If xenstored is actually up and running, its fine and necessary to block. The request will be processed in due course (timing subject to the client and server load). If xenstored isn't up, blocking isn't ok.
Therefore, I think we need to distinguish "not yet on the ring" from "on the ring", as our distinction as to whether cancelling is safe, and ensure we don't queue anything on the ring before we're sure xenstored has started up.
Does this make sense?
~Andrew
Jürgen Groß
maintainerQuoted 34 lines · Andrew Cooper
Quoted 19 lines · Juergen Gross
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
Signed-off-by Juergen Gross <[email protected]>---
V2:
- don't special case SIGKILL as libxenstore is handling-EINTRfine--- drivers/xen/xenbus/xenbus_xs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/xen/xenbus/xenbus_xs.c +0
-0@@ -205,8 +205,15 @@ static bool test_reply(struct xb_req_data *req)static void *read_reply (struct xb_req_data *req) {drivers/xen/xenbus/xenbus_xs.c static bool test_reply(struct xb_req_data *req) · line 216
Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +8
-1+int ret; + do { - wait_event(req->wq, test_reply(req)); + ret = wait_event_interruptible(req->wq, test_reply(req)); + + if (ret == -ERESTARTSYS && signal_pending(current)) { + req->msg.type = XS_ERROR; + return ERR_PTR(-EINTR); + }So now I can talk fully about the situations which lead to this, I think there is a bit more complexity.
Quoted 8 lines
It turns out there are a number of issues related to running a Xen system with no xenstored.
1) If a xenstore-write occurs during startup before init-xenstore-domain runs, the former blocks on
/dev/xen/xenbuswaiting for xenstored to reply, while the latter blocks on/dev/xen/xenbus_backendwhen trying to tell the dom0 kernel that xenstored is in dom1. This effectively deadlocks the system.
This should be easy to solve: any request to /dev/xen/xenbus should block upfront in case xenstored isn't up yet (could e.g. wait interruptible until xenstored_ready is non-zero).
Quoted 3 lines
2) If xenstore-watch is running when xenstored dies, it spins at 100% cpu usage making no system calls at all. This is caused by bad error handling from
xs_watch(), and attempting to debug found:
Can you expand on "bad error handling from xs_watch()", please?
Quoted 3 lines
3) (this issue). If anyone starts xenstore-watch with no xenstored running at all, it blocks in D in the kernel.
Should be handled with solution for 1).
Quoted 25 lines
The cause is the special handling for watch/unwatch commands which, instead of just queuing up the data for xenstore, explicitly waits for an OK for registering the watch. This causes a
write()system call to block waiting for a non-existent entity to reply.So while this patch does resolve the major usability issue I found (I can't even SIGINT and get my terminal back), I think there are issues.
The reason why XS_WATCH/XS_UNWATCH are special cased is because they do require special handling. The main kernel thread for processing incoming data from xenstored does need to know how to associate each async
XS_WATCH_EVENTto the caller who watched the path.Therefore, depending on when this cancellation hits, we might be in any of the following states:
1) the watch is queued in the kernel, but not even sent to xenstored yet
2) the watch is queued in the xenstored ring, but not acted upon
3) the watch is queued in the xenstored ring, and the xenstored has seen it but not replied yet
4) the watch has been processed, but theXS_WATCHreply hasn't been received yet
5) the watch has been processed, and theXS_WATCHreply received
Quoted 8 lines
State 5 (and a little bit) is the normal success path when xenstored has acted upon the request, and the internal kernel infrastructure is set up appropriately to handle
XS_WATCH_EVENTs.States 1 and 2 can be very common if there is no xenstored (or at least, it hasn't started up yet). In reality, there is either no xenstored, or it is up and running (and for a period of time during system startup, these cases occur in sequence).
Yes. this is the reason we can't just reject a user request if xenstored hasn't been detected yet: it could be just starting.
Quoted 4 lines
As soon as the
XS_WATCHevent has been written into the xenstored ring, it is not safe to cancel. You've committed to xenstored processing the request (if it is up).
I'm not sure this is true. Cancelling it might result in a stale watch in xenstored, but there shouldn't be a problem related to that. In case that watch fires the event will normally be discarded by the kernel as no matching watch is found in the kernel's data. In case a new watch has been setup with the same struct xenbus_watch address (which is used as the token), then this new watch might fire without the node of the new watch having changed, but spurious watch events are defined to be okay (OTOH the path in the event might look strange to the handler).
Quoted 11 lines
If xenstored is actually up and running, its fine and necessary to block. The request will be processed in due course (timing subject to the client and server load). If xenstored isn't up, blocking isn't ok.
Therefore, I think we need to distinguish "not yet on the ring" from "on the ring", as our distinction as to whether cancelling is safe, and ensure we don't queue anything on the ring before we're sure xenstored has started up.
Does this make sense?
Basically, yes.
Juergen
Quoted 44 lines · Jürgen Groß
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Juergen Gross
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
Signed-off-by Juergen Gross <[email protected]>---
V2:
- don't special case SIGKILL as libxenstore is handling-EINTRfine
---
drivers/xen/xenbus/xenbus_xs.c| 9 ++++++++-1 file changed, 8 insertions(+), 1 deletion(-)drivers/xen/xenbus/xenbus_xs.c +0
-0@@ -205,8 +205,15 @@ static bool test_reply(struct xb_req_data *req) static void *read_reply(struct xb_req_data *req) {drivers/xen/xenbus/xenbus_xs.c static bool test_reply(struct xb_req_data *req) · line 215
Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +8
-1+ int ret; + do { - wait_event(req->wq, test_reply(req)); + ret = wait_event_interruptible(req->wq, test_reply(req)); + + if (ret == -ERESTARTSYS && signal_pending(current)) { + req->msg.type = XS_ERROR; + return ERR_PTR(-EINTR); + }So now I can talk fully about the situations which lead to this, I think there is a bit more complexity.
Quoted 8 lines
It turns out there are a number of issues related to running a Xen system with no xenstored.
1) If a xenstore-write occurs during startup before init-xenstore-domain runs, the former blocks on
/dev/xen/xenbuswaiting for xenstored to reply, while the latter blocks on/dev/xen/xenbus_backendwhen trying to tell the dom0 kernel that xenstored is in dom1. This effectively deadlocks the system.
Quoted 3 lines
This should be easy to solve: any request to
/dev/xen/xenbusshould block upfront in case xenstored isn't up yet (could e.g. wait interruptible untilxenstored_readyis non-zero).
I'm not sure that that would fix the problem. The problem is that setting the ring details via /dev/xen/xenbus_backend blocks, which prevents us launching the xenstored stubdomain, which prevents the earlier xenbus write being completed.
So long as /dev/xen/xenbus_backend doesn't block, there's no problem with other /dev/xen/xenbus activity being pending briefly.
Looking at the current logic, I'm not completely convinced. Even finding a filled-in evtchn/gfn doesn't mean that xenstored is actually ready.
There are 3 possible cases.
1) PV guest, and details in start_info
2) HVM guest, and details in HVM_PARAMs
3) No details (expected for dom0). Something in userspace must provide details at a later point.
So the setup phases go from nothing, to having ring details, to finding the ring working.
I think it would be prudent to try reading a key between having details and declaring the xenstored_ready. Any activity, even XS_ERROR, indicates that the other end of the ring is listening.
Quoted 6 lines
Quoted 4 lines
2) If xenstore-watch is running when xenstored dies, it spins at 100% cpu usage making no system calls at all. This is caused by bad error handling from
xs_watch(), and attempting to debug found:Can you expand on "bad error handling from
xs_watch()", please?
do_watch() has
for ( ... ) { // defaults to an infinite loop
vec = xs_read_watch();
if (vec == NULL)
continue;
...
}
My next plan was to experiment with break instead of continue, which I'll get to at some point.
Quoted 36 lines
Quoted 4 lines
3) (this issue). If anyone starts xenstore-watch with no xenstored running at all, it blocks in D in the kernel.
Should be handled with solution for 1).
Quoted 29 lines
The cause is the special handling for watch/unwatch commands which, instead of just queuing up the data for xenstore, explicitly waits for an OK for registering the watch. This causes a
write()system call to block waiting for a non-existent entity to reply.So while this patch does resolve the major usability issue I found (I can't even SIGINT and get my terminal back), I think there are issues.
The reason why XS_WATCH/XS_UNWATCH are special cased is because they do require special handling. The main kernel thread for processing incoming data from xenstored does need to know how to associate each async
XS_WATCH_EVENTto the caller who watched the path.Therefore, depending on when this cancellation hits, we might be in any of the following states:
1) the watch is queued in the kernel, but not even sent to xenstored yet
2) the watch is queued in the xenstored ring, but not acted upon
3) the watch is queued in the xenstored ring, and the xenstored has seen it but not replied yet
4) the watch has been processed, but theXS_WATCHreply hasn't been received yet
5) the watch has been processed, and theXS_WATCHreply receivedState 5 (and a little bit) is the normal success path when xenstored has acted upon the request, and the internal kernel infrastructure is set up appropriately to handle
XS_WATCH_EVENTs.
Quoted 7 lines
Quoted 4 lines
States 1 and 2 can be very common if there is no xenstored (or at least, it hasn't started up yet). In reality, there is either no xenstored, or it is up and running (and for a period of time during system startup, these cases occur in sequence).
Yes. this is the reason we can't just reject a user request if xenstored hasn't been detected yet: it could be just starting.
Right, and I'm not suggesting that we'd want to reject accesses while xenstored is starting up.
Quoted 6 lines
Quoted 5 lines
As soon as the
XS_WATCHevent has been written into the xenstored ring, it is not safe to cancel. You've committed to xenstored processing the request (if it is up).
Quoted 8 lines
I'm not sure this is true. Cancelling it might result in a stale watch in xenstored, but there shouldn't be a problem related to that. In case that watch fires the event will normally be discarded by the kernel as no matching watch is found in the kernel's data. In case a new watch has been setup with the same struct
xenbus_watchaddress (which is used as the token), then this new watch might fire without the node of the new watch having changed, but spurious watch events are defined to be okay (OTOH the path in the event might look strange to the handler).
Watches are a quota'd resource in (at least some) xenstored configurations. Losing track of the registration is a resource leak, even if the kernel can filter and discard the unexpected watch events.
Quoted 4 lines
Quoted 4 lines
If xenstored is actually up and running, its fine and necessary to block. The request will be processed in due course (timing subject to the client and server load). If xenstored isn't up, blocking isn't ok.
Quoted 8 lines
Quoted 6 lines
Therefore, I think we need to distinguish "not yet on the ring" from "on the ring", as our distinction as to whether cancelling is safe, and ensure we don't queue anything on the ring before we're sure xenstored has started up.
Does this make sense?
Basically, yes.
Great. If I get any time, I'll try to look into some fixes along the above lines.
~Andrew
Jürgen Groß
maintainerQuoted 45 lines · Andrew Cooper
Quoted 44 lines · Jürgen Groß
Quoted 42 lines · Andrew Cooper
Quoted 19 lines · Juergen Gross
In case a process waits for any Xenstore action in the xenbus driver it should be interruptible by signals.
Signed-off-by Juergen Gross <[email protected]>---
V2:
- don't special case SIGKILL as libxenstore is handling-EINTRfine--- drivers/xen/xenbus/xenbus_xs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/xen/xenbus/xenbus_xs.c +0
-0@@ -205,8 +205,15 @@ static bool test_reply(struct xb_req_data *req) static void *read_reply(struct xb_req_data *req) {drivers/xen/xenbus/xenbus_xs.c static bool test_reply(struct xb_req_data *req) · line 215
Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +8
-1+ int ret; + do { - wait_event(req->wq, test_reply(req)); + ret = wait_event_interruptible(req->wq, test_reply(req)); + + if (ret == -ERESTARTSYS && signal_pending(current)) { + req->msg.type = XS_ERROR; + return ERR_PTR(-EINTR); + }So now I can talk fully about the situations which lead to this, I think there is a bit more complexity.
It turns out there are a number of issues related to running a Xen system with no xenstored.
1) If a xenstore-write occurs during startup before init-xenstore-domain runs, the former blocks on
/dev/xen/xenbuswaiting for xenstored to reply, while the latter blocks on/dev/xen/xenbus_backendwhen trying to tell the dom0 kernel that xenstored is in dom1. This effectively deadlocks the system.
Quoted 8 lines
Quoted 3 lines
This should be easy to solve: any request to
/dev/xen/xenbusshould block upfront in case xenstored isn't up yet (could e.g. wait interruptible untilxenstored_readyis non-zero).I'm not sure that that would fix the problem. The problem is that setting the ring details via
/dev/xen/xenbus_backendblocks, which prevents us launching the xenstored stubdomain, which prevents the earlier xenbus write being completed.
But _why_ is it blocking? Digging through the code I think it blocks is xs_suspend() due to the normal xenstore request being pending. If that request doesn't reach the state to cause blocking in xs_suspend() all is fine.
Quoted 7 lines
So long as
/dev/xen/xenbus_backenddoesn't block, there's no problem with other/dev/xen/xenbusactivity being pending briefly.Looking at the current logic, I'm not completely convinced. Even finding a filled-in evtchn/gfn doesn't mean that xenstored is actually ready.
No, but the deadlock is not going to happen anymore (famous last words).
Quoted 8 lines
There are 3 possible cases.
1) PV guest, and details in
start_info
2) HVM guest, and details inHVM_PARAMs
3) No details (expected for dom0). Something in userspace must provide details at a later point.
Quoted 6 lines
So the setup phases go from nothing, to having ring details, to finding the ring working.
I think it would be prudent to try reading a key between having details and declaring the xenstored_ready. Any activity, even
XS_ERROR, indicates that the other end of the ring is listening.
Yes. But I really think the xs_suspend() is the problematic case. And this will be called _before_ xenstored_ready is being set.
Quoted 18 lines
Quoted 7 lines
Quoted 5 lines
2) If xenstore-watch is running when xenstored dies, it spins at 100% cpu usage making no system calls at all. This is caused by bad error handling from
xs_watch(), and attempting to debug found:Can you expand on "bad error handling from
xs_watch()", please?
do_watch()hasfor ( ... ) { // defaults to an infinite loop vec =
xs_read_watch();
if (vec == NULL)
continue;
...
}
Quoted 2 lines
My next plan was to experiment with break instead of continue, which I'll get to at some point.
I'd rather put a sleep() in. Otherwise you might break some use cases.
Quoted 63 lines
Quoted 37 lines
Quoted 5 lines
3) (this issue). If anyone starts xenstore-watch with no xenstored running at all, it blocks in D in the kernel.
Should be handled with solution for 1).
Quoted 29 lines
The cause is the special handling for watch/unwatch commands which, instead of just queuing up the data for xenstore, explicitly waits for an OK for registering the watch. This causes a
write()system call to block waiting for a non-existent entity to reply.So while this patch does resolve the major usability issue I found (I can't even SIGINT and get my terminal back), I think there are issues.
The reason why XS_WATCH/XS_UNWATCH are special cased is because they do require special handling. The main kernel thread for processing incoming data from xenstored does need to know how to associate each async
XS_WATCH_EVENTto the caller who watched the path.Therefore, depending on when this cancellation hits, we might be in any of the following states:
1) the watch is queued in the kernel, but not even sent to xenstored yet
2) the watch is queued in the xenstored ring, but not acted upon
3) the watch is queued in the xenstored ring, and the xenstored has seen it but not replied yet
4) the watch has been processed, but theXS_WATCHreply hasn't been received yet
5) the watch has been processed, and theXS_WATCHreply receivedState 5 (and a little bit) is the normal success path when xenstored has acted upon the request, and the internal kernel infrastructure is set up appropriately to handle
XS_WATCH_EVENTs.Quoted 7 lines
Quoted 4 lines
States 1 and 2 can be very common if there is no xenstored (or at least, it hasn't started up yet). In reality, there is either no xenstored, or it is up and running (and for a period of time during system startup, these cases occur in sequence).
Yes. this is the reason we can't just reject a user request if xenstored hasn't been detected yet: it could be just starting.
Right, and I'm not suggesting that we'd want to reject accesses while xenstored is starting up.
Quoted 6 lines
Quoted 5 lines
As soon as the
XS_WATCHevent has been written into the xenstored ring, it is not safe to cancel. You've committed to xenstored processing the request (if it is up).Quoted 8 lines
I'm not sure this is true. Cancelling it might result in a stale watch in xenstored, but there shouldn't be a problem related to that. In case that watch fires the event will normally be discarded by the kernel as no matching watch is found in the kernel's data. In case a new watch has been setup with the same struct
xenbus_watchaddress (which is used as the token), then this new watch might fire without the node of the new watch having changed, but spurious watch events are defined to be okay (OTOH the path in the event might look strange to the handler).
Quoted 3 lines
Watches are a quota'd resource in (at least some) xenstored configurations. Losing track of the registration is a resource leak, even if the kernel can filter and discard the unexpected watch events.
Hmm, true.
The correct way to handle it then would be to mark the request to not just drop it in case a late answer is arriving, but to do an unwatch.
A similar handling would be required for a transaction start.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
If xenstored is actually up and running, its fine and necessary to block. The request will be processed in due course (timing subject to the client and server load). If xenstored isn't up, blocking isn't ok.
Therefore, I think we need to distinguish "not yet on the ring" from "on the ring", as our distinction as to whether cancelling is safe, and ensure we don't queue anything on the ring before we're sure xenstored has started up.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Does this make sense?
Basically, yes.
Great. If I get any time, I'll try to look into some fixes along the above lines.
I won't work on those for the coming 3 weeks, so go ahead. :-)
Juergen