Skip to contentSkip to conversation

xen: xenbus_dev_frontend: Fix XS_TRANSACTION_END handling

PATCH NAK Simon Gaiser 3 participants 22 messages Feb 7, 2018 – Mar 2, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 xen: xenbus_dev_frontend: Fix XS_TRANSACTION_END handling

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

Cc <[email protected]> # 4.11
Fixes fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses")
---
 drivers/xen/xenbus/xenbus_dev_frontend.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/xen/xenbus/xenbus_dev_frontend.c +1 -1
index f3b089b7c0b6..d2edbc79384a 100644
@@ -365,7 +365,7 @@ void xenbus_dev_queue_reply(struct xb_req_data *req)
 			if (WARN_ON(rc))
 				goto out;
 		}
-	} else if (req->msg.type == XS_TRANSACTION_END) {
+	} else if (req->type == XS_TRANSACTION_END) {
 		trans = xenbus_get_transaction(u, req->msg.tx_id);
 		if (WARN_ON(!trans))
 			goto out;
Signature · 1 line
2.15.1
#1 lore raw

PATCH 2/2 xen: xenbus: WARN_ON XS_TRANSACTION_{START,END} misuse

As the previous commit shows it's quite easy to confuse the transaction reference counting by ending a transaction twice. So at least try to detect and report it.

---
 drivers/xen/xenbus/xenbus_xs.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/xen/xenbus/xenbus_xs.c +9 -0
index 3e59590c7254..aed954b09b9b 100644
@@ -137,11 +137,20 @@ static uint32_t xs_request_enter(struct xb_req_data *req)
 
 void xs_request_exit(struct xb_req_data *req)
 {
+	unsigned int users_old;
+
 	spin_lock(&xs_state_lock);
+	users_old = xs_state_users;
 	xs_state_users--;
 	if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
 	    req->type == XS_TRANSACTION_END)
 		xs_state_users--;
+	if (WARN_ON(xs_state_users > users_old))
+		/*
+		 * Someone misused XS_TRANSACTION_{START,END}. Reset the
+		 * reference counter so we might survive.
+		 */
+		xs_state_users = 0;
 	spin_unlock(&xs_state_lock);
 
 	if (xs_suspend_active && !xs_state_users)
Signature · 1 line
2.15.1
#2 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START,END} misuse

Quoted 16 lines · Simon Gaiser

As the previous commit shows it's quite easy to confuse the transaction reference counting by ending a transaction twice. So at least try to detect and report it.

---
  drivers/xen/xenbus/xenbus_xs.c | 9 +++++++++
  1 file changed, 9 insertions(+)
drivers/xen/xenbus/xenbus_xs.c +0 -0
index 3e59590c7254..aed954b09b9b 100644
@@ -137,11 +137,20 @@ static uint32_t xs_request_enter(struct xb_req_data *req)
 
  void xs_request_exit(struct xb_req_data *req)

drivers/xen/xenbus/xenbus_xs.c static uint32_t xs_request_enter(struct xb_req_data *req) · line 148

Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +4 -0
  {
+	unsigned int users_old;
+
  	spin_lock(&xs_state_lock);
+	users_old = xs_state_users;
  	xs_state_users--;
  	if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
  	    req->type == XS_TRANSACTION_END)
  		xs_state_users--;
+	if (WARN_ON(xs_state_users > users_old))

WARN_ON_ONCE()?

-boris

Quoted 9 lines
drivers/xen/xenbus/xenbus_xs.c +5 -0
+		/*
+		 * Someone misused XS_TRANSACTION_{START,END}. Reset the
+		 * reference counter so we might survive.
+		 */
+		xs_state_users = 0;
  	spin_unlock(&xs_state_lock);
 
  	if (xs_suspend_active && !xs_state_users)
#3 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START,END} misuse

Boris Ostrovsky:

Quoted 10 lines · Simon Gaiser

drivers/xen/xenbus/xenbus_xs.c static uint32_t xs_request_enter(struct xb_req_data *req) · line 148

Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c +2 -0
+    users_old = xs_state_users;
      xs_state_users--;
      if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
          req->type == XS_TRANSACTION_END)
          xs_state_users--;
+    if (WARN_ON(xs_state_users > users_old))

WARN_ON_ONCE()?

Since we "fix" the wrong decrement by clamping at zero it should not happen immediately again. But if you prefer _ONCE I can change it.

#4 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START, END} misuse

Quoted 10 lines · Simon Gaiser

Boris Ostrovsky:

Quoted 9 lines · Simon Gaiser
Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c +2 -0
+    users_old = xs_state_users;
       xs_state_users--;
       if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
           req->type == XS_TRANSACTION_END)
           xs_state_users--;
+    if (WARN_ON(xs_state_users > users_old))
Quoted 5 lines
Quoted 1 line

WARN_ON_ONCE()?

Since we "fix" the wrong decrement by clamping at zero it should not happen immediately again. But if you prefer _ONCE I can change it.

If this error can happen once then someone at least theoretically can construct a case when it is repeated. So let's switch to _ONCE() variant.

Thanks.
-boris

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START,END} misuse

In reply to Simon Gaiser
Quoted 10 lines · Simon Gaiser

Boris Ostrovsky:

Quoted 9 lines · Simon Gaiser
Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c +2 -0
+    users_old = xs_state_users;
       xs_state_users--;
       if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
           req->type == XS_TRANSACTION_END)
           xs_state_users--;
+    if (WARN_ON(xs_state_users > users_old))
Quoted 5 lines
Quoted 1 line

WARN_ON_ONCE()?

Since we "fix" the wrong decrement by clamping at zero it should not happen immediately again. But if you prefer _ONCE I can change it.

If this error can happen once then someone at least theoretically can construct a case when it is repeated. So let's switch to _ONCE() variant.

Thanks.
-boris

#6 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START, END} misuse

In reply to Boris Ostrovsky

Boris Ostrovsky:

Quoted 10 lines · Simon Gaiser

drivers/xen/xenbus/xenbus_xs.c static uint32_t xs_request_enter(struct xb_req_data *req) · line 148

Quoted 6 lines
drivers/xen/xenbus/xenbus_xs.c +2 -0
+    users_old = xs_state_users;
      xs_state_users--;
      if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
          req->type == XS_TRANSACTION_END)
          xs_state_users--;
+    if (WARN_ON(xs_state_users > users_old))

WARN_ON_ONCE()?

Since we "fix" the wrong decrement by clamping at zero it should not happen immediately again. But if you prefer _ONCE I can change it.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [PATCH 2/2] xen: xenbus: WARN_ON XS_TRANSACTION_{START, END} misuse

In reply to Simon Gaiser
Quoted 16 lines · Simon Gaiser

As the previous commit shows it's quite easy to confuse the transaction reference counting by ending a transaction twice. So at least try to detect and report it.

---
  drivers/xen/xenbus/xenbus_xs.c | 9 +++++++++
  1 file changed, 9 insertions(+)
drivers/xen/xenbus/xenbus_xs.c +0 -0
index 3e59590c7254..aed954b09b9b 100644
@@ -137,11 +137,20 @@ static uint32_t xs_request_enter(struct xb_req_data *req)
 
  void xs_request_exit(struct xb_req_data *req)

drivers/xen/xenbus/xenbus_xs.c static uint32_t xs_request_enter(struct xb_req_data *req) · line 148

Quoted 10 lines
drivers/xen/xenbus/xenbus_xs.c +4 -0
  {
+	unsigned int users_old;
+
  	spin_lock(&xs_state_lock);
+	users_old = xs_state_users;
  	xs_state_users--;
  	if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
  	    req->type == XS_TRANSACTION_END)
  		xs_state_users--;
+	if (WARN_ON(xs_state_users > users_old))

WARN_ON_ONCE()?

-boris

drivers/xen/xenbus/xenbus_xs.c static uint32_t xs_request_enter(struct xb_req_data *req) · line 156

Quoted 9 lines
drivers/xen/xenbus/xenbus_xs.c +5 -0
+		/*
+		 * Someone misused XS_TRANSACTION_{START,END}. Reset the
+		 * reference counter so we might survive.
+		 */
+		xs_state_users = 0;
  	spin_unlock(&xs_state_lock);
 
  	if (xs_suspend_active && !xs_state_users)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

PATCH 2/2 xen: xenbus: WARN_ON XS_TRANSACTION_{START, END} misuse

In reply to Simon Gaiser

As the previous commit shows it's quite easy to confuse the transaction reference counting by ending a transaction twice. So at least try to detect and report it.

---
 drivers/xen/xenbus/xenbus_xs.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/xen/xenbus/xenbus_xs.c +9 -0
index 3e59590c7254..aed954b09b9b 100644
@@ -137,11 +137,20 @@ static uint32_t xs_request_enter(struct xb_req_data *req)
 
 void xs_request_exit(struct xb_req_data *req)
 {
+	unsigned int users_old;
+
 	spin_lock(&xs_state_lock);
+	users_old = xs_state_users;
 	xs_state_users--;
 	if ((req->type == XS_TRANSACTION_START && req->msg.type == XS_ERROR) ||
 	    req->type == XS_TRANSACTION_END)
 		xs_state_users--;
+	if (WARN_ON(xs_state_users > users_old))
+		/*
+		 * Someone misused XS_TRANSACTION_{START,END}. Reset the
+		 * reference counter so we might survive.
+		 */
+		xs_state_users = 0;
 	spin_unlock(&xs_state_lock);
 
 	if (xs_suspend_active && !xs_state_users)
Signature · 7 lines
2.15.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/2
#9 lore raw
In reply to Simon Gaiser
Quoted 19 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

Quoted 3 lines
Cc <[email protected]> # 4.11
Fixes fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses")

(although I'd prefer Juergen to also take a look at this)

Quoted 18 lines
---
  drivers/xen/xenbus/xenbus_dev_frontend.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/xen/xenbus/xenbus_dev_frontend.c +1 -1
index f3b089b7c0b6..d2edbc79384a 100644
@@ -365,7 +365,7 @@ void xenbus_dev_queue_reply(struct xb_req_data *req)
  			if (WARN_ON(rc))
  				goto out;
  		}
-	} else if (req->msg.type == XS_TRANSACTION_END) {
+	} else if (req->type == XS_TRANSACTION_END) {
  		trans = xenbus_get_transaction(u, req->msg.tx_id);
  		if (WARN_ON(!trans))
  			goto out;
Reviewed-by on v1 1/2
#10 lore raw
In reply to Simon Gaiser
Quoted 19 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

Quoted 3 lines
Cc <[email protected]> # 4.11
Fixes fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses")

(although I'd prefer Juergen to also take a look at this)

Quoted 10 lines
---
  drivers/xen/xenbus/xenbus_dev_frontend.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/xen/xenbus/xenbus_dev_frontend.c +0 -0
index f3b089b7c0b6..d2edbc79384a 100644

drivers/xen/xenbus/xenbus_dev_frontend.c void xenbus_dev_queue_reply(struct xb_req_data *req) · line 371

Quoted 8 lines
drivers/xen/xenbus/xenbus_dev_frontend.c +1 -1
@@ -365,7 +365,7 @@ void xenbus_dev_queue_reply(struct xb_req_data *req)
  			if (WARN_ON(rc))
  				goto out;
  		}
-	} else if (req->msg.type == XS_TRANSACTION_END) {
+	} else if (req->type == XS_TRANSACTION_END) {
  		trans = xenbus_get_transaction(u, req->msg.tx_id);
  		if (WARN_ON(!trans))
  			goto out;

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer NAK
#11 lore raw
In reply to Simon Gaiser
Quoted 8 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

The real problem is decrementing the counter when XS_TRANSACTION_END for a non-existing transaction is being sent.

Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

Quoted 4 lines
Cc <[email protected]> # 4.11
Fixes fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses")

So: your patch is a band-aid trying to cure the symptoms, but not the real problem. Please do it properly.

NAK.

Juergen

maintainer
#12 lore raw
Quoted 17 lines · Juergen Gross
Quoted 8 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

The real problem is decrementing the counter when XS_TRANSACTION_END for a non-existing transaction is being sent.

Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Juergen

#13 lore raw

Juergen Gross:

Quoted 5 lines · Simon Gaiser
Quoted 4 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Quoted 8 lines
Quoted 4 lines

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

maintainer
#14 lore raw
Quoted 11 lines · Simon Gaiser

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

Juergen

#15 lore raw

Juergen Gross:

Quoted 64 lines · Simon Gaiser
Quoted 11 lines

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Quoted 7 lines

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

The point of the second patch was to detect such bugs. This would have saved quite some time to find this bug. I added the "fix" of the counter I just because it was trivial after having the if there.

Adding tracking seems to be a quite complex solution for a _potential_ problem.

So I would go with checking ENOENT in xs_request_exit(). Should this be WARN_ON_ONCE()? Since this normally should not happen I would say yes.

Should I keep the reference counter sanity check? And if yes, with the "fix" to the counter?

maintainer
#16 lore raw
Quoted 73 lines · Simon Gaiser

Juergen Gross:

Quoted 64 lines · Simon Gaiser
Quoted 11 lines

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Quoted 7 lines

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

Quoted 6 lines

The point of the second patch was to detect such bugs. This would have saved quite some time to find this bug. I added the "fix" of the counter I just because it was trivial after having the if there.

Adding tracking seems to be a quite complex solution for a _potential_ problem.

I agree.

Quoted 2 lines

So I would go with checking ENOENT in xs_request_exit(). Should this be WARN_ON_ONCE()? Since this normally should not happen I would say yes.

Yes, having a WARN_ON_ONCE here will help.

Quoted 2 lines

Should I keep the reference counter sanity check? And if yes, with the "fix" to the counter?

I'd drop it. This really should not happen and blowing up kernel size with checks of impossible situations isn't the way to go.

In case you really want to do something here you can add something like ASSERT(xs_state_users) before decrementing the counter.

Juergen

maintainer
#17 lore raw
In reply to Simon Gaiser
Quoted 73 lines · Simon Gaiser

Juergen Gross:

Quoted 64 lines · Simon Gaiser
Quoted 11 lines

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Quoted 7 lines

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

Quoted 6 lines

The point of the second patch was to detect such bugs. This would have saved quite some time to find this bug. I added the "fix" of the counter I just because it was trivial after having the if there.

Adding tracking seems to be a quite complex solution for a _potential_ problem.

I agree.

Quoted 2 lines

So I would go with checking ENOENT in xs_request_exit(). Should this be WARN_ON_ONCE()? Since this normally should not happen I would say yes.

Yes, having a WARN_ON_ONCE here will help.

Quoted 2 lines

Should I keep the reference counter sanity check? And if yes, with the "fix" to the counter?

I'd drop it. This really should not happen and blowing up kernel size with checks of impossible situations isn't the way to go.

In case you really want to do something here you can add something like ASSERT(xs_state_users) before decrementing the counter.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw
In reply to Juergen Gross

Juergen Gross:

Quoted 64 lines · Simon Gaiser
Quoted 11 lines

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Quoted 7 lines

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

The point of the second patch was to detect such bugs. This would have saved quite some time to find this bug. I added the "fix" of the counter I just because it was trivial after having the if there.

Adding tracking seems to be a quite complex solution for a _potential_ problem.

So I would go with checking ENOENT in xs_request_exit(). Should this be WARN_ON_ONCE()? Since this normally should not happen I would say yes.

Should I keep the reference counter sanity check? And if yes, with the "fix" to the counter?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer
#19 lore raw
In reply to Simon Gaiser
Quoted 11 lines · Simon Gaiser

Juergen Gross:

Quoted 10 lines · Simon Gaiser
Quoted 8 lines

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Quoted 8 lines
Quoted 3 lines

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

So if xenstore frees the transaction why should we keep it in the list with pending transaction in xenbus_dev_frontend? That's exactly what this patch fixes by always removing it from the list, not only on a successful response (See below for the EINVAL case).

Aah, sorry, I seem to have misread my own coding. :-(

Yes, you are right. Sorry for not seeing it before.

Quoted 29 lines

[...]

Quoted 8 lines
Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

With "considering a transaction closed" I mean "take the code path which removes the transaction from the list with pending transactions".

From the follow-up mail:

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 6 lines

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Quoted 7 lines

Ok, I will handle those cases in xs_request_exit(). Although I don't like that this depends on the internals of xenstore (At least to me it's not obvious why it should only return ENOENT or EINVAL in this case).

In the xenbus_write_transaction() case checking the string before sending the transaction (like the transaction itself is verified) would avoid this problem.

Right. I'd prefer this solution.

Remains the only problem you tried to tackle with your second patch: a kernel driver going crazy and ending transactions it never started (or ending them multiple times). The EINVAL case can't happen here, but ENOENT can. Either ENOENT has to be handled in xs_request_exit() or you need to keep track of the transactions like in the user interface and refuse ending an unknown transaction. Or you trust the kernel users. Trying to fix the usage counter seems to be the wrong approach IMO.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer
#20 lore raw
In reply to Juergen Gross
Quoted 17 lines · Juergen Gross
Quoted 8 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

The real problem is decrementing the counter when XS_TRANSACTION_END for a non-existing transaction is being sent.

Quoted 6 lines
Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Sorry, I stand corrected: the ENOENT case should never happen, as this case is tested in xenbus_write_transaction(). It doesn't hurt to test for ENOENT, though.

What should be handled is EINVAL: this would happen if a user specified a string different from "T" and "F".

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

maintainer NAK
#21 lore raw
In reply to Simon Gaiser
Quoted 8 lines · Simon Gaiser

Commit fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses") made a subtle change to the semantic of xenbus_dev_request_and_reply() and xenbus_transaction_end().

Before on an error response to XS_TRANSACTION_END xenbus_dev_request_and_reply() would not decrement the active transaction counter. But xenbus_transaction_end() has always counted the transaction as finished regardless of the response.

Which is correct now. Xenstore will free all transaction related data regardless of the response. A once failed transaction can't be repaired, it has to be repeated completely.

The real problem is decrementing the counter when XS_TRANSACTION_END for a non-existing transaction is being sent.

Quoted 3 lines

The new behavior is that xenbus_dev_request_and_reply() and xenbus_transaction_end() will always count the transaction as finished regardless the response code (handled in xs_request_exit()).

ENOENT should not decrement the transaction counter, while all other responses to XS_TRANSACTION_END should still do so.

Quoted 5 lines

But xenbus_dev_frontend tries to end a transaction on closing of the device if the XS_TRANSACTION_END failed before. Trying to close the transaction twice corrupts the reference count. So fix this by also considering a transaction closed if we have sent XS_TRANSACTION_END once regardless of the return code.

A transaction in the list of transactions should not considered to be finished. Either it is not on the list or it is still pending.

Quoted 4 lines
Cc <[email protected]> # 4.11
Fixes fd8aa9095a95 ("xen: optimize xenbus driver for multiple concurrent xenstore accesses")

So: your patch is a band-aid trying to cure the symptoms, but not the real problem. Please do it properly.

NAK.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel