Skip to contentSkip to conversation

x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

PATCH In review Varad Gautam 5 participants 7 messages Jan 29, 2020 – Mar 6, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

CC Jan Beulich <[email protected]>
CC Julien Grall <[email protected]>
CC Roger Pau Monné <[email protected]>
CC Andrew Cooper <[email protected]>

v2: Split the check on action->nr_guests > 0 and make it an ASSERT.
v3: Style fixups.

---
 xen/arch/x86/irq.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
xen/arch/x86/irq.c +14 -1
index 310ac00..4b172eb 100644
@@ -1681,7 +1681,20 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
         continue;
-    BUG_ON(i == action->nr_guests);
+    if ( i == action->nr_guests )
+    {
+        ASSERT(action->nr_guests > 0);
+        /*
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        if ( action->shareable )
+            return NULL;
+        BUG();
+    }
+
     memmove(&action->guest[i], &action->guest[i+1],
             (action->nr_guests-i-1) * sizeof(action->guest[0]));
     action->nr_guests--;
Signature · 18 lines
2.7.4




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

Hi,

Quoted 46 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:

   domain_kill()
     -> domain_relinquish_resources()
       -> pci_release_devices()
         -> pci_clean_dpci_irq()
           -> pirq_guest_unbind()
             -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

CC Jan Beulich <[email protected]>
CC Julien Grall <[email protected]>
CC Roger Pau Monné <[email protected]>
CC Andrew Cooper <[email protected]>

v2: Split the check on action->nr_guests > 0 and make it an ASSERT.
v3: Style fixups.

---
  xen/arch/x86/irq.c | 15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)
xen/arch/x86/irq.c +2 -1
index 310ac00..4b172eb 100644
@@ -1681,7 +1681,20 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
      for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
          continue;
-    BUG_ON(i == action->nr_guests);
+    if ( i == action->nr_guests )
+    {

xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1695

Quoted 10 lines
xen/arch/x86/irq.c +10 -0
+        ASSERT(action->nr_guests > 0);
+        /*
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        if ( action->shareable )
+            return NULL;
+        BUG();

Please see my comment on v2 about the BUG(). I am still unconvinced this is the right error handling here.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw
In reply to Varad Gautam

Hello,

Thanks for the patch! Next time could you please try to reply to the previous questions before sending a new version:

https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00257.html

Quoted 17 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Quoted 4 lines

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

So AFAICT this is because pt_pirq_softirq_active() returns true in pci_clean_dpci_irq() and hence the iteration is stopped and hvm_domain_irq(d)->dpci is not set to NULL.

Would it be possible to clean the already processed IRQs from the domain pirq_tree?

pci_clean_dpci_irq() already seems to free part of this structure, and would be nicer IMO if we didn't leave cleaned up stuff behind on ERESTART.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 30 lines · Roger Pau Monné

Hello,

Thanks for the patch! Next time could you please try to reply to the previous questions before sending a new version:

https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00257.html

Quoted 17 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Quoted 4 lines

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

Quoted 6 lines

So AFAICT this is because pt_pirq_softirq_active() returns true in pci_clean_dpci_irq() and hence the iteration is stopped and hvm_domain_irq(d)->dpci is not set to NULL.

Would it be possible to clean the already processed IRQs from the domain pirq_tree?

This might work, perhaps by way of invoking unmap_domain_pirq() right after pirq_guest_unbind(), as long as hvm_dirq_assist() (as called from dpci_softirq()) can be made skip all actual work it means to do in such a case. Unfortunately the two ->masked fields acted upon are different between __pirq_guest_unbind() and hvm_dirq_assist().

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 38 lines · Jan Beulich
Quoted 30 lines · Roger Pau Monné

Hello,

Thanks for the patch! Next time could you please try to reply to the previous questions before sending a new version:

https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00257.html

Quoted 17 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Quoted 4 lines

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

Quoted 6 lines

So AFAICT this is because pt_pirq_softirq_active() returns true in pci_clean_dpci_irq() and hence the iteration is stopped and hvm_domain_irq(d)->dpci is not set to NULL.

Would it be possible to clean the already processed IRQs from the domain pirq_tree?

Quoted 6 lines

This might work, perhaps by way of invoking unmap_domain_pirq() right after pirq_guest_unbind(), as long as hvm_dirq_assist() (as called from dpci_softirq()) can be made skip all actual work it means to do in such a case. Unfortunately the two ->masked fields acted upon are different between __pirq_guest_unbind() and hvm_dirq_assist().

Ping? Unless I hear back soon, I'm afraid I'm going to drop this patch from my "pending" mail folder, as not being agreed whether to stick to the current version or whether to go this alternative route. A more "natural" approach to fixing the issue would be quite nice, after all.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 54 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Jan Beulich
Sent: 05 March 2020 09:37
To: Gautam, Varad <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [Xen-devel] [PATCH v3] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 38 lines · Jan Beulich
Quoted 30 lines · Roger Pau Monné

Hello,

Thanks for the patch! Next time could you please try to reply to the previous questions before sending a new version:

https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00257.html

Quoted 17 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Quoted 4 lines

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

Quoted 6 lines

So AFAICT this is because pt_pirq_softirq_active() returns true in pci_clean_dpci_irq() and hence the iteration is stopped and hvm_domain_irq(d)->dpci is not set to NULL.

Would it be possible to clean the already processed IRQs from the domain pirq_tree?

Quoted 6 lines

This might work, perhaps by way of invoking unmap_domain_pirq() right after pirq_guest_unbind(), as long as hvm_dirq_assist() (as called from dpci_softirq()) can be made skip all actual work it means to do in such a case. Unfortunately the two ->masked fields acted upon are different between __pirq_guest_unbind() and hvm_dirq_assist().

Quoted 5 lines

Ping? Unless I hear back soon, I'm afraid I'm going to drop this patch from my "pending" mail folder, as not being agreed whether to stick to the current version or whether to go this alternative route. A more "natural" approach to fixing the issue would be quite nice, after all.

I'll try to pick this up tomorrow as I helped diagnose the problem being fixed.

Paul

Quoted 7 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 62 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Durrant, Paul
Sent: 05 March 2020 17:37
To: Jan Beulich <[email protected]>; Gautam, Varad <[email protected]>
Cc: [email protected]; Roger Pau Monné <[email protected]>; Julien Grall <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [Xen-devel] [PATCH v3] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 54 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Jan Beulich
Sent: 05 March 2020 09:37
To: Gautam, Varad <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>
Subject: Re: [Xen-devel] [PATCH v3] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 44 lines · Jan Beulich
Quoted 30 lines · Roger Pau Monné

Hello,

Thanks for the patch! Next time could you please try to reply to the previous questions before sending a new version:

https://lists.xenproject.org/archives/html/xen-devel/2019-07/msg00257.html

Quoted 17 lines · Varad Gautam

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Quoted 4 lines

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyways.

Quoted 6 lines

So AFAICT this is because pt_pirq_softirq_active() returns true in pci_clean_dpci_irq() and hence the iteration is stopped and hvm_domain_irq(d)->dpci is not set to NULL.

Would it be possible to clean the already processed IRQs from the domain pirq_tree?

This might work, perhaps by way of invoking unmap_domain_pirq() right after pirq_guest_unbind(), as long as hvm_dirq_assist() (as called from dpci_softirq()) can be made skip all actual work it means to do in such a case. Unfortunately the two ->masked fields acted upon are different between __pirq_guest_unbind() and hvm_dirq_assist().

Quoted 8 lines
Quoted 5 lines

Ping? Unless I hear back soon, I'm afraid I'm going to drop this patch from my "pending" mail folder, as not being agreed whether to stick to the current version or whether to go this alternative route. A more "natural" approach to fixing the issue would be quite nice, after all.

I'll try to pick this up tomorrow as I helped diagnose the problem being fixed.

I'm looking at this now and I am finding the code very confusing, but I think we could achieve the cleanup by passing back the irq index out of __pirq_guest_unbind() such that pirq_guest_unbind() can call clean_domain_irq_pirq(). I still haven't got much of a clue as to how all the data structures hang together though.

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel