Skip to contentSkip to conversation

x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

PATCH Superseded [email protected] 4 participants 14 messages Mar 6, 2020 – Apr 28, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

v4:
- Re-work the guest array search to make it clearer

v3:
- Style fixups

v2:
- Split the check on action->nr_guests > 0 and make it an ASSERT

---
 xen/arch/x86/irq.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
xen/arch/x86/irq.c +17 -3
index cc2eb8e925..32fcb624dc 100644
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     BUG_ON(!(desc->status & IRQ_GUEST));
 
-    for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
-        continue;
-    BUG_ON(i == action->nr_guests);
+    for ( i = 0; i < action->nr_guests; i++ )
+        if ( action->guest[i] == d )
+            break;
+
+    if ( i == action->nr_guests ) /* No matching entry */
+    {
+        /*
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        ASSERT(action->shareable);
+        return NULL;
+    }
+
+    ASSERT(action->nr_guests > 0);
     memmove(&action->guest[i], &action->guest[i+1],
             (action->nr_guests-i-1) * sizeof(action->guest[0]));
     action->nr_guests--;
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

Quoted 2 lines

v4:
- Re-work the guest array search to make it clearer

I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem. But anyway:

Quoted 16 lines
xen/arch/x86/irq.c +7 -3
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     BUG_ON(!(desc->status & IRQ_GUEST));
 
-    for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
-        continue;
-    BUG_ON(i == action->nr_guests);
+    for ( i = 0; i < action->nr_guests; i++ )
+        if ( action->guest[i] == d )
+            break;
+
+    if ( i == action->nr_guests ) /* No matching entry */
+    {
+        /*

xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699

Quoted 10 lines
xen/arch/x86/irq.c +10 -0
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        ASSERT(action->shareable);
+        return NULL;
+    }
+
+    ASSERT(action->nr_guests > 0);

This seems pointless to have here - v3 had it inside the if(), where it would actually guard against coming here with nr_guests equal to zero. v3 also used if() and BUG() instead of ASSERT() inside this if(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 49 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

Quoted 6 lines
Quoted 2 lines

v4:
- Re-work the guest array search to make it clearer

I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.

Please suggest code if you think it ought to be done differentely. I tried.

Quoted 24 lines

But anyway:

Quoted 22 lines
xen/arch/x86/irq.c +13 -3
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     BUG_ON(!(desc->status & IRQ_GUEST));
 
-    for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
-        continue;
-    BUG_ON(i == action->nr_guests);
+    for ( i = 0; i < action->nr_guests; i++ )
+        if ( action->guest[i] == d )
+            break;
+
+    if ( i == action->nr_guests ) /* No matching entry */
+    {
+        /*
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        ASSERT(action->shareable);
Quoted 8 lines

xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699

Quoted 4 lines
xen/arch/x86/irq.c +4 -0
+        return NULL;
+    }
+
+    ASSERT(action->nr_guests > 0);

This seems pointless to have here - v3 had it inside the if(), where it would actually guard against coming here with nr_guests equal to zero.

Why. The code just after this decrements nr_guests so it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.

Quoted 5 lines

v3 also used if() and BUG() instead of ASSERT() inside this if(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?

Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a BUG_ON() did not seem appropriate.

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 36 lines · Paul Durrant
Quoted 27 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29

Quoted 22 lines · On 06.03.2020 17:02, [email protected] wrote:
xen/arch/x86/irq.c +13 -3
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     BUG_ON(!(desc->status & IRQ_GUEST));
 
-    for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
-        continue;
-    BUG_ON(i == action->nr_guests);
+    for ( i = 0; i < action->nr_guests; i++ )
+        if ( action->guest[i] == d )
+            break;
+
+    if ( i == action->nr_guests ) /* No matching entry */
+    {
+        /*
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        ASSERT(action->shareable);
Quoted 8 lines

xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699

Quoted 4 lines
xen/arch/x86/irq.c +4 -0
+        return NULL;
+    }
+
+    ASSERT(action->nr_guests > 0);

This seems pointless to have here - v3 had it inside the if(), where it would actually guard against coming here with nr_guests equal to zero.

Quoted 3 lines

Why. The code just after this decrements nr_guests so it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.

There's no way to come here when nr_guests == 0. This is because in this case the loop will be exited with i being zero, and hence the earlier if()'s body will be entered.

(And no, >= 0 wouldn't make sense to me - it would mean we might have a count of -1 after the decrement.)

Quoted 10 lines
Quoted 5 lines

v3 also used if() and BUG() instead of ASSERT() inside this if(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?

Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a BUG_ON() did not seem appropriate.

It'll mean we have a non-sharable IRQ in a place where this is not supposed to happen. How can we be sure the system is in a state allowing to safely continue? To me, if shareable / non- shareable is of any concern here, then it ought to be BUG(). If it's not, then the ASSERT() ought to be dropped as well.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 45 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 10 March 2020 11:23
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 36 lines · Paul Durrant
Quoted 35 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29

Quoted 16 lines · On 06.03.2020 17:02, [email protected] wrote:
xen/arch/x86/irq.c +7 -3
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(
 
     BUG_ON(!(desc->status & IRQ_GUEST));
 
-    for ( i = 0; (i < action->nr_guests) && (action->guest[i] != d); i++ )
-        continue;
-    BUG_ON(i == action->nr_guests);
+    for ( i = 0; i < action->nr_guests; i++ )
+        if ( action->guest[i] == d )
+            break;
+
+    if ( i == action->nr_guests ) /* No matching entry */
+    {
+        /*

xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699

Quoted 10 lines
xen/arch/x86/irq.c +10 -0
+         * In case the pirq was shared, unbound for this domain in an earlier
+         * call, but still existed on the domain's pirq_tree, we still reach
+         * here if there are any later unbind calls on the same pirq. Return
+         * if such an unbind happens.
+         */
+        ASSERT(action->shareable);
+        return NULL;
+    }
+
+    ASSERT(action->nr_guests > 0);

This seems pointless to have here - v3 had it inside the if(), where it would actually guard against coming here with nr_guests equal to zero.

Quoted 7 lines
Quoted 3 lines

Why. The code just after this decrements nr_guests so it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.

There's no way to come here when nr_guests == 0. This is because in this case the loop will be exited with i being zero, and hence the earlier if()'s body will be entered.

Ok, yes, that's true.

Quoted 15 lines

(And no, >= 0 wouldn't make sense to me - it would mean we might have a count of -1 after the decrement.)

Quoted 10 lines
Quoted 5 lines

v3 also used if() and BUG() instead of ASSERT() inside this if(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?

Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a BUG_ON() did not seem appropriate.

Quoted 5 lines

It'll mean we have a non-sharable IRQ in a place where this is not supposed to happen. How can we be sure the system is in a state allowing to safely continue? To me, if shareable / non- shareable is of any concern here, then it ought to be BUG(). If it's not, then the ASSERT() ought to be dropped as well.

Ok, I'll convert back to a BUG().

Paul

Quoted 2 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Paul Durrant
Quoted 49 lines · Paul Durrant
Quoted 49 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

v4:
- Re-work the guest array search to make it clearer

I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.

Please suggest code if you think it ought to be done differentely. I tried.

How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.

Jan

x86/pass-through: avoid double IRQ unbind during domain cleanup

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

Avoid recurring invocations of pirq_guest_unbind() by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.

xen/arch/x86/irq.c +1 -1
@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p
     }
 
     if ( radix_tree_delete(&d->pirq_tree, pirq->pirq) != pirq )
-        BUG();
+        BUG_ON(!d->is_dying);
 }
 
 /* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */
xen/drivers/passthrough/pci.c +19 -2
@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct dom
         xfree(digl);
     }
 
-    return pt_pirq_softirq_active(pirq_dpci) ? -ERESTART : 0;
+    radix_tree_delete(&d->pirq_tree, dpci_pirq(pirq_dpci)->pirq);
+
+    if ( !pt_pirq_softirq_active(pirq_dpci) )
+        return 0;
+
+    domain_get_irq_dpci(d)->pending_pirq_dpci = pirq_dpci;
+
+    return -ERESTART;
 }
 
 static int pci_clean_dpci_irqs(struct domain *d)
@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do
     hvm_irq_dpci = domain_get_irq_dpci(d);
     if ( hvm_irq_dpci != NULL )
     {
-        int ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
+        int ret = 0;
+
+        if ( hvm_irq_dpci->pending_pirq_dpci )
+        {
+            if ( pt_pirq_softirq_active(hvm_irq_dpci->pending_pirq_dpci) )
+                 ret = -ERESTART;
+            else
+                 hvm_irq_dpci->pending_pirq_dpci = NULL;
+        }
 
+        if ( !ret )
+            ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
         if ( ret )
         {
             spin_unlock(&d->event_lock);
xen/include/asm-x86/hvm/irq.h +2 -0
@@ -158,6 +158,8 @@ struct hvm_irq_dpci {
     DECLARE_BITMAP(isairq_map, NR_ISAIRQS);
     /* Record of mapped Links */
     uint8_t link_cnt[NR_LINK];
+    /* Clean up: Entry with a softirq invocation pending / in progress. */
+    struct hvm_pirq_dpci *pending_pirq_dpci;
 };
 
 /* Machine IRQ to guest device/intx mapping. */

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw
Quoted 66 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 10 March 2020 14:19
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>;

Roger

Quoted 51 lines
Quoted 50 lines

Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

Quoted 2 lines

v4:
- Re-work the guest array search to make it clearer

I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.

Quoted 5 lines
Quoted 1 line

Please suggest code if you think it ought to be done differentely. I tried.

How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.

It's fairly hard to hit IIRC but we could probably engineer it with a one off ERESTART in the right place.

Quoted 76 lines

Jan

x86/pass-through: avoid double IRQ unbind during domain cleanup

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()

    -> domain_relinquish_resources()
      -> pci_release_devices()
        -> pci_clean_dpci_irq()
          -> pirq_guest_unbind()
            -> __pirq_guest_unbind()

Avoid recurring invocations of pirq_guest_unbind() by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.

xen/arch/x86/irq.c +1 -1
@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p
     }
 
     if ( radix_tree_delete(&d->pirq_tree, pirq->pirq) != pirq )
-        BUG();
+        BUG_ON(!d->is_dying);
 }
 
 /* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */
xen/drivers/passthrough/pci.c +19 -2
@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct dom
         xfree(digl);
     }
 
-    return pt_pirq_softirq_active(pirq_dpci) ? -ERESTART : 0;
+    radix_tree_delete(&d->pirq_tree, dpci_pirq(pirq_dpci)->pirq);
+
+    if ( !pt_pirq_softirq_active(pirq_dpci) )
+        return 0;
+
+    domain_get_irq_dpci(d)->pending_pirq_dpci = pirq_dpci;
+
+    return -ERESTART;
 }
 
 static int pci_clean_dpci_irqs(struct domain *d)
@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do
     hvm_irq_dpci = domain_get_irq_dpci(d);
     if ( hvm_irq_dpci != NULL )
     {
-        int ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
+        int ret = 0;
+
+        if ( hvm_irq_dpci->pending_pirq_dpci )
+        {
+            if ( pt_pirq_softirq_active(hvm_irq_dpci->pending_pirq_dpci) )
+                 ret = -ERESTART;
+            else
+                 hvm_irq_dpci->pending_pirq_dpci = NULL;
+        }
 
+        if ( !ret )
+            ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
         if ( ret )
         {
             spin_unlock(&d->event_lock);

xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 164

Quoted 7 lines
xen/include/asm-x86/hvm/irq.h +2 -0
@@ -158,6 +158,8 @@ struct hvm_irq_dpci {
     DECLARE_BITMAP(isairq_map, NR_ISAIRQS);
     /* Record of mapped Links */
     uint8_t link_cnt[NR_LINK];
+    /* Clean up: Entry with a softirq invocation pending / in progress. */
+    struct hvm_pirq_dpci *pending_pirq_dpci;
 };
 

That looks like it will do the job. I'll see if I can get it tested.

Paul

xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 165

Quoted 2 lines
xen/include/asm-x86/hvm/irq.h +0 -0
 /* Machine IRQ to guest device/intx mapping. */

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
Quoted 1 line · Paul Durrant

That looks like it will do the job. I'll see if I can get it tested.

Any luck with this, yet?

Jan

#9 lore raw
Quoted 8 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 5 lines
Quoted 1 line · Paul Durrant

That looks like it will do the job. I'll see if I can get it tested.

Any luck with this, yet?

I have asked Varad to test it. He hopes to get to it this week.

Paul

#10 lore raw
Quoted 9 lines · Paul Durrant
Quoted 7 lines

From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41

Quoted 1 line · Paul Durrant

That looks like it will do the job. I'll see if I can get it tested.

Any luck with this, yet?

I have asked Varad to test it. He hopes to get to it this week.

Any news here?

Jan

#11 lore raw
Quoted 12 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 23 April 2020 12:09
To: [email protected]; 'Varad Gautam' <[email protected]>
Cc: [email protected]; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 3 lines · Paul Durrant
Quoted 3 lines

From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines · Paul Durrant
Quoted 1 line

That looks like it will do the job. I'll see if I can get it tested.

Any luck with this, yet?

I have asked Varad to test it. He hopes to get to it this week.

Any news here?

Varad tells me he is currently testing and will get back to you soon (hopefully today).

Paul

Tested-by Varad Gautam
#12 lore raw
In reply to Jan Beulich

Hi Jan,

Quoted 55 lines · Jan Beulich
Quoted 54 lines · Paul Durrant
Quoted 54 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs

Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:

From: Varad Gautam <[email protected]>

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:

   domain_kill()
     -> domain_relinquish_resources()
       -> pci_release_devices()
         -> pci_clean_dpci_irq()
           -> pirq_guest_unbind()
             -> __pirq_guest_unbind()

For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.

Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.

[taking over from Varad at v4]

---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>

Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.

Quoted 2 lines

v4:
- Re-work the guest array search to make it clearer

I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.

Quoted 8 lines
Quoted 1 line

Please suggest code if you think it ought to be done differentely. I tried.

How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.

Jan

x86/pass-through: avoid double IRQ unbind during domain cleanup

I have tested that this patch prevents __pirq_guest_unbind on an already-unbound pirq during the continuation call for domain_kill -ERESTART, by using a modified xen that forces an -ERESTART from pirq_guest_unbind to create the continuation. It fixes the underlying issue.

Quoted 74 lines

XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:

   domain_kill()
     -> domain_relinquish_resources()
       -> pci_release_devices()
         -> pci_clean_dpci_irq()
           -> pirq_guest_unbind()
             -> __pirq_guest_unbind()

Avoid recurring invocations of pirq_guest_unbind() by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.

xen/arch/x86/irq.c +1 -1
@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p
      }
 
      if ( radix_tree_delete(&d->pirq_tree, pirq->pirq) != pirq )
-        BUG();
+        BUG_ON(!d->is_dying);
  }
 
  /* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */
xen/drivers/passthrough/pci.c +19 -2
@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct dom
          xfree(digl);
      }
 
-    return pt_pirq_softirq_active(pirq_dpci) ? -ERESTART : 0;
+    radix_tree_delete(&d->pirq_tree, dpci_pirq(pirq_dpci)->pirq);
+
+    if ( !pt_pirq_softirq_active(pirq_dpci) )
+        return 0;
+
+    domain_get_irq_dpci(d)->pending_pirq_dpci = pirq_dpci;
+
+    return -ERESTART;
  }
 
  static int pci_clean_dpci_irqs(struct domain *d)
@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do
      hvm_irq_dpci = domain_get_irq_dpci(d);
      if ( hvm_irq_dpci != NULL )
      {
-        int ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
+        int ret = 0;
+
+        if ( hvm_irq_dpci->pending_pirq_dpci )
+        {
+            if ( pt_pirq_softirq_active(hvm_irq_dpci->pending_pirq_dpci) )
+                 ret = -ERESTART;
+            else
+                 hvm_irq_dpci->pending_pirq_dpci = NULL;
+        }
 
+        if ( !ret )
+            ret = pt_pirq_iterate(d, pci_clean_dpci_irq, NULL);
          if ( ret )
          {
              spin_unlock(&d->event_lock);

xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 165

Quoted 8 lines
xen/include/asm-x86/hvm/irq.h +2 -0
@@ -158,6 +158,8 @@ struct hvm_irq_dpci {
      DECLARE_BITMAP(isairq_map, NR_ISAIRQS);
      /* Record of mapped Links */
      uint8_t link_cnt[NR_LINK];
+    /* Clean up: Entry with a softirq invocation pending / in progress. */
+    struct hvm_pirq_dpci *pending_pirq_dpci;
  };
 
  /* Machine IRQ to guest device/intx mapping. */

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

#13 lore raw
Quoted 12 lines · On 28.04.2020 13:58, [email protected] wrote:
Quoted 9 lines · Jan Beulich
Quoted 1 line · Paul Durrant

Please suggest code if you think it ought to be done differentely. I tried.

How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.

Jan

x86/pass-through: avoid double IRQ unbind during domain cleanup

Quoted 6 lines

I have tested that this patch prevents __pirq_guest_unbind on an already-unbound pirq during the continuation call for domain_kill -ERESTART, by using a modified xen that forces an -ERESTART from pirq_guest_unbind to create the continuation. It fixes the underlying issue.

Thanks much; I'll formally submit the patch then.

Jan