x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs
PATCH v4 1/1 x86: irq: Do not BUG_ON multiple unbind calls for shared pirqs
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple __pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()
-> domain_relinquish_resources()
-> pci_release_devices()
-> pci_clean_dpci_irq()
-> pirq_guest_unbind()
-> __pirq_guest_unbind()For a shared pirq (nr_guests > 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call, __pirq_guest_unbind searches for the current domain which has been removed from the guests[] list, and hits a BUG_ON.
Make __pirq_guest_unbind safe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain's pirq_tree during the destruction in complete_domain_destroy anyway.
---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>
Roger suggested cleaning the entry from the domain pirq_tree so that we need not make it safe to re-call __pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.
v4:
- Re-work the guest array search to make it clearer
v3:
- Style fixups
v2:
- Split the check on action->nr_guests > 0 and make it an ASSERT
--- xen/arch/x86/irq.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
xen/arch/x86/irq.c +17 -3
@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(BUG_ON (!(desc->status &IRQ_GUEST )); -for ( i =0 ; (i < action->nr_guests) && (action->guest[i] != d); i++ ) -continue ; -BUG_ON (i == action->nr_guests); +for ( i =0 ; i < action->nr_guests; i++ ) +if ( action->guest[i] == d ) +break ; + +if ( i == action->nr_guests )/* No matching entry */ + { +/* +* In case the pirq was shared, unbound for this domain in an earlier +* call, but still existed on the domain's pirq_tree, we still reach +* here if there are any later unbind calls on the same pirq. Return +* if such an unbind happens. +*/ +ASSERT (action->shareable); +return NULL ; + } + +ASSERT (action->nr_guests >0 );memmove (&action->guest[i], &action->guest[i+1 ], (action->nr_guests-i-1 ) *sizeof (action->guest[0 ])); action->nr_guests--;
Signature · 7 lines
2.20.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple__pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:
domain_kill()-> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()For a shared pirq (
nr_guests> 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call,__pirq_guest_unbindsearches for the current domain which has been removed from the guests[] list, and hits aBUG_ON.Make
__pirq_guest_unbindsafe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain'spirq_treeduring the destruction incomplete_domain_destroyanyway.Signed-off-by Varad Gautam <[email protected]>[taking over from Varad at v4]Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>Roger suggested cleaning the entry from the domain
pirq_treeso that we need not make it safe to re-call__pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.
Quoted 2 lines
v4:
- Re-work the guest array search to make it clearer
I.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem. But anyway:
Quoted 16 lines
xen/arch/x86/irq.c +7
-3@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(BUG_ON (!(desc->status &IRQ_GUEST )); -for ( i =0 ; (i < action->nr_guests) && (action->guest[i] != d); i++ ) -continue ; -BUG_ON (i == action->nr_guests); +for ( i =0 ; i < action->nr_guests; i++ ) +if ( action->guest[i] == d ) +break ; + +if ( i == action->nr_guests )/* No matching entry */ + { +/*
xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699
Quoted 10 lines
xen/arch/x86/irq.c +10
-0+* In case the pirq was shared, unbound for this domain in an earlier +* call, but still existed on the domain's pirq_tree, we still reach +* here if there are any later unbind calls on the same pirq. Return +* if such an unbind happens. +*/ +ASSERT (action->shareable); +return NULL ; + } + +ASSERT (action->nr_guests >0 );
This seems pointless to have here - v3 had it inside the if(), where it would actually guard against coming here with nr_guests equal to zero. v3 also used if() and BUG() instead of ASSERT() inside this if(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 49 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple__pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:
domain_kill()-> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()For a shared pirq (
nr_guests> 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call,__pirq_guest_unbindsearches for the current domain which has been removed from the guests[] list, and hits aBUG_ON.Make
__pirq_guest_unbindsafe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain'spirq_treeduring the destruction incomplete_domain_destroyanyway.Signed-off-by Varad Gautam <[email protected]>[taking over from Varad at v4]Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>Roger suggested cleaning the entry from the domain
pirq_treeso that we need not make it safe to re-call__pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.
Quoted 6 lines
Quoted 2 lines
v4:
- Re-work the guest array search to make it clearerI.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.
Please suggest code if you think it ought to be done differentely. I tried.
Quoted 24 lines
But anyway:
Quoted 22 lines
xen/arch/x86/irq.c +13
-3@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(BUG_ON (!(desc->status &IRQ_GUEST )); -for ( i =0 ; (i < action->nr_guests) && (action->guest[i] != d); i++ ) -continue ; -BUG_ON (i == action->nr_guests); +for ( i =0 ; i < action->nr_guests; i++ ) +if ( action->guest[i] == d ) +break ; + +if ( i == action->nr_guests )/* No matching entry */ + { +/* +* In case the pirq was shared, unbound for this domain in an earlier +* call, but still existed on the domain's pirq_tree, we still reach +* here if there are any later unbind calls on the same pirq. Return +* if such an unbind happens. +*/ +ASSERT (action->shareable);
Quoted 8 lines
xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699
Quoted 4 lines
xen/arch/x86/irq.c +4
-0+return NULL ; + } + +ASSERT (action->nr_guests >0 );This seems pointless to have here - v3 had it inside the
if(), where it would actually guard against coming here withnr_guestsequal to zero.
Why. The code just after this decrements nr_guests so it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.
Quoted 5 lines
v3 also used
if()andBUG()instead ofASSERT()inside thisif(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?
Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a BUG_ON() did not seem appropriate.
Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 36 lines · Paul Durrant
Quoted 27 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29Quoted 22 lines · On 06.03.2020 17:02, [email protected] wrote:
xen/arch/x86/irq.c +13
-3@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(BUG_ON (!(desc->status &IRQ_GUEST )); -for ( i =0 ; (i < action->nr_guests) && (action->guest[i] != d); i++ ) -continue ; -BUG_ON (i == action->nr_guests); +for ( i =0 ; i < action->nr_guests; i++ ) +if ( action->guest[i] == d ) +break ; + +if ( i == action->nr_guests )/* No matching entry */ + { +/* +* In case the pirq was shared, unbound for this domain in an earlier +* call, but still existed on the domain's pirq_tree, we still reach +* here if there are any later unbind calls on the same pirq. Return +* if such an unbind happens. +*/ +ASSERT (action->shareable);Quoted 8 lines
xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699
Quoted 4 lines
xen/arch/x86/irq.c +4
-0+return NULL ; + } + +ASSERT (action->nr_guests >0 );This seems pointless to have here - v3 had it inside the
if(), where it would actually guard against coming here withnr_guestsequal to zero.
Quoted 3 lines
Why. The code just after this decrements
nr_guestsso it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.
There's no way to come here when nr_guests == 0. This is because in this case the loop will be exited with i being zero, and hence the earlier if()'s body will be entered.
(And no, >= 0 wouldn't make sense to me - it would mean we might have a count of -1 after the decrement.)
Quoted 10 lines
Quoted 5 lines
v3 also used
if()andBUG()instead ofASSERT()inside thisif(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a
BUG_ON()did not seem appropriate.
It'll mean we have a non-sharable IRQ in a place where this is not supposed to happen. How can we be sure the system is in a state allowing to safely continue? To me, if shareable / non- shareable is of any concern here, then it ought to be BUG(). If it's not, then the ASSERT() ought to be dropped as well.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 45 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 10 March 2020 11:23
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 36 lines · Paul Durrant
Quoted 35 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29Quoted 16 lines · On 06.03.2020 17:02, [email protected] wrote:
xen/arch/x86/irq.c +7
-3@@ -1680,9 +1680,23 @@ static irq_guest_action_t *__pirq_guest_unbind(BUG_ON (!(desc->status &IRQ_GUEST )); -for ( i =0 ; (i < action->nr_guests) && (action->guest[i] != d); i++ ) -continue ; -BUG_ON (i == action->nr_guests); +for ( i =0 ; i < action->nr_guests; i++ ) +if ( action->guest[i] == d ) +break ; + +if ( i == action->nr_guests )/* No matching entry */ + { +/* xen/arch/x86/irq.c static irq_guest_action_t *__pirq_guest_unbind( · line 1699
Quoted 10 lines
xen/arch/x86/irq.c +10
-0+* In case the pirq was shared, unbound for this domain in an earlier +* call, but still existed on the domain's pirq_tree, we still reach +* here if there are any later unbind calls on the same pirq. Return +* if such an unbind happens. +*/ +ASSERT (action->shareable); +return NULL ; + } + +ASSERT (action->nr_guests >0 );This seems pointless to have here - v3 had it inside the
if(), where it would actually guard against coming here withnr_guestsequal to zero.
Quoted 7 lines
Quoted 3 lines
Why. The code just after this decrements
nr_guestsso it seems like entirely the right point to have the ASSERT. I can make it ASSERT >= 0, if that makes more sense.There's no way to come here when
nr_guests== 0. This is because in this case the loop will be exited with i being zero, and hence the earlier if()'s body will be entered.
Ok, yes, that's true.
Quoted 15 lines
(And no, >= 0 wouldn't make sense to me - it would mean we might have a count of -1 after the decrement.)
Quoted 10 lines
Quoted 5 lines
v3 also used
if()andBUG()instead ofASSERT()inside thisif(), which to me would seem more in line with our current ./CODING_STYLE guidelines of handling unexpected conditions. Could you clarify why you switched things?Because I don't think there is need to kill the host in a non-debug context if we hit this condition; it is entirely survivable as far as I can tell so a
BUG_ON()did not seem appropriate.
Quoted 5 lines
It'll mean we have a non-sharable IRQ in a place where this is not supposed to happen. How can we be sure the system is in a state allowing to safely continue? To me, if shareable / non- shareable is of any concern here, then it ought to be
BUG(). If it's not, then theASSERT()ought to be dropped as well.
Ok, I'll convert back to a BUG().
Paul
Quoted 2 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 1 line · Paul Durrant
Ok, I'll convert back to a
BUG().
Wait a little - I think I have an alternative proposal. Just want to at least smoke test it before sending out.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 49 lines · Paul Durrant
Quoted 49 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple__pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:
domain_kill()-> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()For a shared pirq (
nr_guests> 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call,__pirq_guest_unbindsearches for the current domain which has been removed from the guests[] list, and hits aBUG_ON.Make
__pirq_guest_unbindsafe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain'spirq_treeduring the destruction incomplete_domain_destroyanyway.Signed-off-by Varad Gautam <[email protected]>[taking over from Varad at v4]Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>Roger suggested cleaning the entry from the domain
pirq_treeso that we need not make it safe to re-call__pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
v4:
- Re-work the guest array search to make it clearerI.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.
Please suggest code if you think it ought to be done differentely. I tried.
How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.
Jan
x86/pass-through: avoid double IRQ unbind during domain cleanup
XEN_DOMCTL_destroydomain creates a continuation if domain_kill -ERESTARTS. In that scenario, it is possible to receive multiple _pirq_guest_unbind calls for the same pirq from domain_kill, if the pirq has not yet been removed from the domain's pirq_tree, as:
domain_kill()
-> domain_relinquish_resources()
-> pci_release_devices()
-> pci_clean_dpci_irq()
-> pirq_guest_unbind()
-> __pirq_guest_unbind()Avoid recurring invocations of pirq_guest_unbind() by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.
xen/arch/x86/irq.c +1 -1
@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p }if (radix_tree_delete (&d->pirq_tree, pirq->pirq) != pirq ) -BUG (); +BUG_ON (!d->is_dying); }/* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */
xen/drivers/passthrough/pci.c +19 -2
@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct domxfree (digl); } -return pt_pirq_softirq_active (pirq_dpci) ? -ERESTART :0 ; +radix_tree_delete (&d->pirq_tree,dpci_pirq (pirq_dpci)->pirq); + +if ( !pt_pirq_softirq_active (pirq_dpci) ) +return 0 ; + +domain_get_irq_dpci (d)->pending_pirq_dpci = pirq_dpci; + +return -ERESTART ; }static int pci_clean_dpci_irqs (struct domain *d)
@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do hvm_irq_dpci =domain_get_irq_dpci (d);if ( hvm_irq_dpci !=NULL ) { -int ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL ); +int ret =0 ; + +if ( hvm_irq_dpci->pending_pirq_dpci ) + { +if (pt_pirq_softirq_active (hvm_irq_dpci->pending_pirq_dpci) ) + ret = -ERESTART ; +else + hvm_irq_dpci->pending_pirq_dpci =NULL ; + } +if ( !ret ) + ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL );if ( ret ) {spin_unlock (&d->event_lock);
xen/include/asm-x86/hvm/irq.h +2 -0
@@ -158,6 +158,8 @@ struct hvm_irq_dpci {DECLARE_BITMAP (isairq_map,NR_ISAIRQS );/* Record of mapped Links */ uint8_t link_cnt[NR_LINK ]; +/* Clean up: Entry with a softirq invocation pending / in progress. */ +struct hvm_pirq_dpci *pending_pirq_dpci; };/* Machine IRQ to guest device/intx mapping. */
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 66 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 10 March 2020 14:19
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 5 lines · Paul Durrant
Quoted 5 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>;Roger
Quoted 51 lines
Quoted 50 lines
Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple__pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:
domain_kill()-> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()For a shared pirq (
nr_guests> 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call,__pirq_guest_unbindsearches for the current domain which has been removed from the guests[] list, and hits aBUG_ON.Make
__pirq_guest_unbindsafe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain'spirq_treeduring the destruction incomplete_domain_destroyanyway.Signed-off-by Varad Gautam <[email protected]>[taking over from Varad at v4]Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>Roger suggested cleaning the entry from the domain
pirq_treeso that we need not make it safe to re-call__pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.Quoted 2 lines
v4:
- Re-work the guest array search to make it clearerI.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.
Quoted 5 lines
Quoted 1 line
Please suggest code if you think it ought to be done differentely. I tried.
How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.
It's fairly hard to hit IIRC but we could probably engineer it with a one off ERESTART in the right place.
Quoted 76 lines
Jan
x86/pass-through: avoid double IRQ unbind during domain cleanup
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple_pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:
domain_kill()-> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()Avoid recurring invocations of
pirq_guest_unbind()by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.Reported-by Varad Gautam <[email protected]>Signed-off-by Jan Beulich <[email protected]>xen/arch/x86/irq.c +1
-1@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p }if (radix_tree_delete (&d->pirq_tree, pirq->pirq) != pirq ) -BUG (); +BUG_ON (!d->is_dying); }/* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */ xen/drivers/passthrough/pci.c +19
-2@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct domxfree (digl); } -return pt_pirq_softirq_active (pirq_dpci) ? -ERESTART :0 ; +radix_tree_delete (&d->pirq_tree,dpci_pirq (pirq_dpci)->pirq); + +if ( !pt_pirq_softirq_active (pirq_dpci) ) +return 0 ; + +domain_get_irq_dpci (d)->pending_pirq_dpci = pirq_dpci; + +return -ERESTART ; }static int pci_clean_dpci_irqs (struct domain *d)@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do hvm_irq_dpci =domain_get_irq_dpci (d);if ( hvm_irq_dpci !=NULL ) { -int ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL ); +int ret =0 ; + +if ( hvm_irq_dpci->pending_pirq_dpci ) + { +if (pt_pirq_softirq_active (hvm_irq_dpci->pending_pirq_dpci) ) + ret = -ERESTART ; +else + hvm_irq_dpci->pending_pirq_dpci =NULL ; + } +if ( !ret ) + ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL );if ( ret ) {spin_unlock (&d->event_lock);
xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 164
Quoted 7 lines
xen/include/asm-x86/hvm/irq.h +2
-0@@ -158,6 +158,8 @@ struct hvm_irq_dpci {DECLARE_BITMAP (isairq_map,NR_ISAIRQS );/* Record of mapped Links */ uint8_t link_cnt[NR_LINK ]; +/* Clean up: Entry with a softirq invocation pending / in progress. */ +struct hvm_pirq_dpci *pending_pirq_dpci; };
That looks like it will do the job. I'll see if I can get it tested.
Paul
xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 165
Quoted 2 lines
xen/include/asm-x86/hvm/irq.h +0
-0/* Machine IRQ to guest device/intx mapping. */
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 1 line · Paul Durrant
That looks like it will do the job. I'll see if I can get it tested.
Any luck with this, yet?
Jan
Quoted 8 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41
To: [email protected]
Cc: [email protected]; 'Varad Gautam' <[email protected]>; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqs
Quoted 5 lines
Quoted 1 line · Paul Durrant
That looks like it will do the job. I'll see if I can get it tested.
Any luck with this, yet?
I have asked Varad to test it. He hopes to get to it this week.
Paul
Quoted 9 lines · Paul Durrant
Quoted 7 lines
From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41Quoted 1 line · Paul Durrant
That looks like it will do the job. I'll see if I can get it tested.
Any luck with this, yet?
I have asked Varad to test it. He hopes to get to it this week.
Any news here?
Jan
Quoted 12 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 23 April 2020 12:09
To: [email protected]; 'Varad Gautam' <[email protected]>
Cc: [email protected]; 'Julien Grall' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Andrew Cooper' <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 3 lines · Paul Durrant
Quoted 3 lines
From: Jan Beulich <[email protected]>
Sent: 31 March 2020 08:41
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines · Paul Durrant
Quoted 1 line
That looks like it will do the job. I'll see if I can get it tested.
Any luck with this, yet?
I have asked Varad to test it. He hopes to get to it this week.
Any news here?
Varad tells me he is currently testing and will get back to you soon (hopefully today).
Paul
[email protected]
Tested-by Varad GautamHi Jan,
Quoted 55 lines · Jan Beulich
Quoted 54 lines · Paul Durrant
Quoted 54 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 09 March 2020 16:29
To: [email protected]
Cc: [email protected]; Varad Gautam <[email protected]>; Julien Grall <[email protected]>; Roger Pau Monné <[email protected]>; Andrew Cooper <[email protected]>
Subject: Re: [PATCH v4] x86: irq: Do notBUG_ONmultiple unbind calls for shared pirqsQuoted 40 lines · On 06.03.2020 17:02, [email protected] wrote:
From: Varad Gautam <[email protected]>
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple__pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:domain_kill() -> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()For a shared pirq (
nr_guests> 1), the first call would zap the current domain from the pirq's guests[] list, but the action handler is never freed as there are other guests using this pirq. As a result, on the second call,__pirq_guest_unbindsearches for the current domain which has been removed from the guests[] list, and hits aBUG_ON.Make
__pirq_guest_unbindsafe to be called multiple times by letting xen continue if a shared pirq has already been unbound from this guest. The PIRQ will be cleaned up from the domain'spirq_treeduring the destruction incomplete_domain_destroyanyway.Signed-off-by Varad Gautam <[email protected]>[taking over from Varad at v4]Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Roger Pau Monné <[email protected]>
Cc: Andrew Cooper <[email protected]>Roger suggested cleaning the entry from the domain
pirq_treeso that we need not make it safe to re-call__pirq_guest_unbind(). This seems like a reasonable suggestion but the semantics of the code are almost impenetrable (e.g. 'pirq' is used to mean an index, a pointer and is also the name of struct so you generally have little idea what it actally means) so I prefer to stick with a small fix that I can actually reason about.Quoted 2 lines
v4:
- Re-work the guest array search to make it clearerI.e. there are cosmetic differences to v3 (see below), but technically it's still the same. I can't believe the re-use of "pirq" for different entities is this big of a problem.
Quoted 8 lines
Quoted 1 line
Please suggest code if you think it ought to be done differentely. I tried.
How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.
Jan
x86/pass-through: avoid double IRQ unbind during domain cleanup
I have tested that this patch prevents __pirq_guest_unbind on an already-unbound pirq during the continuation call for domain_kill -ERESTART, by using a modified xen that forces an -ERESTART from pirq_guest_unbind to create the continuation. It fixes the underlying issue.
Quoted 74 lines
XEN_DOMCTL_destroydomaincreates a continuation ifdomain_kill-ERESTARTS. In that scenario, it is possible to receive multiple_pirq_guest_unbindcalls for the same pirq fromdomain_kill, if the pirq has not yet been removed from the domain'spirq_tree, as:domain_kill() -> domain_relinquish_resources() -> pci_release_devices() -> pci_clean_dpci_irq() -> pirq_guest_unbind() -> __pirq_guest_unbind()Avoid recurring invocations of
pirq_guest_unbind()by removing the pIRQ from the tree being iterated after the first call there. In case such a removed entry still has a softirq outstanding, record it and re-check upon re-invocation.Reported-by Varad Gautam <[email protected]>Signed-off-by Jan Beulich <[email protected]>xen/arch/x86/irq.c +1
-1@@ -1323,7 +1323,7 @@ void (pirq_cleanup_check)(struct pirq *p }if (radix_tree_delete (&d->pirq_tree, pirq->pirq) != pirq ) -BUG (); +BUG_ON (!d->is_dying); }/* Flush all ready EOIs from the top of this CPU's pending-EOI stack. */ xen/drivers/passthrough/pci.c +19
-2@@ -873,7 +873,14 @@ static int pci_clean_dpci_irq(struct domxfree (digl); } -return pt_pirq_softirq_active (pirq_dpci) ? -ERESTART :0 ; +radix_tree_delete (&d->pirq_tree,dpci_pirq (pirq_dpci)->pirq); + +if ( !pt_pirq_softirq_active (pirq_dpci) ) +return 0 ; + +domain_get_irq_dpci (d)->pending_pirq_dpci = pirq_dpci; + +return -ERESTART ; }static int pci_clean_dpci_irqs (struct domain *d)@@ -890,8 +897,18 @@ static int pci_clean_dpci_irqs(struct do hvm_irq_dpci =domain_get_irq_dpci (d);if ( hvm_irq_dpci !=NULL ) { -int ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL ); +int ret =0 ; + +if ( hvm_irq_dpci->pending_pirq_dpci ) + { +if (pt_pirq_softirq_active (hvm_irq_dpci->pending_pirq_dpci) ) + ret = -ERESTART ; +else + hvm_irq_dpci->pending_pirq_dpci =NULL ; + } +if ( !ret ) + ret =pt_pirq_iterate (d, pci_clean_dpci_irq,NULL );if ( ret ) {spin_unlock (&d->event_lock);
xen/include/asm-x86/hvm/irq.h struct hvm_irq_dpci { · line 165
Quoted 8 lines
xen/include/asm-x86/hvm/irq.h +2
-0@@ -158,6 +158,8 @@ struct hvm_irq_dpci {DECLARE_BITMAP (isairq_map,NR_ISAIRQS );/* Record of mapped Links */ uint8_t link_cnt[NR_LINK ]; +/* Clean up: Entry with a softirq invocation pending / in progress. */ +struct hvm_pirq_dpci *pending_pirq_dpci; };/* Machine IRQ to guest device/intx mapping. */
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Quoted 12 lines · On 28.04.2020 13:58, [email protected] wrote:
Quoted 9 lines · Jan Beulich
Quoted 1 line · Paul Durrant
Please suggest code if you think it ought to be done differentely. I tried.
How about this? It's admittedly more code, but imo less ad hoc. I've smoke tested it, but I depend on you or Varad to check that it actually addresses the reported issue.
Jan
x86/pass-through: avoid double IRQ unbind during domain cleanup
Quoted 6 lines
I have tested that this patch prevents
__pirq_guest_unbindon an already-unbound pirq during the continuation call fordomain_kill-ERESTART, by using a modified xen that forces an-ERESTARTfrompirq_guest_unbindto create the continuation. It fixes the underlying issue.Tested-by Varad Gautam <[email protected]>
Thanks much; I'll formally submit the patch then.
Jan