xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed
PATCH v3 1/1 xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.
This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.
As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.
We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().
NOTE: - The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
- Whilst it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence make sure the VPMU_CONTEXT_ALLOCATED flag is cleared at the end of vpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>
v2:
- Re-word commit comment slightly
- Re-enforce idempotency of vmpu_arch_destroy()
- Move invocation of vpmu_destroy() earlier in
domain_relinquish_resources()
--- xen/arch/x86/cpu/vpmu.c | 47 +++++++++++++++++++++++------------------ xen/arch/x86/domain.c | 10 +++++---- 2 files changed, 33 insertions(+), 24 deletions(-)
xen/arch/x86/cpu/vpmu.c +27 -20
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); } -void vpmu_destroy (struct vcpu *v) +static void vpmu_cleanup (struct vcpu *v) { +struct vpmu_struct *vpmu =vcpu_vpmu (v); +mfn_t mfn; +void *xenpmu_data; + +spin_lock (&vpmu->vpmu_lock); +vpmu_arch_destroy (v); + xenpmu_data = vpmu->xenpmu_data; + vpmu->xenpmu_data =NULL ; + +spin_unlock (&vpmu->vpmu_lock); + +if ( xenpmu_data ) + { + mfn =domain_page_map_to_mfn (xenpmu_data); +ASSERT (mfn_valid (mfn)); +unmap_domain_page_global (xenpmu_data); +put_page_and_type (mfn_to_page (mfn)); + } +} + +void vpmu_destroy (struct vcpu *v) +{ +vpmu_cleanup (v);put_vpmu (v); }
@@ -639,9 +664,6 @@ static int pvpmu_init(struct domain *d, xen_pmu_params_t *params)static void pvpmu_finish (struct domain *d,xen_pmu_params_t *params) {struct vcpu *v; -struct vpmu_struct *vpmu; -mfn_t mfn; -void *xenpmu_data;if ( (params->vcpu >= d->max_vcpus) || (d->vcpu[params->vcpu] ==NULL ) )return ;
@@ -650,22 +672,7 @@ static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)if ( v != current )vcpu_pause (v); - vpmu =vcpu_vpmu (v); -spin_lock (&vpmu->vpmu_lock); - -vpmu_arch_destroy (v); - xenpmu_data = vpmu->xenpmu_data; - vpmu->xenpmu_data =NULL ; - -spin_unlock (&vpmu->vpmu_lock); - -if ( xenpmu_data ) - { - mfn =domain_page_map_to_mfn (xenpmu_data); -ASSERT (mfn_valid (mfn)); -unmap_domain_page_global (xenpmu_data); -put_page_and_type (mfn_to_page (mfn)); - } +vpmu_cleanup (v);if ( v != current )vcpu_unpause (v);
xen/arch/x86/domain.c +6 -4
@@ -454,9 +454,6 @@ void arch_vcpu_destroy(struct vcpu *v)xfree (v->arch.msrs); v->arch.msrs =NULL ; -if ( !is_idle_domain (v->domain) ) -vpmu_destroy (v); -if (is_hvm_vcpu (v) )hvm_vcpu_destroy (v);else
@@ -2136,12 +2133,17 @@ int domain_relinquish_resources(struct domain *d) PROGRESS(vcpu_pagetables): -/* Drop the in-use references to page-table bases. */ +/* +* Drop the in-use references to page-table bases and clean +* up vPMU instances. +*/ for_each_vcpu (d ,v ) { ret =vcpu_destroy_pagetables (v);if ( ret )return ret; + +vpmu_destroy (v); }if (altp2m_active (d) )
Signature · 7 lines
2.20.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 39 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXENPMU_finishis called.This means that if the guest fails to invoke
XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.
We can re-use the same logic as in
pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called fromvpmu_destroy().NOTE: - The call to
vpmu_destroy()must also be moved from
arch_vcpu_destroy()intodomain_relinquish_resources()such that the reference on the mapped page does not preventdomain_destroy()(which callsarch_vcpu_destroy()) from being called.
- Whilst it appears thatvpmu_arch_destroy()is idempotent it is by no means obvious. Hence make sure theVPMU_CONTEXT_ALLOCATEDflag is cleared at the end ofvpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).Signed-off-by Julien Grall <[email protected]>Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>
Quoted 5 lines
v2:
- Re-word commit comment slightly
- Re-enforce idempotency ofvmpu_arch_destroy()
- Move invocation ofvpmu_destroy()earlier in
domain_relinquish_resources()
What about v3?
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581
Quoted 9 lines
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }
Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 54 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 28 November 2019 10:23
To: Durrant, Paul <[email protected]>; Boris Ostrovsky <[email protected]>
Cc: [email protected]; Grall, Julien <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Jun Nakajima <[email protected]>; Kevin Tian <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v3] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyedQuoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXENPMU_finishis called.This means that if the guest fails to invoke
XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped inthe
Quoted 7 lines
hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.
As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.
Quoted 7 lines
We can re-use the same logic as in
pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called fromvpmu_destroy().NOTE: - The call to
vpmu_destroy()must also be moved from
arch_vcpu_destroy()intodomain_relinquish_resources()such that the reference on the mapped page does not prevent
domain_destroy()Quoted 17 lines
(which calls
arch_vcpu_destroy()) from being called.
- Whilst it appears thatvpmu_arch_destroy()is idempotent it is by no means obvious. Hence make sure theVPMU_CONTEXT_ALLOCATEDflag is cleared at the end ofvpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).Signed-off-by Julien Grall <[email protected]>Signed-off-by Paul Durrant <[email protected]>---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>
Quoted 7 lines
Quoted 5 lines
v2:
- Re-word commit comment slightly
- Re-enforce idempotency ofvmpu_arch_destroy()
- Move invocation ofvpmu_destroy()earlier in
domain_relinquish_resources()What about v3?
Oh, sorry:
v3:
- Add comment regarding XSA-163
- Revert changes setting VPMU_CONTEXT_ALLOCATED in common code
Paul
Quoted 14 lines
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581
Quoted 10 lines
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Quoted 1 line
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 6 lines · Jan Beulich
Quoted 4 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +0
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
Quoted 7 lines
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581
Quoted 5 lines
xen/arch/x86/cpu/vpmu.c +2
-0vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Yes, this is fine.
(I probably would clear it in arch_vpmu_destroy op since it is set in arch-specific code but either way works)
-boris
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Boris,
Quoted 15 lines · Boris Ostrovsky
Quoted 6 lines · Jan Beulich
Quoted 4 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +0
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)Quoted 7 lines
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581
Quoted 5 lines
xen/arch/x86/cpu/vpmu.c +2
-0vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Quoted 1 line
Yes, this is fine.
Can we take this as an ACK?
Quoted 6 lines
(I probably would clear it in
arch_vpmu_destroyop since it is set in arch-specific code but either way works)-boris
Cheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 20 lines · Julien Grall
Hi Boris,
Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580
Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Quoted 3 lines
Quoted 1 line
Yes, this is fine.
Can we take this as an ACK?
Yes, of course.
-boris
Quoted 6 lines
Quoted 4 lines
(I probably would clear it in
arch_vpmu_destroyop since it is set in arch-specific code but either way works)
-borisCheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 20 lines · Boris Ostrovsky
Quoted 20 lines · Julien Grall
Hi Boris,
Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580
Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Yes, this is fine.
Can we take this as an ACK?
Yes, of course.
Thank you!
@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 25 lines · Julien Grall
Quoted 22 lines · Boris Ostrovsky
Quoted 22 lines · Julien Grall
Hi Boris,
Quoted 14 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580
Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Yes, this is fine.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Can we take this as an ACK?
Yes, of course.
Thank you!
@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.
It has been committed already.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Julien Grall
Quoted 25 lines · Boris Ostrovsky
Quoted 23 lines · Julien Grall
Hi Boris,
Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich
xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580
Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2
-0@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v) vpmu->arch_vpmu_ops->arch_vpmu_destroy (v); } + +vpmu_reset (vpmu,VPMU_CONTEXT_ALLOCATED ); }Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?
Quoted 1 line
Yes, this is fine.
Can we take this as an ACK?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Yes, of course.
Thank you!
@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.
It has been committed already.
Oh, I didn't spot it in git log. Sorry for the noise.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 29 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXENPMU_finishis called.This means that if the guest fails to invoke
XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.
We can re-use the same logic as in
pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called fromvpmu_destroy().NOTE: - The call to
vpmu_destroy()must also be moved from
arch_vcpu_destroy()intodomain_relinquish_resources()such that the reference on the mapped page does not preventdomain_destroy()(which callsarch_vcpu_destroy()) from being called.
- Whilst it appears thatvpmu_arch_destroy()is idempotent it is by no means obvious. Hence make sure theVPMU_CONTEXT_ALLOCATEDflag is cleared at the end ofvpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).
Quoted 2 lines
Signed-off-by Julien Grall <[email protected]>Signed-off-by Paul Durrant <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel