xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed
PATCH 1/1 xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.
This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor. One of the consequence is the domain can never be fully destroyed as some of its memory is still mapped.
As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap pages during the domain destruction if there are any left.
We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().
NOTE: The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
--- xen/arch/x86/cpu/vpmu.c | 45 +++++++++++++++++++++++------------------ xen/arch/x86/domain.c | 6 +++--- 2 files changed, 28 insertions(+), 23 deletions(-)
xen/arch/x86/cpu/vpmu.c +25 -20
@@ -578,9 +578,32 @@ static void vpmu_arch_destroy(struct vcpu *v) } } -void vpmu_destroy (struct vcpu *v) +static void vpmu_cleanup (struct vcpu *v) { +struct vpmu_struct *vpmu =vcpu_vpmu (v); +mfn_t mfn; +void *xenpmu_data; + +spin_lock (&vpmu->vpmu_lock); +vpmu_arch_destroy (v); + xenpmu_data = vpmu->xenpmu_data; + vpmu->xenpmu_data =NULL ; + +spin_unlock (&vpmu->vpmu_lock); + +if ( xenpmu_data ) + { + mfn =domain_page_map_to_mfn (xenpmu_data); +ASSERT (mfn_valid (mfn)); +unmap_domain_page_global (xenpmu_data); +put_page_and_type (mfn_to_page (mfn)); + } +} + +void vpmu_destroy (struct vcpu *v) +{ +vpmu_cleanup (v);put_vpmu (v); }
@@ -639,9 +662,6 @@ static int pvpmu_init(struct domain *d, xen_pmu_params_t *params)static void pvpmu_finish (struct domain *d,xen_pmu_params_t *params) {struct vcpu *v; -struct vpmu_struct *vpmu; -mfn_t mfn; -void *xenpmu_data;if ( (params->vcpu >= d->max_vcpus) || (d->vcpu[params->vcpu] ==NULL ) )return ;
@@ -650,22 +670,7 @@ static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)if ( v != current )vcpu_pause (v); - vpmu =vcpu_vpmu (v); -spin_lock (&vpmu->vpmu_lock); - -vpmu_arch_destroy (v); - xenpmu_data = vpmu->xenpmu_data; - vpmu->xenpmu_data =NULL ; - -spin_unlock (&vpmu->vpmu_lock); - -if ( xenpmu_data ) - { - mfn =domain_page_map_to_mfn (xenpmu_data); -ASSERT (mfn_valid (mfn)); -unmap_domain_page_global (xenpmu_data); -put_page_and_type (mfn_to_page (mfn)); - } +vpmu_cleanup (v);if ( v != current )vcpu_unpause (v);
xen/arch/x86/domain.c +3 -3
@@ -454,9 +454,6 @@ void arch_vcpu_destroy(struct vcpu *v)xfree (v->arch.msrs); v->arch.msrs =NULL ; -if ( !is_idle_domain (v->domain) ) -vpmu_destroy (v); -if (is_hvm_vcpu (v) )hvm_vcpu_destroy (v);else
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d); + for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }
Signature · 7 lines
2.20.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228
Quoted 9 lines
xen/arch/x86/domain.c +3
-0@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d); + for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }
I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 17 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 27 November 2019 09:44
To: Durrant, Paul <[email protected]>; Grall, Julien <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyedQuoted 6 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.
Quoted 8 lines
Quoted 2 lines
This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Isn't this still too weak a description? It's not the tool stack invoking
XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.
Ok, how about 'if the guest fails to invoke XENPMU_finish, e.g. if it is destroyed, rather than cleanly shut down'?
Quoted 4 lines
Quoted 4 lines
xen/arch/x86/domain.c +0
-0@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d);
Quoted 12 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228
Quoted 5 lines
xen/arch/x86/domain.c +3
-0+ for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }I think simple things which may allow shrinking the page lists should be done early in the function. As
vpmu_destroy()looks to be idempotent, how about leveraging the very firstfor_each_vcpu()loop in the function (there are too many of them there anyway, at least for my taste)?
Ok. I did wonder where in the sequence was best... Leaving to the end obviously puts it closer to where it was previously called, but I can't see any harm in moving it earlier.
Paul
Quoted 1 line
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 20 lines · Paul
Quoted 20 lines
-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 27 November 2019 09:44
To: Durrant, Paul <[email protected]>; Grall, Julien <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyedQuoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Quoted 8 lines
Quoted 5 lines
Isn't this still too weak a description? It's not the tool stack invoking
XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.Ok, how about 'if the guest fails to invoke
XENPMU_finish, e.g. if it is destroyed, rather than cleanly shut down'?
Sounds good.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi,
Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Isn't this still too weak a description? It's not the tool stack invoking
XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.Quoted 4 lines
xen/arch/x86/domain.c +0
-0@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d);
Quoted 11 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228
Quoted 5 lines
xen/arch/x86/domain.c +3
-0+ for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }I think simple things which may allow shrinking the page lists should be done early in the function. As
vpmu_destroy()looks to be idempotent, how about leveraging the very firstfor_each_vcpu()loop in the function (there are too many of them there anyway, at least for my taste)?
This is not entirely obvious that vpmu_destroy() is idempotent.
For instance, I can't find out who is clearing VCPU_CONTEXT_ALLOCATED. so I think vcpu_arch_destroy() would be executed over and over.
I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?
Cheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 44 lines
-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 27 November 2019 11:15
To: Jan Beulich <[email protected]>; Durrant, Paul <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyedHi,
Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Isn't this still too weak a description? It's not the tool stack invoking
XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.Quoted 4 lines
xen/arch/x86/domain.c +0
-0@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d);Quoted 11 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228
Quoted 5 lines
xen/arch/x86/domain.c +3
-0+ for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }I think simple things which may allow shrinking the page lists should be done early in the function. As
vpmu_destroy()looks to be idempotent, how about leveraging the very firstfor_each_vcpu()loop in the function (there are too many of them there anyway, at least for my taste)?
Quoted 7 lines
This is not entirely obvious that
vpmu_destroy()is idempotent.For instance, I can't find out who is clearing
VCPU_CONTEXT_ALLOCATED. so I thinkvcpu_arch_destroy()would be executed over and over.I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?
It's sufficiently unobvious that it is a concern whether a guest invoking XENPMU_finish multiple times can cause harm. I'll see if I can clean that up.
Paul
Quoted 2 lines
Cheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 34 lines · Julien Grall
Hi,
Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant
From: Julien Grall <[email protected]>
A guest will setup a shared page with the hypervisor for each vCPU via
XENPMU_init. The page will then get mapped in the hypervisor and only released whenXEMPMU_finishis called.This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.
Isn't this still too weak a description? It's not the tool stack invoking
XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.Quoted 4 lines
xen/arch/x86/domain.c +0
-0@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)if (is_hvm_domain (d) )hvm_domain_relinquish_resources (d);Quoted 11 lines
xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228
Quoted 5 lines
xen/arch/x86/domain.c +3
-0+ for_each_vcpu ( d, v ) +vpmu_destroy (v); +return 0 ; }I think simple things which may allow shrinking the page lists should be done early in the function. As
vpmu_destroy()looks to be idempotent, how about leveraging the very firstfor_each_vcpu()loop in the function (there are too many of them there anyway, at least for my taste)?
Quoted 7 lines
This is not entirely obvious that
vpmu_destroy()is idempotent.For instance, I can't find out who is clearing
VCPU_CONTEXT_ALLOCATED. so I thinkvcpu_arch_destroy()would be executed over and over.I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?
If the function wasn't idempotent, then calling it unconditionally from domain_relinquish_resources() would be wrong too. After all the guest may have invoked XENPMU_finish.
As to VCPU_CONTEXT_ALLOCATED - I don't think this ever gets cleared anywhere. But this by itself doesn't make the function non- idempotent. The vpmu_clear_last invocation, for example, will happen just once. And {amd,core2}_vpmu_destroy() look okay at the first glance, too.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel