Skip to contentSkip to conversation

xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

PATCH Superseded Paul Durrant 3 participants 7 messages Nov 26, 2019 – Nov 27, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor. One of the consequence is the domain can never be fully destroyed as some of its memory is still mapped.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>

---
 xen/arch/x86/cpu/vpmu.c | 45 +++++++++++++++++++++++------------------
 xen/arch/x86/domain.c   |  6 +++---
 2 files changed, 28 insertions(+), 23 deletions(-)
xen/arch/x86/cpu/vpmu.c +25 -20
index f397183ec3..9ae4ed48c8 100644
@@ -578,9 +578,32 @@ static void vpmu_arch_destroy(struct vcpu *v)
     }
 }
 
-void vpmu_destroy(struct vcpu *v)
+static void vpmu_cleanup(struct vcpu *v)
 {
+    struct vpmu_struct *vpmu = vcpu_vpmu(v);
+    mfn_t mfn;
+    void *xenpmu_data;
+
+    spin_lock(&vpmu->vpmu_lock);
+
     vpmu_arch_destroy(v);
+    xenpmu_data = vpmu->xenpmu_data;
+    vpmu->xenpmu_data = NULL;
+
+    spin_unlock(&vpmu->vpmu_lock);
+
+    if ( xenpmu_data )
+    {
+        mfn = domain_page_map_to_mfn(xenpmu_data);
+        ASSERT(mfn_valid(mfn));
+        unmap_domain_page_global(xenpmu_data);
+        put_page_and_type(mfn_to_page(mfn));
+    }
+}
+
+void vpmu_destroy(struct vcpu *v)
+{
+    vpmu_cleanup(v);
 
     put_vpmu(v);
 }
@@ -639,9 +662,6 @@ static int pvpmu_init(struct domain *d, xen_pmu_params_t *params)
 static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
 {
     struct vcpu *v;
-    struct vpmu_struct *vpmu;
-    mfn_t mfn;
-    void *xenpmu_data;
 
     if ( (params->vcpu >= d->max_vcpus) || (d->vcpu[params->vcpu] == NULL) )
         return;
@@ -650,22 +670,7 @@ static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
     if ( v != current )
         vcpu_pause(v);
 
-    vpmu = vcpu_vpmu(v);
-    spin_lock(&vpmu->vpmu_lock);
-
-    vpmu_arch_destroy(v);
-    xenpmu_data = vpmu->xenpmu_data;
-    vpmu->xenpmu_data = NULL;
-
-    spin_unlock(&vpmu->vpmu_lock);
-
-    if ( xenpmu_data )
-    {
-        mfn = domain_page_map_to_mfn(xenpmu_data);
-        ASSERT(mfn_valid(mfn));
-        unmap_domain_page_global(xenpmu_data);
-        put_page_and_type(mfn_to_page(mfn));
-    }
+    vpmu_cleanup(v);
 
     if ( v != current )
         vcpu_unpause(v);
xen/arch/x86/domain.c +3 -3
index f1dd86e12e..1d75b2e6c3 100644
@@ -454,9 +454,6 @@ void arch_vcpu_destroy(struct vcpu *v)
     xfree(v->arch.msrs);
     v->arch.msrs = NULL;
 
-    if ( !is_idle_domain(v->domain) )
-        vpmu_destroy(v);
-
     if ( is_hvm_vcpu(v) )
         hvm_vcpu_destroy(v);
     else
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
     if ( is_hvm_domain(d) )
         hvm_domain_relinquish_resources(d);
 
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
     return 0;
 }
 
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228

Quoted 9 lines
xen/arch/x86/domain.c +3 -0
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
     if ( is_hvm_domain(d) )
         hvm_domain_relinquish_resources(d);
 
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
     return 0;
 }

I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 17 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 27 November 2019 09:44
To: Durrant, Paul <[email protected]>; Grall, Julien <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Quoted 6 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

Quoted 8 lines
Quoted 2 lines

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

Ok, how about 'if the guest fails to invoke XENPMU_finish, e.g. if it is destroyed, rather than cleanly shut down'?

Quoted 4 lines
Quoted 4 lines
xen/arch/x86/domain.c +0 -0
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
     if ( is_hvm_domain(d) )
         hvm_domain_relinquish_resources(d);
 
Quoted 12 lines

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228

Quoted 5 lines
xen/arch/x86/domain.c +3 -0
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
     return 0;
 }

I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?

Ok. I did wonder where in the sequence was best... Leaving to the end obviously puts it closer to where it was previously called, but I can't see any harm in moving it earlier.

Paul

Quoted 1 line

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 20 lines · Paul
Quoted 20 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 27 November 2019 09:44
To: Durrant, Paul <[email protected]>; Grall, Julien <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Quoted 8 lines
Quoted 5 lines

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

Ok, how about 'if the guest fails to invoke XENPMU_finish, e.g. if it is destroyed, rather than cleanly shut down'?

Sounds good.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
In reply to Jan Beulich

Hi,

Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

Quoted 4 lines
xen/arch/x86/domain.c +0 -0
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
      if ( is_hvm_domain(d) )
          hvm_domain_relinquish_resources(d);
 
Quoted 11 lines

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228

Quoted 5 lines
xen/arch/x86/domain.c +3 -0
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
      return 0;
  }

I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?

This is not entirely obvious that vpmu_destroy() is idempotent.

For instance, I can't find out who is clearing VCPU_CONTEXT_ALLOCATED. so I think vcpu_arch_destroy() would be executed over and over.

I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 44 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 27 November 2019 11:15
To: Jan Beulich <[email protected]>; Durrant, Paul <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Hi,

Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

Quoted 4 lines
xen/arch/x86/domain.c +0 -0
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
      if ( is_hvm_domain(d) )
          hvm_domain_relinquish_resources(d);
 
Quoted 11 lines

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228

Quoted 5 lines
xen/arch/x86/domain.c +3 -0
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
      return 0;
  }

I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?

Quoted 7 lines

This is not entirely obvious that vpmu_destroy() is idempotent.

For instance, I can't find out who is clearing VCPU_CONTEXT_ALLOCATED. so I think vcpu_arch_destroy() would be executed over and over.

I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?

It's sufficiently unobvious that it is a concern whether a guest invoking XENPMU_finish multiple times can cause harm. I'll see if I can clean that up.

Paul

Quoted 2 lines

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Julien Grall
Quoted 34 lines · Julien Grall

Hi,

Quoted 19 lines · Jan Beulich
Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XEMPMU_finish is called.

This means that if the guest is not shutdown gracefully (such as via xl destroy), the page will stay mapped in the hypervisor.

Isn't this still too weak a description? It's not the tool stack invoking XENPMU_finish, but the guest itself afaics. I.e. a misbehaving guest could prevent proper cleanup even with graceful shutdown.

Quoted 4 lines
xen/arch/x86/domain.c +0 -0
@@ -2224,6 +2221,9 @@ int domain_relinquish_resources(struct domain *d)
      if ( is_hvm_domain(d) )
          hvm_domain_relinquish_resources(d);
 
Quoted 11 lines

xen/arch/x86/domain.c int domain_relinquish_resources(struct domain *d) · line 2228

Quoted 5 lines
xen/arch/x86/domain.c +3 -0
+    for_each_vcpu ( d, v )
+        vpmu_destroy(v);
+
      return 0;
  }

I think simple things which may allow shrinking the page lists should be done early in the function. As vpmu_destroy() looks to be idempotent, how about leveraging the very first for_each_vcpu() loop in the function (there are too many of them there anyway, at least for my taste)?

Quoted 7 lines

This is not entirely obvious that vpmu_destroy() is idempotent.

For instance, I can't find out who is clearing VCPU_CONTEXT_ALLOCATED. so I think vcpu_arch_destroy() would be executed over and over.

I don't know whether this is an issue, but I can't figure out that is it not one. Did I miss anything?

If the function wasn't idempotent, then calling it unconditionally from domain_relinquish_resources() would be wrong too. After all the guest may have invoked XENPMU_finish.

As to VCPU_CONTEXT_ALLOCATED - I don't think this ever gets cleared anywhere. But this by itself doesn't make the function non- idempotent. The vpmu_clear_last invocation, for example, will happen just once. And {amd,core2}_vpmu_destroy() look okay at the first glance, too.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel