Skip to contentSkip to conversation

xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

PATCH Superseded Paul Durrant 4 participants 6 messages Nov 27, 2019 – Nov 28, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
Also, whils it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence move manipulation of the VPMU_CONTEXT_ALLOCATED flag out of implementation specific code and make sure it is cleared at the end of vpmu_arch_destroy().

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>

v2:
- Re-word commit comment slightly
- Re-enforce idempotency of vmpu_arch_destroy()
- Move invocation of vpmu_destroy() earlier in
domain_relinquish_resources()

---
 xen/arch/x86/cpu/vpmu.c       | 49 +++++++++++++++++++++--------------
 xen/arch/x86/cpu/vpmu_amd.c   |  1 -
 xen/arch/x86/cpu/vpmu_intel.c |  2 --
 xen/arch/x86/domain.c         | 10 ++++---
 4 files changed, 35 insertions(+), 27 deletions(-)
xen/arch/x86/cpu/vpmu.c +29 -20
index f397183ec3..08742a5e22 100644
@@ -479,6 +479,8 @@ static int vpmu_arch_initialise(struct vcpu *v)
 
     if ( ret )
         printk(XENLOG_G_WARNING "VPMU: Initialization failed for %pv\n", v);
+    else
+        vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);
 
     return ret;
 }
@@ -576,11 +578,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }
 
-void vpmu_destroy(struct vcpu *v)
+static void vpmu_cleanup(struct vcpu *v)
 {
+    struct vpmu_struct *vpmu = vcpu_vpmu(v);
+    mfn_t mfn;
+    void *xenpmu_data;
+
+    spin_lock(&vpmu->vpmu_lock);
+
     vpmu_arch_destroy(v);
+    xenpmu_data = vpmu->xenpmu_data;
+    vpmu->xenpmu_data = NULL;
+
+    spin_unlock(&vpmu->vpmu_lock);
+
+    if ( xenpmu_data )
+    {
+        mfn = domain_page_map_to_mfn(xenpmu_data);
+        ASSERT(mfn_valid(mfn));
+        unmap_domain_page_global(xenpmu_data);
+        put_page_and_type(mfn_to_page(mfn));
+    }
+}
+
+void vpmu_destroy(struct vcpu *v)
+{
+    vpmu_cleanup(v);
 
     put_vpmu(v);
 }
@@ -639,9 +666,6 @@ static int pvpmu_init(struct domain *d, xen_pmu_params_t *params)
 static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
 {
     struct vcpu *v;
-    struct vpmu_struct *vpmu;
-    mfn_t mfn;
-    void *xenpmu_data;
 
     if ( (params->vcpu >= d->max_vcpus) || (d->vcpu[params->vcpu] == NULL) )
         return;
@@ -650,22 +674,7 @@ static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
     if ( v != current )
         vcpu_pause(v);
 
-    vpmu = vcpu_vpmu(v);
-    spin_lock(&vpmu->vpmu_lock);
-
-    vpmu_arch_destroy(v);
-    xenpmu_data = vpmu->xenpmu_data;
-    vpmu->xenpmu_data = NULL;
-
-    spin_unlock(&vpmu->vpmu_lock);
-
-    if ( xenpmu_data )
-    {
-        mfn = domain_page_map_to_mfn(xenpmu_data);
-        ASSERT(mfn_valid(mfn));
-        unmap_domain_page_global(xenpmu_data);
-        put_page_and_type(mfn_to_page(mfn));
-    }
+    vpmu_cleanup(v);
 
     if ( v != current )
         vcpu_unpause(v);
xen/arch/x86/cpu/vpmu_amd.c +0 -1
index 3c6799b42c..8ca26f1e3a 100644
@@ -534,7 +534,6 @@ int svm_vpmu_initialise(struct vcpu *v)
 
     vpmu->arch_vpmu_ops = &amd_vpmu_ops;
 
-    vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);
     return 0;
 }
 
xen/arch/x86/cpu/vpmu_intel.c +0 -2
index 6e27f6ec8e..a92d882597 100644
@@ -483,8 +483,6 @@ static int core2_vpmu_alloc_resource(struct vcpu *v)
         memcpy(&vpmu->xenpmu_data->pmu.c.intel, core2_vpmu_cxt, regs_off);
     }
 
-    vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);
-
     return 1;
 
 out_err:
xen/arch/x86/domain.c +6 -4
index f1dd86e12e..f5c0c378ef 100644
@@ -454,9 +454,6 @@ void arch_vcpu_destroy(struct vcpu *v)
     xfree(v->arch.msrs);
     v->arch.msrs = NULL;
 
-    if ( !is_idle_domain(v->domain) )
-        vpmu_destroy(v);
-
     if ( is_hvm_vcpu(v) )
         hvm_vcpu_destroy(v);
     else
@@ -2136,12 +2133,17 @@ int domain_relinquish_resources(struct domain *d)
 
     PROGRESS(vcpu_pagetables):
 
-        /* Drop the in-use references to page-table bases. */
+        /*
+         * Drop the in-use references to page-table bases and clean
+         * up vPMU instances.
+         */
         for_each_vcpu ( d, v )
         {
             ret = vcpu_destroy_pagetables(v);
             if ( ret )
                 return ret;
+
+            vpmu_destroy(v);
         }
 
         if ( altp2m_active(d) )
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 12 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -479,6 +479,8 @@ static int vpmu_arch_initialise(struct vcpu *v)
 
     if ( ret )
         printk(XENLOG_G_WARNING "VPMU: Initialization failed for %pv\n", v);
+    else
+        vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);
 
     return ret;
 }

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 583

Quoted 6 lines
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +578,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris,

I'd like to ask that you comment on this part of the change at least, as I seem to vaguely recall that things were intentionally not done this way originally.

Paul,

everything else looks god to me now.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 9 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static int vpmu_arch_initialise(struct vcpu *v) · line 483

Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -479,6 +479,8 @@ static int vpmu_arch_initialise(struct vcpu *v)
 
     if ( ret )
         printk(XENLOG_G_WARNING "VPMU: Initialization failed for %pv\n", v);
+    else
+        vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);

That won't work I think.

On Intel the context is allocated lazily for HVM/PVH guests during the first MSR access. For example:

core2_vpmu_do_wrmsr() ->
    core2_vpmu_msr_common_check()):
        if ( unlikely(!vpmu_is_set(vpmu, VPMU_CONTEXT_ALLOCATED)) && !core2_vpmu_alloc_resource(current) )
                return 0;

For PV guests the context *is* allocated from vmx_vpmu_initialise().

I don't remember why only PV does eager allocation but I think doing it for all guests would make code much simpler and then this patch will be correct.

-boris

Quoted 16 lines

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 583

Quoted 10 lines
xen/arch/x86/cpu/vpmu.c +2 -0
     return ret;
 }
@@ -576,11 +578,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris,

I'd like to ask that you comment on this part of the change at least, as I seem to vaguely recall that things were intentionally not done this way originally.

Quoted 5 lines

Paul,

everything else looks god to me now.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 34 lines

-----Original Message-----
From: Boris Ostrovsky <[email protected]>
Sent: 27 November 2019 16:32
To: Jan Beulich <[email protected]>; Durrant, Paul <[email protected]>
Cc: Grall, Julien <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Jun Nakajima <[email protected]>; Kevin Tian <[email protected]>; Wei Liu <[email protected]>; [email protected]
Subject: Re: [PATCH v2] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Quoted 7 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static int vpmu_arch_initialise(struct vcpu *v) · line 481

Quoted 6 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +0 -0
@@ -479,6 +479,8 @@ static int vpmu_arch_initialise(struct vcpu *v)
 
     if ( ret )
         printk(XENLOG_G_WARNING "VPMU: Initialization failed for

%pv\n", v);

Quoted 2 lines

xen/arch/x86/cpu/vpmu.c static int vpmu_arch_initialise(struct vcpu *v) · line 483

Quoted 2 lines
xen/arch/x86/cpu/vpmu.c +2 -0
+    else
+        vpmu_set(vpmu, VPMU_CONTEXT_ALLOCATED);

That won't work I think.

On Intel the context is allocated lazily for HVM/PVH guests during the first MSR access. For example:

core2_vpmu_do_wrmsr() ->
    core2_vpmu_msr_common_check()):
        if ( unlikely(!vpmu_is_set(vpmu, VPMU_CONTEXT_ALLOCATED)) && !core2_vpmu_alloc_resource(current) )
                return 0;

Quoted 6 lines

For PV guests the context *is* allocated from vmx_vpmu_initialise().

I don't remember why only PV does eager allocation but I think doing it for all guests would make code much simpler and then this patch will be correct.

Ok. Simpler if I leave setting the flag in the implementation code. I think clearing it in vcpu_arch_destroy() would still be correct in all cases.

Paul

Quoted 24 lines

-boris

Quoted 21 lines

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 583

Quoted 10 lines
xen/arch/x86/cpu/vpmu.c +2 -0
     return ret;
 }
@@ -576,11 +578,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris,

I'd like to ask that you comment on this part of the change at least, as I seem to vaguely recall that things were intentionally not done this way originally.

Paul,

everything else looks god to me now.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
In reply to Paul Durrant

Hi Paul,

Quoted 19 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

Quoted 8 lines

NOTE: The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
Also, whils it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence move manipulation of the VPMU_CONTEXT_ALLOCATED flag out of implementation specific code and make sure it is cleared at the end of vpmu_arch_destroy().

If you resend the patch, it might be worth to add a line about the lack of XSA. Something like:

There is no associated XSA because vPMU is not security supported (see XSA-163).

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 44 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Sent: 27 November 2019 19:42
To: Durrant, Paul <[email protected]>; [email protected]
Cc: Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; Jun Nakajima <[email protected]>; Kevin Tian <[email protected]>
Subject: Re: [PATCH v2] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Hi Paul,

Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in

the

Quoted 7 lines

hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

Quoted 6 lines

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that

the

Quoted 1 line

reference on the mapped page does not prevent domain_destroy()

(which

Quoted 2 lines

calls arch_vcpu_destroy()) from being called.
Also, whils it appears that vpmu_arch_destroy() is idempotent it

is

Quoted 2 lines

by no means obvious. Hence move manipulation of the VPMU_CONTEXT_ALLOCATED flag out of implementation specific code

Quoted 8 lines

and

Quoted 1 line

make sure it is cleared at the end of vpmu_arch_destroy().

If you resend the patch, it might be worth to add a line about the lack of XSA. Something like:

There is no associated XSA because vPMU is not security supported (see XSA-163).

Sure, I'll add another note.

Paul

Quoted 2 lines

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel