Skip to contentSkip to conversation

xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

PATCH In review Paul Durrant 4 participants 10 messages Nov 28, 2019 – Dec 4, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: - The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
- Whilst it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence make sure the VPMU_CONTEXT_ALLOCATED flag is cleared at the end of vpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>

v2:
- Re-word commit comment slightly
- Re-enforce idempotency of vmpu_arch_destroy()
- Move invocation of vpmu_destroy() earlier in
domain_relinquish_resources()

---
 xen/arch/x86/cpu/vpmu.c | 47 +++++++++++++++++++++++------------------
 xen/arch/x86/domain.c   | 10 +++++----
 2 files changed, 33 insertions(+), 24 deletions(-)
xen/arch/x86/cpu/vpmu.c +27 -20
index f397183ec3..792953e7cb 100644
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }
 
-void vpmu_destroy(struct vcpu *v)
+static void vpmu_cleanup(struct vcpu *v)
 {
+    struct vpmu_struct *vpmu = vcpu_vpmu(v);
+    mfn_t mfn;
+    void *xenpmu_data;
+
+    spin_lock(&vpmu->vpmu_lock);
+
     vpmu_arch_destroy(v);
+    xenpmu_data = vpmu->xenpmu_data;
+    vpmu->xenpmu_data = NULL;
+
+    spin_unlock(&vpmu->vpmu_lock);
+
+    if ( xenpmu_data )
+    {
+        mfn = domain_page_map_to_mfn(xenpmu_data);
+        ASSERT(mfn_valid(mfn));
+        unmap_domain_page_global(xenpmu_data);
+        put_page_and_type(mfn_to_page(mfn));
+    }
+}
+
+void vpmu_destroy(struct vcpu *v)
+{
+    vpmu_cleanup(v);
 
     put_vpmu(v);
 }
@@ -639,9 +664,6 @@ static int pvpmu_init(struct domain *d, xen_pmu_params_t *params)
 static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
 {
     struct vcpu *v;
-    struct vpmu_struct *vpmu;
-    mfn_t mfn;
-    void *xenpmu_data;
 
     if ( (params->vcpu >= d->max_vcpus) || (d->vcpu[params->vcpu] == NULL) )
         return;
@@ -650,22 +672,7 @@ static void pvpmu_finish(struct domain *d, xen_pmu_params_t *params)
     if ( v != current )
         vcpu_pause(v);
 
-    vpmu = vcpu_vpmu(v);
-    spin_lock(&vpmu->vpmu_lock);
-
-    vpmu_arch_destroy(v);
-    xenpmu_data = vpmu->xenpmu_data;
-    vpmu->xenpmu_data = NULL;
-
-    spin_unlock(&vpmu->vpmu_lock);
-
-    if ( xenpmu_data )
-    {
-        mfn = domain_page_map_to_mfn(xenpmu_data);
-        ASSERT(mfn_valid(mfn));
-        unmap_domain_page_global(xenpmu_data);
-        put_page_and_type(mfn_to_page(mfn));
-    }
+    vpmu_cleanup(v);
 
     if ( v != current )
         vcpu_unpause(v);
xen/arch/x86/domain.c +6 -4
index f1dd86e12e..f5c0c378ef 100644
@@ -454,9 +454,6 @@ void arch_vcpu_destroy(struct vcpu *v)
     xfree(v->arch.msrs);
     v->arch.msrs = NULL;
 
-    if ( !is_idle_domain(v->domain) )
-        vpmu_destroy(v);
-
     if ( is_hvm_vcpu(v) )
         hvm_vcpu_destroy(v);
     else
@@ -2136,12 +2133,17 @@ int domain_relinquish_resources(struct domain *d)
 
     PROGRESS(vcpu_pagetables):
 
-        /* Drop the in-use references to page-table bases. */
+        /*
+         * Drop the in-use references to page-table bases and clean
+         * up vPMU instances.
+         */
         for_each_vcpu ( d, v )
         {
             ret = vcpu_destroy_pagetables(v);
             if ( ret )
                 return ret;
+
+            vpmu_destroy(v);
         }
 
         if ( altp2m_active(d) )
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 39 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: - The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
- Whilst it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence make sure the VPMU_CONTEXT_ALLOCATED flag is cleared at the end of vpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>

Quoted 5 lines

v2:
- Re-word commit comment slightly
- Re-enforce idempotency of vmpu_arch_destroy()
- Move invocation of vpmu_destroy() earlier in
domain_relinquish_resources()

What about v3?

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581

Quoted 9 lines
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 54 lines

-----Original Message-----
From: Jan Beulich <[email protected]>
Sent: 28 November 2019 10:23
To: Durrant, Paul <[email protected]>; Boris Ostrovsky <[email protected]>
Cc: [email protected]; Grall, Julien <[email protected]>; Andrew Cooper <[email protected]>; Roger Pau Monné <[email protected]>; Jun Nakajima <[email protected]>; Kevin Tian <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH v3] xen/x86: vpmu: Unmap per-vCPU PMU page when the domain is destroyed

Quoted 8 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in

the

Quoted 7 lines

hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

Quoted 7 lines

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: - The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent

domain_destroy()

Quoted 17 lines

(which calls arch_vcpu_destroy()) from being called.
- Whilst it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence make sure the VPMU_CONTEXT_ALLOCATED flag is cleared at the end of vpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).

---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: "Roger Pau Monné" <[email protected]>
Cc: Jun Nakajima <[email protected]>
Cc: Kevin Tian <[email protected]>

Quoted 7 lines
Quoted 5 lines

v2:
- Re-word commit comment slightly
- Re-enforce idempotency of vmpu_arch_destroy()
- Move invocation of vpmu_destroy() earlier in
domain_relinquish_resources()

What about v3?

Oh, sorry:

v3:
- Add comment regarding XSA-163
- Revert changes setting VPMU_CONTEXT_ALLOCATED in common code

Paul

Quoted 14 lines

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581

Quoted 10 lines
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Quoted 1 line

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
In reply to Jan Beulich
Quoted 6 lines · Jan Beulich
Quoted 4 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +0 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
Quoted 7 lines

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581

Quoted 5 lines
xen/arch/x86/cpu/vpmu.c +2 -0
          vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Yes, this is fine.

(I probably would clear it in arch_vpmu_destroy op since it is set in arch-specific code but either way works)

-boris

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Hi Boris,

Quoted 15 lines · Boris Ostrovsky
Quoted 6 lines · Jan Beulich
Quoted 4 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +0 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
 
Quoted 7 lines

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 581

Quoted 5 lines
xen/arch/x86/cpu/vpmu.c +2 -0
           vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
      }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
  }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Quoted 1 line

Yes, this is fine.

Can we take this as an ACK?

Quoted 6 lines

(I probably would clear it in arch_vpmu_destroy op since it is set in arch-specific code but either way works)

-boris

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 20 lines · Julien Grall

Hi Boris,

Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580

Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
            vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
     }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
 }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Quoted 3 lines
Quoted 1 line

Yes, this is fine.

Can we take this as an ACK?

Yes, of course.

-boris

Quoted 6 lines
Quoted 4 lines

(I probably would clear it in arch_vpmu_destroy op since it is set in arch-specific code but either way works)
-boris

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 20 lines · Boris Ostrovsky
Quoted 20 lines · Julien Grall

Hi Boris,

Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580

Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
             vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
      }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
  }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Yes, this is fine.

Can we take this as an ACK?

Yes, of course.

Thank you!

@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw
Quoted 25 lines · Julien Grall
Quoted 22 lines · Boris Ostrovsky
Quoted 22 lines · Julien Grall

Hi Boris,

Quoted 14 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580

Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
             vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
      }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
  }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Yes, this is fine.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Can we take this as an ACK?

Yes, of course.

Thank you!

@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.

It has been committed already.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Julien Grall
Quoted 25 lines · Boris Ostrovsky
Quoted 23 lines · Julien Grall

Hi Boris,

Quoted 13 lines · Boris Ostrovsky
Quoted 12 lines · Jan Beulich

xen/arch/x86/cpu/vpmu.c static void vpmu_arch_destroy(struct vcpu *v) · line 580

Quoted 8 lines · Paul Durrant
xen/arch/x86/cpu/vpmu.c +2 -0
@@ -576,11 +576,36 @@ static void vpmu_arch_destroy(struct vcpu *v)
              vpmu->arch_vpmu_ops->arch_vpmu_destroy(v);
       }
+
+    vpmu_reset(vpmu, VPMU_CONTEXT_ALLOCATED);
   }

Boris, to be on the safe side - are you in agreement with this change, now that the setting of the flag is being left untouched?

Quoted 1 line

Yes, this is fine.

Can we take this as an ACK?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

Yes, of course.

Thank you!

@Andrew, @Jan: this is x86 code, but I am happy to commit it if you prefer.

It has been committed already.

Oh, I didn't spot it in git log. Sorry for the noise.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v3 1/1
#9 lore raw
In reply to Paul Durrant
Quoted 29 lines · Paul Durrant

From: Julien Grall <[email protected]>

A guest will setup a shared page with the hypervisor for each vCPU via XENPMU_init. The page will then get mapped in the hypervisor and only released when XENPMU_finish is called.

This means that if the guest fails to invoke XENPMU_finish, e.g if it is destroyed rather than cleanly shut down, the page will stay mapped in the hypervisor. One of the consequences is the domain can never be fully destroyed as a page reference is still held.

As Xen should never rely on the guest to correctly clean-up any allocation in the hypervisor, we should also unmap such pages during the domain destruction if there are any left.

We can re-use the same logic as in pvpmu_finish(). To avoid duplication, move the logic in a new function that can also be called from vpmu_destroy().

NOTE: - The call to vpmu_destroy() must also be moved from
arch_vcpu_destroy() into domain_relinquish_resources() such that the reference on the mapped page does not prevent domain_destroy() (which calls arch_vcpu_destroy()) from being called.
- Whilst it appears that vpmu_arch_destroy() is idempotent it is by no means obvious. Hence make sure the VPMU_CONTEXT_ALLOCATED flag is cleared at the end of vpmu_arch_destroy().
- This is not an XSA because vPMU is not security supported (see XSA-163).

Quoted 2 lines

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel