Skip to contentSkip to conversation

x86/altp2m: Allow setting the #VE info page for an arbitrary VCPU

PATCH In review Adrian Pop 7 participants 23 messages Sep 4, 2018 – Sep 20, 2018

lore
Share & tools
lore

Adrian Pop

author
#0 lore raw

PATCH v2 1/1 x86/altp2m: Allow setting the #VE info page for an arbitrary VCPU

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +10 -9
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);
 
     if ( d == NULL )
         return -ESRCH;
@@ -4605,26 +4604,28 @@ static int do_altp2m_op(
 
     case HVMOP_altp2m_vcpu_enable_notify:
     {
-        struct vcpu *curr = current;
+        struct vcpu *v;
         p2m_type_t p2mt;
 
-        if ( a.u.enable_notify.pad || a.domain != DOMID_SELF ||
-             a.u.enable_notify.vcpu_id != curr->vcpu_id )
+        if ( a.u.enable_notify.pad ||
+             a.u.enable_notify.vcpu_id >= d->max_vcpus )
         {
             rc = -EINVAL;
             break;
         }
 
-        if ( !gfn_eq(vcpu_altp2m(curr).veinfo_gfn, INVALID_GFN) ||
-             mfn_eq(get_gfn_query_unlocked(curr->domain,
+        v = d->vcpu[a.u.enable_notify.vcpu_id];
+
+        if ( !gfn_eq(vcpu_altp2m(v).veinfo_gfn, INVALID_GFN) ||
+             mfn_eq(get_gfn_query_unlocked(v->domain,
                     a.u.enable_notify.gfn, &p2mt), INVALID_MFN) )
         {
             rc = -EINVAL;
             break;
         }
 
-        vcpu_altp2m(curr).veinfo_gfn = _gfn(a.u.enable_notify.gfn);
-        altp2m_vcpu_update_vmfunc_ve(curr);
+        vcpu_altp2m(v).veinfo_gfn = _gfn(a.u.enable_notify.gfn);
+        altp2m_vcpu_update_vmfunc_ve(v);
         break;
     }
 
Signature · 7 lines
2.18.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 70 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +1 -2
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

Quoted 40 lines
xen/arch/x86/hvm/hvm.c +9 -8
     if ( d == NULL )
         return -ESRCH;
@@ -4605,26 +4604,28 @@ static int do_altp2m_op(
 
     case HVMOP_altp2m_vcpu_enable_notify:
     {
-        struct vcpu *curr = current;
+        struct vcpu *v;
         p2m_type_t p2mt;
 
-        if ( a.u.enable_notify.pad || a.domain != DOMID_SELF ||
-             a.u.enable_notify.vcpu_id != curr->vcpu_id )
+        if ( a.u.enable_notify.pad ||
+             a.u.enable_notify.vcpu_id >= d->max_vcpus )
         {
             rc = -EINVAL;
             break;
         }
 
-        if ( !gfn_eq(vcpu_altp2m(curr).veinfo_gfn, INVALID_GFN) ||
-             mfn_eq(get_gfn_query_unlocked(curr->domain,
+        v = d->vcpu[a.u.enable_notify.vcpu_id];
+
+        if ( !gfn_eq(vcpu_altp2m(v).veinfo_gfn, INVALID_GFN) ||
+             mfn_eq(get_gfn_query_unlocked(v->domain,
                     a.u.enable_notify.gfn, &p2mt), INVALID_MFN) )
         {
             rc = -EINVAL;
             break;
         }
 
-        vcpu_altp2m(curr).veinfo_gfn = _gfn(a.u.enable_notify.gfn);
-        altp2m_vcpu_update_vmfunc_ve(curr);
+        vcpu_altp2m(v).veinfo_gfn = _gfn(a.u.enable_notify.gfn);
+        altp2m_vcpu_update_vmfunc_ve(v);
         break;
     }
 
--
Quoted 1 line

2.18.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 74 lines · Tamas K Lengyel
Quoted 73 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 93 lines
Quoted 1 line
Quoted 1 line · On 04.09.18 at 22:58, <[email protected]> wrote:
Quoted 54 lines · Tamas K Lengyel
Quoted 53 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
Quoted 5 lines

tools/libxc/xc_altp2m.c int xc_altp2m_set_domain_state(xc_interface *handle,

Quoted 5 lines
tools/libxc/xc_altp2m.c +0 -0
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, 

uint32_t dom, bool state)

Quoted 15 lines
Quoted 15 lines
tools/libxc/xc_altp2m.c +0 -1
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

But a guest exposed interface can't be changed like this: If a.domain was ignored for this sub-op before, it needs to continue to be ignored.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 93 lines · Jan Beulich
Quoted 93 lines
Quoted 1 line
Quoted 1 line · On 04.09.18 at 22:58, <[email protected]> wrote:
Quoted 54 lines · Tamas K Lengyel
Quoted 53 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
Quoted 5 lines

tools/libxc/xc_altp2m.c int xc_altp2m_set_domain_state(xc_interface *handle,

Quoted 5 lines
tools/libxc/xc_altp2m.c +0 -0
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, 

uint32_t dom, bool state)

Quoted 15 lines
Quoted 15 lines
tools/libxc/xc_altp2m.c +0 -1
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 7 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

But a guest exposed interface can't be changed like this: If a.domain was ignored for this sub-op before, it needs to continue to be ignored.

The point of this patch is twofold (as hopefully explained, or at least implied, by the use-case in the description): one is to now allow setting the #VE information for any VCPU, not just current; the second, to be able to make the call from any sufficiently privileged domain (so as to allow the introspection agent to control behaviour).

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
In reply to Jan Beulich
Quoted 93 lines · Jan Beulich
Quoted 93 lines
Quoted 1 line
Quoted 1 line · On 04.09.18 at 22:58, <[email protected]> wrote:
Quoted 54 lines · Tamas K Lengyel
Quoted 53 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
Quoted 5 lines

tools/libxc/xc_altp2m.c int xc_altp2m_set_domain_state(xc_interface *handle,

Quoted 5 lines
tools/libxc/xc_altp2m.c +0 -0
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, 

uint32_t dom, bool state)

Quoted 15 lines
Quoted 15 lines
tools/libxc/xc_altp2m.c +0 -1
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 7 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

But a guest exposed interface can't be changed like this: If a.domain was ignored for this sub-op before, it needs to continue to be ignored.

Also, technically speaking a.domain is not currently ignored - it's just checked against DOMID_SELF:

4596 case HVMOP_altp2m_vcpu_enable_notify:
4597 {
4598 struct vcpu *curr = current;
4599 p2m_type_t p2mt;
4600
4601 if ( a.u.enable_notify.pad || a.domain != DOMID_SELF ||
4602 a.u.enable_notify.vcpu_id != curr->vcpu_id )
4603 {
4604 rc = -EINVAL;
4605 break;
4606 }
4607
4608 if ( !gfn_eq(vcpu_altp2m(curr).veinfo_gfn, INVALID_GFN) ||
4609 mfn_eq(get_gfn_query_unlocked(curr->domain,
4610 a.u.enable_notify.gfn, &p2mt), INVALID_MFN) )
4611 {
4612 rc = -EINVAL;
4613 break;
4614 }
4615

4616         vcpu_altp2m(curr).veinfo_gfn = _gfn(a.u.enable_notify.gfn);
4617         altp2m_vcpu_update_vmfunc_ve(curr);
4618         break;

4619 }
4620

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 38 lines
Quoted 1 line
Quoted 1 line · On 05.09.18 at 10:14, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 04.09.18 at 22:58, <[email protected]> wrote:
Quoted 7 lines · Tamas K Lengyel
Quoted 6 lines · Adrian Pop
xen/arch/x86/hvm/hvm.c +0 -0
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 7 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

But a guest exposed interface can't be changed like this: If a.domain was ignored for this sub-op before, it needs to continue to be ignored.

Also, technically speaking a.domain is not currently ignored - it's just checked against DOMID_SELF:

Quoted 11 lines

4596 case HVMOP_altp2m_vcpu_enable_notify:
4597 {
4598 struct vcpu *curr = current;
4599 p2m_type_t p2mt;
4600
4601 if ( a.u.enable_notify.pad || a.domain != DOMID_SELF ||
4602 a.u.enable_notify.vcpu_id != curr->vcpu_id )
4603 {
4604 rc = -EINVAL;
4605 break;
4606 }

Ah, yes, I think that's fine then.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw
In reply to Razvan Cojocaru
Quoted 92 lines · Razvan Cojocaru
Quoted 74 lines · Tamas K Lengyel
Quoted 73 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Sounds good, thanks!

Tamas

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
Quoted 94 lines · Tamas K Lengyel
Quoted 92 lines · Razvan Cojocaru
Quoted 74 lines · Tamas K Lengyel
Quoted 73 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 6 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw
Quoted 101 lines · Razvan Cojocaru
Quoted 99 lines · Tamas K Lengyel
Quoted 96 lines · Razvan Cojocaru
Quoted 74 lines · Tamas K Lengyel
Quoted 73 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Quoted 4 lines
Quoted 1 line

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

Tamas

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw
Quoted 98 lines · Tamas K Lengyel
Quoted 96 lines · Razvan Cojocaru
Quoted 95 lines · Tamas K Lengyel
Quoted 93 lines · Razvan Cojocaru
Quoted 73 lines · Tamas K Lengyel
Quoted 72 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

That doesn't prohibit you providing a Reviewed-by: tag :)

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by
#11 lore raw
Quoted 103 lines · Andrew Cooper
Quoted 101 lines · Tamas K Lengyel
Quoted 99 lines · Razvan Cojocaru
Quoted 91 lines · Tamas K Lengyel
Quoted 89 lines · Razvan Cojocaru
Quoted 73 lines · Tamas K Lengyel
Quoted 72 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 5 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

Quoted 7 lines
Quoted 2 lines

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

That doesn't prohibit you providing a Reviewed-by: tag :)

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

Indeed :)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw
Quoted 107 lines · Tamas K Lengyel
Quoted 106 lines · Andrew Cooper
Quoted 103 lines · Tamas K Lengyel
Quoted 92 lines · Razvan Cojocaru
Quoted 91 lines · Tamas K Lengyel
Quoted 89 lines · Razvan Cojocaru
Quoted 73 lines · Tamas K Lengyel
Quoted 72 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

Quoted 8 lines
Quoted 4 lines

That doesn't prohibit you providing a Reviewed-by: tag :)

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

Indeed :)

Are there any issues preventing this patch to go in? Possibly missing acks?

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by
#13 lore raw
Quoted 110 lines · Razvan Cojocaru
Quoted 109 lines · Tamas K Lengyel
Quoted 108 lines · Andrew Cooper
Quoted 98 lines · Tamas K Lengyel
Quoted 96 lines · Razvan Cojocaru
Quoted 95 lines · Tamas K Lengyel
Quoted 93 lines · Razvan Cojocaru
Quoted 73 lines · Tamas K Lengyel
Quoted 72 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
tools/libxc/xc_altp2m.c +0 -1
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, uint32_t dom, bool state)
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

That doesn't prohibit you providing a Reviewed-by: tag :)

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

Indeed :)

Are there any issues preventing this patch to go in? Possibly missing acks?

I don't think so, unless an explicit ack for the deletion of that comment is required. In any case:

I will commit this patch shortly.

Wei.

Quoted 4 lines

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw
Quoted 8 lines · Wei Liu
Quoted 2 lines
Quoted 1 line

Are there any issues preventing this patch to go in? Possibly missing acks?

I don't think so, unless an explicit ack for the deletion of that comment is required. In any case:

I will commit this patch shortly.

Thanks!

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw
In reply to Razvan Cojocaru
Quoted 113 lines
Quoted 1 line
Quoted 1 line · On 20.09.18 at 10:11, <[email protected]> wrote:
Quoted 1 line · Tamas K Lengyel

On Wed, Sep 5, 2018 at 12:45 PM Andrew Cooper <[email protected]>

Quoted 61 lines · wrote:
Quoted 61 lines · Tamas K Lengyel
Quoted 59 lines
Quoted 57 lines · Razvan Cojocaru
Quoted 56 lines · Tamas K Lengyel
Quoted 54 lines · Razvan Cojocaru
Quoted 53 lines · Tamas K Lengyel
Quoted 52 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

tools/libxc/xc_altp2m.c int xc_altp2m_set_domain_state(xc_interface *handle,

Quoted 5 lines
tools/libxc/xc_altp2m.c +0 -0
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, 

uint32_t dom, bool state)

Quoted 42 lines
Quoted 42 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Quoted 15 lines
Quoted 15 lines
tools/libxc/xc_altp2m.c +0 -1
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

That doesn't prohibit you providing a Reviewed-by: tag :)

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

Indeed :)

Are there any issues preventing this patch to go in? Possibly missing acks?

Well, afaics the patch has no x86 maintainer ack, nor - considering it's an mm function sitting in the "wrong" file, at least one from the mm maintainer. As mentioned a number of times before, it is the submitter's responsibility to chase acks, not the committers' or maintainers'.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw
Quoted 118 lines · Jan Beulich
Quoted 118 lines
Quoted 1 line
Quoted 1 line · On 20.09.18 at 10:11, <[email protected]> wrote:
Quoted 1 line · Tamas K Lengyel

On Wed, Sep 5, 2018 at 12:45 PM Andrew Cooper <[email protected]>

Quoted 61 lines · wrote:
Quoted 61 lines · Tamas K Lengyel
Quoted 59 lines
Quoted 57 lines · Razvan Cojocaru
Quoted 56 lines · Tamas K Lengyel
Quoted 54 lines · Razvan Cojocaru
Quoted 53 lines · Tamas K Lengyel
Quoted 52 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

---
Changes in v2:
- remove the "__get_vcpu()" helper

---
 tools/libxc/xc_altp2m.c |  1 -
 xen/arch/x86/hvm/hvm.c  | 19 ++++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

tools/libxc/xc_altp2m.c int xc_altp2m_set_domain_state(xc_interface *handle,

Quoted 5 lines
tools/libxc/xc_altp2m.c +0 -0
index ce4a1e4d60..528e929d7a 100644
@@ -68,7 +68,6 @@ int xc_altp2m_set_domain_state(xc_interface *handle, 

uint32_t dom, bool state)

Quoted 47 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 30 lines
Quoted 30 lines
Quoted 15 lines
Quoted 15 lines
tools/libxc/xc_altp2m.c +0 -1
     return rc;
 }
 
-/* This is a bit odd to me that it acts on current.. */
 int xc_altp2m_set_vcpu_enable_notify(xc_interface *handle, uint32_t domid,
                                      uint32_t vcpuid, xen_pfn_t gfn)
 {
xen/arch/x86/hvm/hvm.c +0 -0
index 72c51faecb..49c3bbee94 100644
@@ -4533,8 +4533,7 @@ static int do_altp2m_op(
         return -EOPNOTSUPP;
     }
 
Quoted 6 lines

xen/arch/x86/hvm/hvm.c static int do_altp2m_op( · line 4536

Quoted 3 lines
xen/arch/x86/hvm/hvm.c +1 -2
-    d = ( a.cmd != HVMOP_altp2m_vcpu_enable_notify ) ?
-        rcu_lock_domain_by_any_id(a.domain) : rcu_lock_current_domain();
+    d = rcu_lock_domain_by_any_id(a.domain);

Does rcu_lock_domain_by_any_id work if its from the current domain? If not, doesn't that change this function's accessibility to be from exclusively usable only by the outside agent?

The code says it should be safe:

633 struct domain *rcu_lock_domain_by_any_id(domid_t dom) 634 {
635 if ( dom == DOMID_SELF )
636 return rcu_lock_current_domain();
637 return rcu_lock_domain_by_id(dom);
638 }

Quoted 5 lines
Quoted 4 lines

as long as dom == DOMID_SELF. I think the old behaviour assumed that HVMOP_altp2m_vcpu_enable_notify alone would only ever be used from the current domain, and this change expands its usability (Adrian should correct me if I'm wrong here).

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

Quoted 7 lines
Quoted 2 lines

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

That doesn't prohibit you providing a Reviewed-by: tag :)

The statement itself is useful and hold weight, even if it isn't in code you are a maintainer of.

Indeed :)

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Are there any issues preventing this patch to go in? Possibly missing acks?

Well, afaics the patch has no x86 maintainer ack, nor - considering it's an mm function sitting in the "wrong" file, at least one from the mm maintainer. As mentioned a number of times before, it is the submitter's responsibility to chase acks, not the committers' or maintainers'.

Oh, sorry. I have missed that this at least needs an ack from George. I will wait until EOD for George to give an ack. If there isn't one by then I will revert the patch in staging.

Wei.

Quoted 3 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw
Quoted 10 lines · Wei Liu
Quoted 6 lines
Quoted 2 lines
Quoted 1 line

Are there any issues preventing this patch to go in? Possibly missing acks?

Well, afaics the patch has no x86 maintainer ack, nor - considering it's an mm function sitting in the "wrong" file, at least one from the mm maintainer. As mentioned a number of times before, it is the submitter's responsibility to chase acks, not the committers' or maintainers'.

Oh, sorry. I have missed that this at least needs an ack from George. I will wait until EOD for George to give an ack. If there isn't one by then I will revert the patch in staging.

Added George to the email recipients.

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw
In reply to Razvan Cojocaru
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 20.09.18 at 10:11, <[email protected]> wrote:

Are there any issues preventing this patch to go in? Possibly missing acks?

Oh, and btw - irrespective what MAINTAINERS says - patches to this function would better also be Cc-ed to George as the mm maintainer. Even more so that I think it has become clear from past discussions that I'm not going to ack any patches to this function, the existence / placement of which I question in the first place. It's just that, with recent clarifications, I won't object to any such additions anymore.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw
In reply to Tamas K Lengyel
Quoted 5 lines · Tamas K Lengyel
Quoted 3 lines
Quoted 1 line

Sounds good, thanks!

May we take that as an Acked-by, or are there are other things you think we should address?

A Reviewed-by would be appropriate, I don't think the files touched in this patch fall under our umbrella.

You're right, I just saw the Cc but didn't check.

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#20 lore raw
In reply to Adrian Pop
Quoted 44 lines · Adrian Pop

In a classic HVI + Xen setup, the introspection engine would monitor legacy guest page-tables by marking them read-only inside the EPT; this way any modification explicitly made by the guest or implicitly made by the CPU page walker would trigger an EPT violation, which would be forwarded by Xen to the SVA and thus the HVI agent. The HVI agent would analyse the modification, and act upon it - for example, a virtual page may be remapped (its guest physical address changed inside the page-table), in which case the introspection logic would update the protection accordingly (remove EPT hook on the old gpa, and place a new EPT hook on the new gpa). In other cases, the modification may be of no interest to the introspection engine - for example, the accessed/dirty bits may be cleared by the operating system or the accessed/dirty bits may be set by the CPU page walker.

In our tests we discovered that the vast majority of guest page-table modifications fall in the second category (especially on Windows 10 RS4 x64 - more than 95% of ALL the page-table modifications are irrelevant to us) - they are of no interest to the introspection logic, but they trigger a very costly EPT violation nonetheless. Therefore, we decided to make use of the new #VE & VMFUNC features in recent Intel CPUs to accelerate the guest page-tables monitoring in the following way:

1. Each monitored page-table would be flagged as being convertible inside the EPT, thus enabling the CPU to deliver a virtualization exception to he guest instead of generating a traditional EPT violation.
2. We inject a small filtering driver inside the protected guest VM, which would intercept the virtualization exception in order to handle guest page-table modifications.
3. We create a dedicated EPT view (altp2m) for the in-guest agent, which would isolate the agent from the rest of the operating system; the agent will switch in and out of the protected EPT view via the VMFUNC instruction placed inside a trampoline page, thus making the agent immune to malicious code inside the guest.

This way, all the page-table accesses would generate a virtualization-exception inside the guest instead of a costly EPT violation; the #VE agent would emulate and analyse the modification, and decide whether it is relevant for the main introspection logic; if it is relevant, it would do a VMCALL and notify the introspection engine about the modification; otherwise, it would resume normal instruction execution, thus avoiding a very costly VM exit.

Quoted 1 line

I don't have any objections to the code; I think it can stay in the tree as far as I'm concerned.

I do have a comment on the commit message, just for future reference at this point. It goes into a lot of detail about the architecture of what you're trying to accomplish, but not what this patch actually does. I think something like "Allow a dom0 agent to enable vcpu_notify on guest vcpus, to enable an out-of-guest introspection agent to insert an in-guest agent into a guest" would have been enough.

I do have a question about your proposed use case. You're running this in 'mixed' mode, right, and using the altp2m to hide a secure bit of code from the operating system? What's to stop a rogue operating system that doesn't want to be introspected from calling HVMOP_altp2m_vcpu_enable_notify with INVALID_GFN to disable this?

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw
Quoted 5 lines · George Dunlap

I do have a question about your proposed use case. You're running this in 'mixed' mode, right, and using the altp2m to hide a secure bit of code from the operating system? What's to stop a rogue operating system that doesn't want to be introspected from calling HVMOP_altp2m_vcpu_enable_notify with INVALID_GFN to disable this?

Nothing, but we're not running this in mixed mode. :) We're after 'external', for the very same reasons you've mentioned.

Everything important is done in dom0-only. If there's something to be done that the in-guest agent would like, it has to ask the introspection agent in dom0 via VMCALL events.

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#22 lore raw
Quoted 8 lines · Razvan Cojocaru
Quoted 5 lines · George Dunlap

I do have a question about your proposed use case. You're running this in 'mixed' mode, right, and using the altp2m to hide a secure bit of code from the operating system? What's to stop a rogue operating system that doesn't want to be introspected from calling HVMOP_altp2m_vcpu_enable_notify with INVALID_GFN to disable this?

Quoted 6 lines

Nothing, but we're not running this in mixed mode. :) We're after 'external', for the very same reasons you've mentioned.

Everything important is done in dom0-only. If there's something to be done that the in-guest agent would like, it has to ask the introspection agent in dom0 via VMCALL events.

OK, got it, thanks.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel