Intel EPT-Based Sub-page Write Protection Support.

RFC PATCH In review Zhang Yi 3 participants 32 messages Oct 19, 2017 – Oct 25, 2017

lore

Share & tools

lore

Zhang Yi

author

2017-10-19 08:04 UTC #0 lore raw

cover Intel EPT-Based Sub-page Write Protection Support.

From: Zhang Yi Z <[email protected]>

Hi All,

Here is a patch-series which adding EPT-Based Sub-page Write Protection Support. You can get It's software developer manuals from:

https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

In Chapter 4 EPT-BASED SUB-PAGE PERMISSIONS.

Introduction:

EPT-Based Sub-page Write Protection referred to as SPP, it is a capability which allow Virtual Machine Monitors(VMM) to specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When this capability is utilized, the CPU enforces write-access permissions for sub-page regions of 4K pages as specified by the VMM. EPT-based sub-page permissions is intended to enable fine-grained memory write enforcement by a VMM for security(guest OS monitoring) and usages such as device virtualization and memory check-point.

How SPP Works:

SPP is active when the "sub-page write protection" VM-execution control is 1. A new 4-level paging structure named SPP page table(SPPT) is introduced, SPPT will look up the guest physical addresses to derive a 64 bit "sub-page permission" value containing sub-page write permissions. The lookup from guest-physical addresses to the sub-page region permissions is determined by a set of this SPPT paging structures.

The SPPT is used to lookup write permission bits for the 128 byte sub-page regions containing in the 4KB guest physical page. EPT specifies the 4KB page level privileges that software is allowed when accessing the guest physical address, whereas SPPT defines the write permissions for software at the 128 byte granularity regions within a 4KB page. Write accesses prevented due to sub-page permissions looked up via SPPT are reported as EPT violation VM exits. Similar to EPT, a logical processor uses SPPT to lookup sub-page region write permissions for guest-physical addresses only when those addresses are used to access memory.

Guest write access --> GPA --> Walk EPT --> EPT leaf entry -┐
┌-----------------------------------------------------------┘
└-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61)
     |
     └-> <false> --> EPT legacy behavior
     |
     |
     └-> <true>  --> if ept_leaf_entry.writable
                      |
                      └-> <true>  --> Ignore SPP
                      |
		      └-> <false> --> GPA --> Walk SPP 4-level table--┐
                                                                      |
┌------------<----------get-the-SPPT-point-from-VMCS-filed-----<------┘
|
Walk SPP L4E table
|
└┐--> entry misconfiguration ------------>----------┐<----------------┐
 |                                                  |                 |
else                                                |                 |
 |                                                  |                 |
 |   ┌------------------SPP VMexit<-----------------┘                 |
 |   |                                                                |
 |   └-> exit_qualification & sppt_misconfig --> sppt misconfig       |
 |   |                                                                |
 |   └-> exit_qualification & sppt_miss --> sppt miss                 |
 └--┐                                                                 |
    |                                                                 |
walk SPPT L3E--┐--> if-entry-misconfiguration------------>------------┘
               |                                                      |
	      else                                                    |
	       |                                                      |
	       |                                                      |
        walk SPPT L2E --┐--> if-entry-misconfiguration-------->-------┘
                        |                                             |
                       else                                           |
			|                                             |
			|                                             |
	         walk SPPT L1E --┐-> if-entry-misconfiguration--->----┘
                                 |
			        else
				 |
                                 └-> if sub-page writable
                                      └-> <true>  allow, write access
	                              └-> <false> disallow, EPT violation

Patch-sets Description:

Patch 1: Documentations.

Patch 2: this patch adds reporting SPP capability from VMX Procbased MSR, according to the definition of hardware spec, bit 23 is the control of the SPP capability.

Patch 3: Add new secondary processor-based VM-execution control bit which defined as "sub-page write permission", same as VMX Procbased MSR, bit 23 is the enable bit of SPP. Also we introduced a kernel parameter "enable_ept_spp", now SPP is active when the "Sub-page Write Protection" in Secondary VM-Execution Control is set and enable the kernel parameter by "enable_ept_spp=1".

Patch 4: Introduced the spptp and spp page table.
The sub-page permission table is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address. The index and encoding for this VMCS field if defined 0x2030 at this time. The format of SPPTP is shown in below figure:

---------------------------------------------------------------|
| Bit    | Contents                                            |
:--------------------------------------------------------------|
| 11:0   | Reserved (0)                                        |
| N-1:12 | Physical address of 4KB aligned SPPT L4E Table      |
| 51:N   | Reserved (0)                                        |
| 63:52  | Reserved (0)                                        |
---------------------------------------------------------------|

This patch also introduced the Spp paging structures, which root page will created at p2m_alloc_table.

Patch 5: Introduced the SPP-Induced VM exit and it's handle.
Accesses using guest-physical addresses may cause SPP-induced VM exits due to an SPPT misconfiguration or an SPPT miss. The basic VM exit reason code reported for SPP-induced VM exits is 66.

Also Introduced the new exit qualification for SPPT-induced vmexits.

| Bit   | Contents                                                          |
| :---- | :---------------------------------------------------------------- |
| 10:0  | Reserved (0).                                                     |
| 11    | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. |
| 12    | NMI unblocking due to IRET                                        |
| 63:13 | Reserved (0)                                                      |

Patch 7 ~ Patch 8: Setup spp page table and update the EPT leaf entry indicated with the SPP enable bit.
If the sub-page write permission VM-execution control is set, treatment of write accesses to guest-physical accesses depends on the state of the accumulated write-access bit (position 1) and sub-page permission bit (position 61) in the EPT leaf paging-structure.
Software will update the EPT leaf entry sub-page permission bit while xen_set_subpage(patch 7). If the EPT write-access bit set to 0 and the SPP bit set to 1 in the leaf EPT paging-structure entry that maps a 4KB page, then the hardware will look up a VMM-managed Sub-Page Permission Table (SPPT), which will be prepared by setup xen_set_subpage(patch 8). The hardware uses the guest-physical address and bits 11:7 of the address accessed to lookup the SPPT to fetch a write permission bit for the 128 byte wide sub-page region being accessed within the 4K guest-physical page. If the sub-page region write permission bit is set, the write is allowed, otherwise the write is disallowed and results in an EPT violation. Guest-physical pages mapped via leaf EPT-paging-structures for which the accumulated write-access bit and the SPP bits are both clear (0) generate EPT violations on memory writes accesses. Guest-physical pages mapped via EPT-paging-structure for which the accumulated write-access bit is set (1) allow writes, effectively ignoring the SPP bit on the leaf EPT-paging structure. Software will setup the spp page table level4,3,2 as well as EPT page structure, and fill the level1 page via the 32 bit bitmaps per a single 4K page. Now it could be divided to 32 x 128 sub-pages.

The SPP L4E L3E L2E is defined as below figure.

| Bit    | Contents                                                               |
| :----- | :--------------------------------------------------------------------- |
| 0      | Valid entry when set; indicates whether the entry is present           |
| 11:1   | Reserved (0)                                                           |
| N-1:12 | Physical address of 4K aligned SPPT LX-1 Table referenced by the entry |
| 51:N   | Reserved (0)                                                           |
| 63:52  | Reserved (0)                                                           |

Note: N is the physical address width supported by the processor, X is the page level

The SPP L1E format is defined as below figure.

| Bit   | Contents                                                          |
| :---- | :---------------------------------------------------------------- |
| 0+2i  | Write permission for i-th 128 byte sub-page region.               |
| 1+2i  | Reserved (0).                                                     |

Note: 0<=i<=31

Patch 9 ~ Patch 10: Introduced a hyper call to set subpage protection.

The new Xen Hypercall is HVMOP_set_subpage.
The Interface's parameters is defined as

    struct xen_hvm_subpage {
        domid_t  domid;
	uint32_t access_map;
	uint64_t gfn;
    }

Patch 11: Added a handle of EPT subpage write protection fault.
A control bit in EPT leaf paging-structure entries is defined as “Sub-Page Permission” (SPP bit). The bit position is 61; it is chosen from among the bits that are currently ignored by the processor and available to software. While hardware walking the SPP page table, If the sub-page region write permission bit is set, the write is allowed, else the write is disallowed and results an EPT violation.
We need peek this case in EPT violation handler.

Patch 13 ~ Patch 14: Implement a user tool to set subpage protection.
It is a tool could set a 4K page corresponding a 32 bit bitmap: xen-subpage -m [domid] set [gfn] [bitmap]

Zhang Yi Z (14):
xen: vmx: Added EPT based Subpage Write Protection Doc.
xen: vmx: Added VMX SPP feature flags and VM-Execution Controls.
xen: vmx: Introduce the SPPTP and SPP page table.
xen: vmx: Introduce SPP-Induced vm exit and it's handle.
xen: vmx: Disable the 2M/1G superpage when SPP enabled
xen: vmx: Added SPP flags in EPT leaf entry.
xen: vmx: Update the EPT leaf entry indicated with the SPP enable bit.
xen: vmx: Added setup spp page structure.
xen: vmx: Introduce a Hyper call to set subpage
xen: vmx: Implement the Hypercall p2m_set_subpage
xen: vmx: Added handle of SPP write protection fault
xen: vmx: Support for clear EPT SPP write Protect bit
xen: tools: Introduce the set-subpage into xenctrl
xen: tools: Added xen-subpage tool.

 docs/txt/misc/spp_xen.txt             | 259 ++++++++++++++++++++++++++++++++++
 tools/libxc/include/xenctrl.h         |   2 +
 tools/libxc/xc_mem_paging.c           |  20 +++
 tools/tests/xen-subpage/Makefile      |  30 ++++
 tools/tests/xen-subpage/xen-subpage.c | 125 ++++++++++++++++
 xen/arch/x86/hvm/hvm.c                |  21 +++
 xen/arch/x86/hvm/vmx/vmcs.c           |  20 +++
 xen/arch/x86/hvm/vmx/vmx.c            |  40 ++++++
 xen/arch/x86/mm/mem_access.c          |  80 +++++++++++
 xen/arch/x86/mm/p2m-ept.c             | 152 ++++++++++++++++++++
 xen/arch/x86/mm/p2m.c                 |  12 +-
 xen/include/asm-x86/hvm/hvm.h         |   2 +
 xen/include/asm-x86/hvm/vmx/vmcs.h    |  14 ++
 xen/include/asm-x86/hvm/vmx/vmx.h     |  26 +++-
 xen/include/asm-x86/p2m.h             |  13 +-
 xen/include/public/hvm/hvm_op.h       |   9 ++
 xen/include/xen/mem_access.h          |   3 +
 17 files changed, 824 insertions(+), 4 deletions(-)
 create mode 100644 docs/txt/misc/spp_xen.txt
 create mode 100644 tools/tests/xen-subpage/Makefile
 create mode 100644 tools/tests/xen-subpage/xen-subpage.c

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:07 UTC #1 lore raw

RFC PATCH 1/14 xen: vmx: Added EPT based Subpage Write Protection Doc.

From: Zhang Yi Z <[email protected]>

Signed-off-by Zhang Yi Z <[email protected]>

---
 docs/txt/misc/spp_xen.txt | 259 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 259 insertions(+)
 create mode 100644 docs/txt/misc/spp_xen.txt

docs/txt/misc/spp_xen.txt new file +259 -0

new file mode 100644
index 0000000..d84eca2
@@ -0,0 +1,259 @@
+DRAFT: EPT-Based Sub-Page Protection (SPP) Design Doc for Xen
+=============================================================
+
+1. Overview
+
+EPT-based Sub-Page Protection (SPP) capability to allow Virtual Machine
+Monitors to specify write-protection for guest physical memory at a
+sub-page (128 byte) granularity. When this capability is utilized, the
+CPU enforces write-access permissions for sub-page regions of 4K pages
+as specified by the VMM.
+
+2. Operation of SPP
+
+Sub-Page Protection Table (SPPT) is introduced to manage sub-page
+write-access.
+
+SPPT is active when the "sub-page write protection" VM-execution control
+is 1. SPPT looks up the guest physical addresses to derive a 64 bit
+"sub-page permission" value containing sub-page write permissions. The
+lookup from guest-physical addresses to the sub-page region permissions
+is determined by a set of SPPT paging structures.
+
+When the "sub-page write protection" VM-execution control is 1, the SPPT
+is used to lookup write permission bits for the 128 byte sub-page regions
+containing in the 4KB guest physical page. EPT specifies the 4KB page
+level privileges that software is allowed when accessing the guest
+physical address, whereas SPPT defines the write permissions for software
+at the 128 byte granularity regions within a 4KB page. Write accesses
+prevented due to sub-page permissions looked up via SPPT are reported as
+EPT violation VM exits. Similar to EPT, a logical processor uses SPPT to
+lookup sub-page region write permissions for guest-physical addresses
+only when those addresses are used to access memory.
+______________________________________________________________________________
+
+How SPP hardware works:
+_______________________________________________________________________________
+
+Guest write access --> GPA --> Walk EPT --> EPT leaf entry -┐
+┌-----------------------------------------------------------┘
+└-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61)
+     |
+     └-> <false> --> EPT legacy behavior
+     |
+     |
+     └-> <true>  --> if ept_leaf_entry.writable
+                      |
+                      └-> <true>  --> Ignore SPP
+                      |
+		      └-> <false> --> GPA --> Walk SPP 4-level table--┐
+                                                                      |
+┌------------<----------get-the-SPPT-point-from-VMCS-filed-----<------┘
+|
+Walk SPP L4E table
+|
+└┐--> entry misconfiguration ------------>----------┐<----------------┐
+ |                                                  |                 |
+else                                                |                 |
+ |                                                  |                 |
+ |   ┌------------------SPP VMexit<-----------------┘                 |
+ |   |                                                                |
+ |   └-> exit_qualification & sppt_misconfig --> sppt misconfig       |
+ |   |                                                                |
+ |   └-> exit_qualification & sppt_miss --> sppt miss                 |
+ └--┐                                                                 |
+    |                                                                 |
+walk SPPT L3E--┐--> if-entry-misconfiguration------------>------------┘
+               |                                                      |
+	      else                                                    |
+	       |                                                      |
+	       |                                                      |
+        walk SPPT L2E --┐--> if-entry-misconfiguration-------->-------┘
+                        |                                             |
+                       else                                           |
+			|                                             |
+			|                                             |
+	         walk SPPT L1E --┐-> if-entry-misconfiguration--->----┘
+                                 |
+			        else
+				 |
+                                 └-> if sub-page writable
+                                      └-> <true>  allow, write access
+	                              └-> <false> disallow, EPT violation
+______________________________________________________________________________
+
+3. Interfaces
+
+* Feature enabling
+
+Add "spp_enable=1" to Xen Command line to enable SPP feature, default is off.
+
+* Get/Set sub-page write access permission
+
+New Xen HVM Hyper Call:
+
+`HVMOP_set_subpage`:
+Set sub-pages write access bitmap corresponding to given gfn.
+
+```c
+/* for Xen HVMOP_set_subpage */
+struct xen_hvm_subpage_t {
+	domid domid;
+	__u64 gfn;
+	__u32 access_map; /* sub-page write-access bitmap */
+};
+
+#define HVMOP_set_subpage          26
+
+xencall2(handle->xcall, __HYPERVISOR_hvm_op, HVMOP_set_subpage,
+         HYPERCALL_BUFFER_AS_ARG(arg))
+```
+
+4. SPPT initialization
+
+* SPPT root page allocation
+
+  SPPT is referenced via a 64-bit control field called "sub-page
+  protection table pointe" (SPPTP, encoding 0x2030) which contains a
+  4K-align physical address.
+
+  SPPT also has 4 level table as well as EPT. So, as EPT does, when Xen
+  loads mmu, we allocate a root page for SPPT L4 table.
+
+* EPT leaf entry SPP bit
+
+  Set 0 to SPP bit to close SPP by default.
+
+5. Set/Get Sub-Page access bitmap for bunch of guest physical pages
+
+* To utilize SPP feature, system admin should Set a Sub-page access write via
+  SPP Xen hyper call `HVMOP_set_subpage`, which will prepared the flowing things.
+
+   (1.Got the corresponding EPT leaf entry via the guest physical address.
+   (2.If it is a 4K page frame, flag the bit 61 to enable subpage protection on this page.
+   (3.Setup spp page structure, the page structure format is list following.
+
+   Format of the SPPT L4E, L3E, L2E:
+   | Bit    | Contents                                                                 |
+   | :----- | :------------------------------------------------------------------------|
+   | 0      | Valid entry when set; indicates whether the entry is present             |
+   | 11:1   | Reserved (0)                                                             |
+   | N-1:12 | Physical address of 4KB aligned SPPT LX-1 Table referenced by this entry |
+   | 51:N   | Reserved (0)                                                             |
+   | 63:52  | Reserved (0)                                                             |
+   Note: N is the physical address width supported by the processor. X is the page level
+
+   Format of the SPPT L1E:
+   | Bit   | Contents                                                          |
+   | :---- | :---------------------------------------------------------------- |
+   | 0+2i  | Write permission for i-th 128 byte sub-page region.               |
+   | 1+2i  | Reserved (0).                                                     |
+   Note: `0<=i<=31`
+
+* Sub-page write access bitmap setting pseudo-code:
+
+```c
+static int p2m_set_subpage(struct domain *d,
+			   struct xen_hvm_subpage_t *spp_info)
+{
+    gfn_t *gfns = spp_info->gfns;
+    u64 *access_map = spp_info->access_map;
+
+    sanity_check();
+
+    /* SPP works when the page is unwritable */
+    if (set_ept_leaf_level_unwritable(gfn) == success)
+
+        if (p2m_set_spp_page_st(gfn) == success)
+	    
+	    success;
+
+}
+```
+
+User could get the subpage info via SPP Xen hyper call `HVMOP_get_subpage`.
+
+* Sub-page get subpage info pseudo-code:
+
+```c
+static int p2m_get_subpage(struct domain *d
+			   struct xen_hvm_subpage_t *spp_info)
+{
+	gfn_t *gfns = spp_info->gfns;
+
+	sanity_check(gfn);
+	spp_info = p2m_get_spp_page_frame(gfn);
+}
+```
+
+6. SPPT-induced vmexits
+
+* SPP VM exits
+
+Accesses using guest physical addresses may cause VM exits due to a SPPT
+Misconfiguration or a SPPT Miss.
+
+A SPPT Misconfiguration vmexit occurs when, in the course of translating
+a guest physical address, the logical proceesor encounters a leaf EPT
+paging-structure entry mapping a 4KB page, with SPP enabled, during the
+SPPT lookup, a SPPT paging-structure entry contains an unsupported
+value.
+
+A SPPT Miss vmexit occurs during the SPPT lookup there is no SPPT
+misconfiguration but any level of SPPT paging-structure entries are not
+present.
+
+NOTE. SPPT misconfigurations and SPPT miss can occur only due to an
+attempt to write memory with a guest physical address.
+
+* EPT violation vmexits due to SPPT
+
+EPT violations due to memory write accesses disallowed due to sub-page
+protection permissions specified in the SPPT are reported via EPT
+violation VM exits.
+
+7. SPPT-induced vmexits handling
+
+```c
+#define EXIT_REASON_SPP                 66
+vmx_vmexit_handler {
+	...
+	[EXIT_REASON_SPP]                     = vmx_handle_spp,
+	...
+};
+```
+New exit qualification for SPPT-induced vmexits.
+
+| Bit   | Contents                                                          |
+| :---- | :---------------------------------------------------------------- |
+| 10:0  | Reserved (0).                                                     |
+| 11    | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. |
+| 12    | NMI unblocking due to IRET                                        |
+| 63:13 | Reserved (0)                                                      |
+
+* SPPT miss and misconfiguration
+
+SPP VMexit handler Pseudo-code:
+```c
+static int vmx_handle_spp()
+{
+	exit_qualification = vmcs_readl(EXIT_QUALIFICATION);
+	if (exit_qualification & SPP_EXIT_TYPE_BIT) {
+		/* SPPT Miss */
+	} else {
+		/* SPPT Misconfig */
+		WARN_ON(1);
+	}
+	return 0;
+}
+```
+
+8. EPT violation vmexits due to SPPT
+
+While hardware walking the SPP page table, If the sub-page region write
+permission bit is set, the write is allowed, else the write is disallowed
+and results in an EPT violation.
+
+we need peek this case in EPT violation handler.
+
+

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:08 UTC #2 lore raw

RFC PATCH 2/14 xen: vmx: Added VMX SPP feature flags and VM-Execution Controls.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

Add new secondary processor-based VM-execution control bit which defined as "sub-page write permission", VMX Procbased MSR - MSR_IA32_VMX_EXIT_CTLS bit 23 is the capability bit of SPP. And VMX_SECONDARY_EXEC_CONTROL bit 23 is the enable bit of SPP.

Also we introduced a spp_enable parameter to control the SPP is ON/OFF, Set the default is OFF as we are on the way of enabling. we can add spp_enable=1 Xen boot cmdline to enable Xen SPP.

Now SPP is active when the "Sub-page Write Protection" in Secondary VM-Execution Control is set and enable the Xen parameter by "spp_enable=1".

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/vmx/vmcs.c        | 14 ++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmcs.h |  3 +++
 2 files changed, 17 insertions(+)

xen/arch/x86/hvm/vmx/vmcs.c +14 -0

index 8103b20..bee5d74 100644
@@ -50,6 +50,9 @@ boolean_param("unrestricted_guest", opt_unrestricted_guest_enabled);
 static bool_t __read_mostly opt_apicv_enabled = 1;
 boolean_param("apicv", opt_apicv_enabled);
 
+static bool_t __read_mostly opt_spp_enabled = 0;
+boolean_param("spp_enable", opt_spp_enabled);
+
 /*
  * These two parameters are used to config the controls for Pause-Loop Exiting:
  * ple_gap:    upper bound on the amount of time between two successive

@@ -138,6 +141,7 @@ static void __init vmx_display_features(void)
     P(cpu_has_vmx_virt_exceptions, "Virtualisation Exceptions");
     P(cpu_has_vmx_pml, "Page Modification Logging");
     P(cpu_has_vmx_tsc_scaling, "TSC Scaling");
+    P(cpu_has_vmx_ept_spp, "EPT Sub-page Write Protection");
 #undef P
 
     if ( !printed )

@@ -243,6 +247,8 @@ static int vmx_init_vmcs_config(void)
             opt |= SECONDARY_EXEC_UNRESTRICTED_GUEST;
         if ( opt_pml_enabled )
             opt |= SECONDARY_EXEC_ENABLE_PML;
+        if ( opt_spp_enabled )
+            opt |= SECONDARY_EXEC_ENABLE_SPP;
 
         /*
          * "APIC Register Virtualization" and "Virtual Interrupt Delivery"

@@ -336,6 +342,14 @@ static int vmx_init_vmcs_config(void)
         _vmx_secondary_exec_control &= ~ SECONDARY_EXEC_PAUSE_LOOP_EXITING;
     }
 
+    /* SPP cannot be supported if EPT is not used */
+    if ( !(_vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_EPT) )
+        _vmx_secondary_exec_control &= ~SECONDARY_EXEC_ENABLE_SPP;
+
+    /* Turn off opt_spp_enabled if SPP feature is not present */
+    if ( !(_vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_SPP) )
+        opt_spp_enabled = 0;
+
     min = VM_EXIT_ACK_INTR_ON_EXIT;
     opt = VM_EXIT_SAVE_GUEST_PAT | VM_EXIT_LOAD_HOST_PAT |
           VM_EXIT_CLEAR_BNDCFGS;

xen/include/asm-x86/hvm/vmx/vmcs.h +3 -0

index e3cdfdf..139f590 100644
@@ -235,6 +235,7 @@ extern u32 vmx_vmentry_control;
 #define SECONDARY_EXEC_ENABLE_PML               0x00020000
 #define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS   0x00040000
 #define SECONDARY_EXEC_XSAVES                   0x00100000
+#define SECONDARY_EXEC_ENABLE_SPP               0x00800000
 #define SECONDARY_EXEC_TSC_SCALING              0x02000000
 extern u32 vmx_secondary_exec_control;

@@ -312,6 +313,8 @@ extern u64 vmx_ept_vpid_cap;
     (vmx_secondary_exec_control & SECONDARY_EXEC_XSAVES)
 #define cpu_has_vmx_tsc_scaling \
     (vmx_secondary_exec_control & SECONDARY_EXEC_TSC_SCALING)
+#define cpu_has_vmx_ept_spp \
+    (vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_SPP)
 
 #define VMCS_RID_TYPE_MASK              0x80000000

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:09 UTC #3 lore raw

RFC PATCH 3/14 xen: vmx: Introduce the SPPTP and SPP page table.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

SPPT has 4-level paging structure that is similar to EPT except L1E.
The sub-page permission table is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address, the index and encoding for this VMCS field is defined 0x2030 at this time.

The format of SPPTP is shown in below figure

---------------------------------------------------------------|
| Bit    | Contents                                            |
:--------------------------------------------------------------|
| 11:0   | Reserved (0)                                        |
| N-1:12 | Physical address of 4KB aligned SPPT L4E Table      |
| 51:N   | Reserved (0)                                        |
| 63:52  | Reserved (0)                                        |
---------------------------------------------------------------|

Note: N is the physical address width supported by the processor.

This patch introduced the Spp paging structures, which root page will created at p2m_alloc_table. and free at p2m_teardown. Same as EPT page table, We initialized the SPPT, and write the SPPT point into VMCS field.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/vmx/vmcs.c        |  6 ++++++
 xen/arch/x86/mm/p2m.c              | 12 +++++++++++-
 xen/include/asm-x86/hvm/vmx/vmcs.h | 11 +++++++++++
 xen/include/asm-x86/p2m.h          |  8 +++++++-
 4 files changed, 35 insertions(+), 2 deletions(-)

xen/arch/x86/hvm/vmx/vmcs.c +6 -0

index bee5d74..e2a1f1f 100644
@@ -1273,6 +1273,12 @@ static int construct_vmcs(struct vcpu *v)
 
         ept->mfn = pagetable_get_pfn(p2m_get_pagetable(p2m));
         __vmwrite(EPT_POINTER, ept->eptp);
+
+        if ( cpu_has_vmx_ept_spp ) {
+            struct spp_data *spp = &p2m->spptp;
+            spp->mfn = pagetable_get_pfn(p2m_get_spp_pagetable(p2m));
+            __vmwrite(SPPT_POINT, spp->sppt_point);
+        }
     }
 
     if ( paging_mode_hap(d) )

xen/arch/x86/mm/p2m.c +11 -1

index e8a57d1..3d618e9 100644
@@ -609,7 +609,7 @@ void p2m_free_ptp(struct p2m_domain *p2m, struct page_info *pg)
  */
 int p2m_alloc_table(struct p2m_domain *p2m)
 {
-    struct page_info *p2m_top;
+    struct page_info *p2m_top, *p2m_spp;
     struct domain *d = p2m->domain;
     int rc = 0;

@@ -639,8 +639,17 @@ int p2m_alloc_table(struct p2m_domain *p2m)
         return -ENOMEM;
     }
 
+    p2m_spp = p2m_alloc_ptp(p2m, PGT_l4_page_table);
+    if ( p2m_spp == NULL )
+    {
+        p2m_unlock(p2m);
+        return -ENOMEM;
+    }
+
     p2m->phys_table = pagetable_from_mfn(page_to_mfn(p2m_top));
 
+    p2m->spp_phys_table = pagetable_from_mfn(page_to_mfn(p2m_spp));
+
     if ( hap_enabled(d) )
         iommu_share_p2m_table(d);

@@ -678,6 +687,7 @@ void p2m_teardown(struct p2m_domain *p2m)
     p2m_lock(p2m);
     ASSERT(atomic_read(&d->shr_pages) == 0);
     p2m->phys_table = pagetable_null();
+    p2m->spp_phys_table = pagetable_null();
 
     while ( (pg = page_list_remove_head(&p2m->pages)) )
         d->arch.paging.free_page(d, pg);

xen/include/asm-x86/hvm/vmx/vmcs.h +11 -0

index 139f590..4843bc4 100644
@@ -56,6 +56,16 @@ struct ept_data {
     cpumask_var_t invalidate;
 };
 
+struct spp_data {
+   union {
+        struct {
+            u32 reserved:12;
+            u64 mfn:52;
+        };
+        u64 sppt_point;
+   };
+};
+
 #define _VMX_DOMAIN_PML_ENABLED    0
 #define VMX_DOMAIN_PML_ENABLED     (1ul << _VMX_DOMAIN_PML_ENABLED)
 struct vmx_domain {

@@ -391,6 +401,7 @@ enum vmcs_field {
     VMWRITE_BITMAP                  = 0x00002028,
     VIRT_EXCEPTION_INFO             = 0x0000202a,
     XSS_EXIT_BITMAP                 = 0x0000202c,
+    SPPT_POINT                      = 0x00002030,
     TSC_MULTIPLIER                  = 0x00002032,
     GUEST_PHYSICAL_ADDRESS          = 0x00002400,
     VMCS_LINK_POINTER               = 0x00002800,

xen/include/asm-x86/p2m.h +7 -1

index 6395e8f..0561643 100644
@@ -193,6 +193,8 @@ struct p2m_domain {
     /* Shadow translated domain: p2m mapping */
     pagetable_t        phys_table;
 
+    pagetable_t        spp_phys_table;
+
     /* Same as domain_dirty_cpumask but limited to
      * this p2m and those physical cpus whose vcpu's are in
      * guestmode.

@@ -339,6 +341,9 @@ struct p2m_domain {
         struct ept_data ept;
         /* NPT-equivalent structure could be added here. */
     };
+    union {
+        struct spp_data spptp;
+    };
 
      struct {
          spinlock_t lock;

@@ -385,7 +390,8 @@ static inline bool_t p2m_is_altp2m(const struct p2m_domain *p2m)
     return p2m->p2m_class == p2m_alternate;
 }
 
-#define p2m_get_pagetable(p2m)  ((p2m)->phys_table)
+#define p2m_get_pagetable(p2m)      ((p2m)->phys_table)
+#define p2m_get_spp_pagetable(p2m)  ((p2m)->spp_phys_table)
 
 /*
  * Ensure any deferred p2m TLB flush has been completed on all VCPUs.

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:09 UTC #4 lore raw

RFC PATCH 4/14 xen: vmx: Introduce SPP-Induced vm exit and it's handle.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

Accesses using guest-physical addresses may cause SPP-induced VM exits due to an SPPT misconfiguration or an
SPPT miss. The basic VM exit reason code reported for SPP-induced VM exits is 66.

An SPPT misconfiguration VM exit occurs when, in the course of translating a guest-physical address, the logical processor encounters a leaf EPT paging-structure entry mapping a 4KB page for which the sub-page write permission control bit is set and during the SPPT lookup an SPPT paging-structure entry contains an unsupported value.

An SPPT miss VM exit occurs when, in the course of translation a guest-physical address, the logical processor encounters a leaf EPT paging-structure entry for which the sub-page write permission control bit is set and during the SPPT lookup there is no SPPT misconfiguration but any level of SPPT paging-structure entries are not-present.

Also Introduced the new exit qualification for SPPT-induced vmexits.

| Bit   | Contents                                                          |
| :---- | :---------------------------------------------------------------- |
| 10:0  | Reserved (0).                                                     |
| 11    | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. |
| 12    | NMI unblocking due to IRET                                        |
| 63:13 | Reserved (0)                                                      |

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/vmx/vmx.c        | 31 +++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmx.h | 11 +++++++++++
 2 files changed, 42 insertions(+)

xen/arch/x86/hvm/vmx/vmx.c +31 -0

index 69ce3aa..04ae0d6 100644
@@ -3521,6 +3521,28 @@ static int vmx_handle_apic_write(void)
     return vlapic_apicv_write(current, exit_qualification & 0xfff);
 }
 
+static int vmx_handle_spp(spp_qual_t q, paddr_t gpa)
+{
+    if ( q.sppt_miss_type )
+    {
+        /*
+         * SPPT Miss :
+         * Subpage Protection Table not present
+         */
+        printk("SPP miss occured at gpa:%lx\n", gpa);
+
+        return true;
+    }
+
+    /*
+     * SPPT Misconfig
+     * This is probably possible that your sppt table
+     * set as a incorrect format
+     */
+    WARN_ON(1);
+    return false;
+}
+
 void vmx_vmexit_handler(struct cpu_user_regs *regs)
 {
     unsigned long exit_qualification, exit_reason, idtv_info, intr_info = 0;

@@ -4124,6 +4146,15 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
     case EXIT_REASON_ACCESS_LDTR_OR_TR:
         vmx_handle_descriptor_access(exit_reason);
         break;
+    case EXIT_REASON_SPP:
+    {
+        paddr_t gpa;
+
+        __vmread(GUEST_PHYSICAL_ADDRESS, &gpa);
+        __vmread(EXIT_QUALIFICATION, &exit_qualification);
+        vmx_handle_spp(exit_qualification, gpa);
+        break;
+    }
 
     case EXIT_REASON_VMX_PREEMPTION_TIMER_EXPIRED:
     case EXIT_REASON_INVPCID:

xen/include/asm-x86/hvm/vmx/vmx.h +11 -0

index 4889a64..35aada6 100644
@@ -213,6 +213,7 @@ static inline void pi_clear_sn(struct pi_desc *pi_desc)
 #define EXIT_REASON_PML_FULL            62
 #define EXIT_REASON_XSAVES              63
 #define EXIT_REASON_XRSTORS             64
+#define EXIT_REASON_SPP                 66
 
 /*
  * Interruption-information format

@@ -616,6 +617,16 @@ typedef union ept_qual {
     };
 } __transparent__ ept_qual_t;
 
+/* SPP induced vmexit qualifications definitions */
+typedef union spp_qual {
+    unsigned long raw;
+    struct {
+        unsigned long reserved   :11;
+        bool sppt_miss_type      :1;
+        unsigned long reserved2  :52;
+    };
+} __transparent__ spp_qual_t;
+
 #define EPT_L4_PAGETABLE_SHIFT      39
 #define EPT_PAGETABLE_ENTRIES       512

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:10 UTC #5 lore raw

RFC PATCH 5/14 xen: vmx: Disable the 2M/1G superpage when SPP enabled

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

Current we only support Sub-page Protection on the 4k page table.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)

xen/arch/x86/hvm/vmx/vmx.c +6 -0

index 04ae0d6..a4c24bb 100644
@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 
+    if ( cpu_has_vmx_ept_spp )
+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();
 
     lbr_tsx_fixup_check();

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:10 UTC #11 lore raw

RFC PATCH 6/14 xen: vmx: Added SPP flags in EPT leaf entry.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

This change also modified the p2m_type width to 5, bits 52:56; the p2m_access_t bits 60:57, as the bit 61 is hardware using for EPT leaf entry SPP flags.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/include/asm-x86/hvm/vmx/vmx.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

xen/include/asm-x86/hvm/vmx/vmx.h +3 -2

index 35aada6..18383b8 100644
@@ -42,8 +42,9 @@ typedef union {
         snp         :   1,  /* bit 11 - VT-d snoop control in shared
                                EPT/VT-d usage */
         mfn         :   40, /* bits 51:12 - Machine physical frame number */
-        sa_p2mt     :   6,  /* bits 57:52 - Software available 2 */
-        access      :   4,  /* bits 61:58 - p2m_access_t */
+        sa_p2mt     :   5,  /* bits 56:52 - Software available 2 */
+        access      :   4,  /* bits 60:57 - p2m_access_t */
+        spp         :   1,  /* bits 61 - SPP flags */
         tm          :   1,  /* bit 62 - VT-d transient-mapping hint in
                                shared EPT/VT-d usage */
         suppress_ve :   1;  /* bit 63 - suppress #VE */

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:11 UTC #12 lore raw

RFC PATCH 7/14 xen: vmx: Update the EPT leaf entry indicated with the SPP enable bit.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

If the sub-page write permission VM-execution control is set, treatment of write accesses to guest-physical accesses depends on the state of the accumulated write-access bit (position 1) and sub-page permission bit (position 61) in the EPT leaf paging-structure.

Software will update the EPT leaf entry sub-page permission bit while kvm_set_subpage. If the EPT write-access bit set to 0 and the SPP bit set to 1 in the leaf EPT paging-structure entry that maps a 4KB page, then the hardware will look up a VMM-managed Sub-Page Permission Table (SPPT), which will also be prepared by setup kvm_set_subpage.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/mm/mem_access.c | 24 ++++++++++++++++++++++
 xen/arch/x86/mm/p2m-ept.c    | 47 ++++++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/p2m.h    |  2 ++
 3 files changed, 73 insertions(+)

xen/arch/x86/mm/mem_access.c +24 -0

index 5adaf6d..a471c74 100644
@@ -466,6 +466,30 @@ int p2m_get_mem_access(struct domain *d, gfn_t gfn, xenmem_access_t *access)
     return _p2m_get_mem_access(p2m, gfn, access);
 }
 
+int p2m_set_mem_spp_wp(struct domain *d, gfn_t gfn)
+{
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+    mfn_t mfn;
+    p2m_access_t old_a;
+    int rc = -1;
+    p2m_type_t t;
+    unsigned long gfn_l = gfn_x(gfn);
+
+    p2m_lock(p2m);
+    mfn = p2m->get_entry(p2m, gfn_l, &t, &old_a, 0, NULL, NULL);
+    if( mfn_eq(mfn, INVALID_MFN) )
+    {
+        rc = -1;
+        goto unlock_exit;
+    }
+    if ( p2m->update_ept_spp_wp )
+        rc = p2m->update_ept_spp_wp(p2m, gfn_l);
+
+unlock_exit:
+    p2m_unlock(p2m);
+    return rc;
+}
+
 /*
  * Local variables:
  * mode: C

xen/arch/x86/mm/p2m-ept.c +47 -0

59 lines not on this page · raw message

xen/include/asm-x86/p2m.h +2 -0

index 0561643..adbc1c6 100644
@@ -266,6 +266,8 @@ struct p2m_domain {
                                           unsigned long gfn, l1_pgentry_t *p,
                                           l1_pgentry_t new, unsigned int level);
     long               (*audit_p2m)(struct p2m_domain *p2m);
+    int                (*update_ept_spp_wp)(struct p2m_domain *p2m,
+                                 unsigned long gfn);
 
     /*
      * P2M updates may require TLBs to be flushed (invalidated).

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:12 UTC #13 lore raw

RFC PATCH 8/14 xen: vmx: Added setup spp page structure.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

The hardware uses the guest-physical address and bits 11:7 of the address accessed to lookup the SPPT to fetch a write permission bit for the 128 byte wide sub-page region being accessed within the 4K guest-physical page. If the sub-page region write permission bit is set, the write is allowed; otherwise the write is disallowed and results in an EPT violation.

Guest-physical pages mapped via leaf EPT-paging-structures for which the accumulated write-access bit and the SPP bits are both clear (0) generate
EPT violations on memory writes accesses. Guest-physical pages mapped via EPT-paging-structure for which the accumulated write-access bit is set (1) allow writes, effectively ignoring the SPP bit on the leaf EPT-paging structure.

Software will setup the spp page table level4,3,2 as well as EPT page structure, and fill the level1 via the 32 bit bitmap per a single 4K page.
Now it could be divided to 32 x 128 sub-pages.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/mm/mem_access.c      | 35 +++++++++++++++
 xen/arch/x86/mm/p2m-ept.c         | 94 +++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmx.h | 10 +++++
 xen/include/asm-x86/p2m.h         |  3 ++
 4 files changed, 142 insertions(+)

xen/arch/x86/mm/mem_access.c +35 -0

index a471c74..1b97469 100644
@@ -490,6 +490,41 @@ unlock_exit:
     return rc;
 }
 
+static u64 format_spp_spte(u32 spp_wp_bitmap)
+{
+	u64 new_spte = 0;
+	int i = 0;
+
+	/*
+	 * One 4K page contains 32 sub-pages, in SPP table L4E, old bits
+	 * are reserved, so we need to transfer u32 subpage write
+	 * protect bitmap to u64 SPP L4E format.
+	 */
+	while ( i < 32 ) {
+		if ( spp_wp_bitmap & (1ULL << i) )
+			new_spte |= 1ULL << (i * 2);
+
+		i++;
+	}
+
+	return new_spte;
+}
+
+int p2m_set_spp_page_st(struct domain *d, gfn_t gfn, uint32_t access_map)
+{
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+    u64 access = format_spp_spte(access_map);
+    unsigned long gfn_l = gfn_x(gfn);
+    int ret = -1;
+
+    p2m_lock(p2m);
+    if ( p2m->spp_set_entry )
+        ret = p2m->spp_set_entry(p2m, gfn_l, access);
+    p2m_unlock(p2m);
+
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C

xen/arch/x86/mm/p2m-ept.c +94 -0

124 lines not on this page · raw message

xen/include/asm-x86/hvm/vmx/vmx.h +10 -0

index 18383b8..655ce80 100644
@@ -52,6 +52,16 @@ typedef union {
     u64 epte;
 } ept_entry_t;
 
+typedef union {
+    struct {
+        u64 present     :   1,  /* bit 0 - spp middle table is present */
+        reserved        :   11, /* bit 1:11 - reserved */
+        mfn             :   40, /* bit 12:51 - Machine physical frame number */
+        reserved2       :   12; /* bit 52:63 - reserved */
+    };
+    u64 spp;
+} spp_entry_t;
+
 typedef struct {
     /*use lxe[0] to save result */
     ept_entry_t lxe[5];

xen/include/asm-x86/p2m.h +3 -0

index adbc1c6..b94ebb2 100644
@@ -268,6 +268,9 @@ struct p2m_domain {
     long               (*audit_p2m)(struct p2m_domain *p2m);
     int                (*update_ept_spp_wp)(struct p2m_domain *p2m,
                                  unsigned long gfn);
+    int                (*spp_set_entry)(struct p2m_domain *p2m,
+                                unsigned long gfn,
+                                u64 access);
 
     /*
      * P2M updates may require TLBs to be flushed (invalidated).

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:12 UTC #16 lore raw

RFC PATCH 9/14 xen: vmx: Introduce a Hyper call to set subpage

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

The Hypercall is defined as HVMOP_set_subpage

And the Interface's parameters is defined as

struct xen_hvm_subpage {
    domid_t  domid;
    uint32_t access_map;
    uint64_t gfn;
}

The user application: xl, or some other security control daemon. will set the protection bitmap via this hyper call.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/hvm.c          | 8 ++++++++
 xen/include/public/hvm/hvm_op.h | 9 +++++++++
 2 files changed, 17 insertions(+)

xen/arch/x86/hvm/hvm.c +8 -0

index 0b1aba7..54cd916 100644
@@ -4694,6 +4694,14 @@ long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void) arg)
         rc = do_altp2m_op(arg);
         break;
 
+    case HVMOP_set_subpage: {
+        xen_hvm_subpage_t subpage;
+
+        if ( copy_from_guest(&subpage, arg, 1 ) )
+            return -EFAULT;
+        break;
+    }
+
     default:
     {
         gdprintk(XENLOG_DEBUG, "Bad HVM op %ld.\n", op);

xen/include/public/hvm/hvm_op.h +9 -0

index 0bdafdf..0fa5b88 100644
@@ -205,6 +205,15 @@ struct xen_hvm_altp2m_domain_state {
 };
 typedef struct xen_hvm_altp2m_domain_state xen_hvm_altp2m_domain_state_t;
 DEFINE_XEN_GUEST_HANDLE(xen_hvm_altp2m_domain_state_t);
+#define HVMOP_set_subpage          26
+struct xen_hvm_subpage {
+    domid_t  domid;
+    uint32_t access_map;
+    uint64_t gfn;
+};
+typedef struct xen_hvm_subpage xen_hvm_subpage_t;
+DEFINE_XEN_GUEST_HANDLE(xen_hvm_subpage_t);
+
 
 struct xen_hvm_altp2m_vcpu_enable_notify {
     uint32_t vcpu_id;

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:13 UTC #19 lore raw

RFC PATCH 10/14 xen: vmx: Implement the Hypercall p2m_set_subpage

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

To utilize SPP feature, system admin should Set a Sub-page access write via SPP Hypercall HVMOP_set_subpage, which will prepared the flowing things.

(1.Got the corresponding EPT leaf entry via the guest physical address. (2.If it is a 4K page frame, flag the bit 61 to enable subpage protection on this page. (3.Setup spp page structure.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/hvm.c       |  8 ++++++++
 xen/arch/x86/mm/mem_access.c | 15 +++++++++++++++
 xen/include/xen/mem_access.h |  2 ++
 3 files changed, 25 insertions(+)

xen/arch/x86/hvm/hvm.c +8 -0

index 54cd916..afc4620 100644
@@ -4696,9 +4696,17 @@ long do_hvm_op(unsigned long op, XEN_GUEST_HANDLE_PARAM(void) arg)
 
     case HVMOP_set_subpage: {
         xen_hvm_subpage_t subpage;
+        struct domain *d;
 
         if ( copy_from_guest(&subpage, arg, 1 ) )
             return -EFAULT;
+
+        d = rcu_lock_domain_by_any_id(subpage.domid);
+        if ( d == NULL )
+            return -ESRCH;
+
+        rc = p2m_set_subpage(d, (_gfn)(subpage.gfn), subpage.access_map);
+        rcu_unlock_domain(d);
         break;
     }

xen/arch/x86/mm/mem_access.c +15 -0

index 1b97469..fdedc4a 100644
@@ -525,6 +525,21 @@ int p2m_set_spp_page_st(struct domain *d, gfn_t gfn, uint32_t access_map)
     return ret;
 }
 
+int p2m_set_subpage(struct domain *d, gfn_t gfn, uint32_t access_map)
+{
+    int ret;
+    ret = p2m_set_mem_spp_wp(d, gfn);
+    if ( ret < 0 )
+    {
+        printk("SPP: Set subpage ept wp failed!! %x\n", ret);
+        return ret;
+    }
+    ret = p2m_set_spp_page_st(d, gfn, access_map);
+    if ( ret < 0 )
+        printk("SPP: Set subpage table failed!! %x\n", ret);
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C

xen/include/xen/mem_access.h +2 -0

index 5ab34c1..28eb70c 100644
@@ -78,6 +78,8 @@ long p2m_set_mem_access_multi(struct domain *d,
  */
 int p2m_get_mem_access(struct domain *d, gfn_t gfn, xenmem_access_t *access);
 
+int p2m_set_subpage(struct domain *d, gfn_t gfn, uint32_t access_map);
+
 #ifdef CONFIG_HAS_MEM_ACCESS
 int mem_access_memop(unsigned long cmd,
                      XEN_GUEST_HANDLE_PARAM(xen_mem_access_op_t) arg);

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:13 UTC #20 lore raw

RFC PATCH 11/14 xen: vmx: Added handle of SPP write protection fault

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

While hardware walking the SPP page table, If the sub-page region write permission bit is set, the write is allowed, else the write is disallowed and results in an EPT violation.

we need peek this case in EPT violation handler.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/hvm/hvm.c        | 5 +++++
 xen/arch/x86/hvm/vmx/vmx.c    | 3 +++
 xen/arch/x86/mm/p2m-ept.c     | 2 ++
 xen/include/asm-x86/hvm/hvm.h | 2 ++
 xen/include/xen/mem_access.h  | 1 +
 5 files changed, 13 insertions(+)

xen/arch/x86/hvm/hvm.c +5 -0

index afc4620..a7ced32 100644
@@ -1777,6 +1777,11 @@ int hvm_hap_nested_page_fault(paddr_t gpa, unsigned long gla,
         case p2m_access_rwx:
             violation = 0;
             break;
+        case p2m_access_spp:
+            printk("SPP: spp write protect: acc mode:%d\n", npfec.write_access);
+            violation = npfec.write_access;
+            rc = HVM_SPP_WRITE_PROTECTED;
+            goto out_put_gfn;
         }
 
         if ( violation )

xen/arch/x86/hvm/vmx/vmx.c +3 -0

index a4c24bb..0481ffd 100644
@@ -3295,6 +3295,9 @@ static void ept_handle_violation(ept_qual_t q, paddr_t gpa)
                         nestedhvm_paging_mode_hap(current ) )
             __vmwrite(EPT_POINTER, get_shadow_eptp(current));
         return;
+    case HVM_SPP_WRITE_PROTECTED:
+        update_guest_eip();
+        return;
     case -1:        // This vioaltion should be injected to L1 VMM
         vcpu_nestedhvm(current).nv_vmexit_pending = 1;
         return;

xen/arch/x86/mm/p2m-ept.c +2 -0

index c9dc29c..065beb9 100644
@@ -214,6 +214,7 @@ static void ept_p2m_type_to_flags(struct p2m_domain *p2m, ept_entry_t *entry,
             entry->x = 0;
             break;           
         case p2m_access_rwx:
+        case p2m_access_spp:
             break;
     }

@@ -756,6 +757,7 @@ ept_spp_update_wp(struct p2m_domain *p2m, unsigned long gfn)
     new_entry = atomic_read_ept_entry(ept_entry);
     new_entry.spp = 1;
     new_entry.w = 0;
+    new_entry.access = p2m_access_spp;
     write_atomic(&(ept_entry->epte), new_entry.epte);
 
     ept_sync_domain(p2m);

xen/include/asm-x86/hvm/hvm.h +2 -0

index b687e03..30c6775 100644
@@ -80,6 +80,8 @@ enum hvm_intblk {
 #define HVM_EVENT_VECTOR_UNSET    (-1)
 #define HVM_EVENT_VECTOR_UPDATING (-2)
 
+#define HVM_SPP_WRITE_PROTECTED 2
+
 /*
  * The hardware virtual machine (HVM) interface abstracts away from the
  * x86/x86_64 CPU virtualization assist specifics. Currently this interface

xen/include/xen/mem_access.h +1 -0

index 28eb70c..b5811dd 100644
@@ -54,6 +54,7 @@ typedef enum {
     p2m_access_n2rwx = 9, /* Special: page goes from N to RWX on access, *
                            * generates an event but does not pause the
                            * vcpu */
+    p2m_access_spp = 0x0d,
 
     /* NOTE: Assumed to be only 4 bits right now on x86. */
 } p2m_access_t;

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:14 UTC #21 lore raw

RFC PATCH 12/14 xen: vmx: Support for clear EPT SPP write Protect bit

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

This is a implement of clear the Write protect bit.

EPT SPP Write Protect bit will remove when we set all the 32 sub-pages is writeable. We should also update the EPT page frame w bit as the whole page is writeable.

Signed-off-by Zhang Yi Z <[email protected]>

---
 xen/arch/x86/mm/mem_access.c | 12 +++++++++---
 xen/arch/x86/mm/p2m-ept.c    | 17 +++++++++++++----
 xen/include/asm-x86/p2m.h    |  2 +-
 3 files changed, 23 insertions(+), 8 deletions(-)

xen/arch/x86/mm/mem_access.c +9 -3

index fdedc4a..fddb6e3 100644
@@ -466,7 +466,7 @@ int p2m_get_mem_access(struct domain *d, gfn_t gfn, xenmem_access_t *access)
     return _p2m_get_mem_access(p2m, gfn, access);
 }
 
-int p2m_set_mem_spp_wp(struct domain *d, gfn_t gfn)
+int p2m_set_mem_spp_wp(struct domain *d, gfn_t gfn, bool spp, bool w)
 {
     struct p2m_domain *p2m = p2m_get_hostp2m(d);
     mfn_t mfn;

@@ -482,8 +482,9 @@ int p2m_set_mem_spp_wp(struct domain *d, gfn_t gfn)
         rc = -1;
         goto unlock_exit;
     }
+
     if ( p2m->update_ept_spp_wp )
-        rc = p2m->update_ept_spp_wp(p2m, gfn_l);
+        rc = p2m->update_ept_spp_wp(p2m, gfn_l, spp, w);
 
 unlock_exit:
     p2m_unlock(p2m);

@@ -528,7 +529,12 @@ int p2m_set_spp_page_st(struct domain *d, gfn_t gfn, uint32_t access_map)
 int p2m_set_subpage(struct domain *d, gfn_t gfn, uint32_t access_map)
 {
     int ret;
-    ret = p2m_set_mem_spp_wp(d, gfn);
+    bool w, spp = 0;
+
+    w = !~access_map;
+    spp = !w;
+    printk("SPP: Set subpage spp=%d, is write:%d\n", spp, w);
+    ret = p2m_set_mem_spp_wp(d, gfn, spp, w);
     if ( ret < 0 )
     {
         printk("SPP: Set subpage ept wp failed!! %x\n", ret);

xen/arch/x86/mm/p2m-ept.c +13 -4

index 065beb9..55c3ded 100644
@@ -46,6 +46,11 @@ static inline bool_t is_epte_valid(ept_entry_t *e)
     return ((e->epte & ~(1ul << 63)) != 0 && e->sa_p2mt != p2m_invalid);
 }
 
+static inline bool_t is_epte_spp_wp(ept_entry_t *e)
+{
+    return ( e->spp && !(e->w) && (e->access == p2m_access_spp) );
+}
+
 /* returns : 0 for success, -errno otherwise */
 static int atomic_write_ept_entry(ept_entry_t *entryptr, ept_entry_t new,
                                   int level)

@@ -725,7 +730,7 @@ bool_t ept_handle_misconfig(uint64_t gpa)
 }
 
 static int
-ept_spp_update_wp(struct p2m_domain *p2m, unsigned long gfn)
+ept_spp_update_wp(struct p2m_domain *p2m, unsigned long gfn, bool spp, bool w)
 {
     ept_entry_t *table, *ept_entry = NULL;
     unsigned long gfn_remainder = gfn;

@@ -754,12 +759,16 @@ ept_spp_update_wp(struct p2m_domain *p2m, unsigned long gfn)
         goto out;
     }
 
+    if ( is_epte_spp_wp(ept_entry) && spp )
+        goto sync_out;
+
     new_entry = atomic_read_ept_entry(ept_entry);
-    new_entry.spp = 1;
-    new_entry.w = 0;
-    new_entry.access = p2m_access_spp;
+    new_entry.spp = spp;
+    new_entry.w = w;
+    new_entry.access = spp? p2m_access_spp:p2m_access_n;
     write_atomic(&(ept_entry->epte), new_entry.epte);
 
+sync_out:
     ept_sync_domain(p2m);
     rc = 0;
 out:

xen/include/asm-x86/p2m.h +1 -1

index b94ebb2..36d1cc9 100644
@@ -267,7 +267,7 @@ struct p2m_domain {
                                           l1_pgentry_t new, unsigned int level);
     long               (*audit_p2m)(struct p2m_domain *p2m);
     int                (*update_ept_spp_wp)(struct p2m_domain *p2m,
-                                 unsigned long gfn);
+                                 unsigned long gfn, bool spp, bool w);
     int                (*spp_set_entry)(struct p2m_domain *p2m,
                                 unsigned long gfn,
                                 u64 access);

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:15 UTC #22 lore raw

RFC PATCH 13/14 xen: tools: Introduce the set-subpage into xenctrl

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

Introduce the Xen Hypercall HVMOP_set_subpage into Xenctl. The API is defined as flowing.

int xc_mem_set_subpage(xc_interface *handle, domid_t domid,

                       xen_pfn_t gfn, uint32_t access);

Signed-off-by Zhang, Yi Z <[email protected]>

---
 tools/libxc/include/xenctrl.h |  2 ++
 tools/libxc/xc_mem_paging.c   | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+)

tools/libxc/include/xenctrl.h +2 -0

index bde8313..a13f2c7 100644
@@ -1956,6 +1956,8 @@ int xc_mem_paging_evict(xc_interface *xch, domid_t domain_id, uint64_t gfn);
 int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, uint64_t gfn);
 int xc_mem_paging_load(xc_interface *xch, domid_t domain_id,
                        uint64_t gfn, void *buffer);
+int xc_mem_set_subpage(xc_interface *handle, domid_t domid,
+                             xen_pfn_t gfn, uint32_t access);
 
 /** 
  * Access tracking operations.

tools/libxc/xc_mem_paging.c +20 -0

index 28611f4..36f7949 100644
@@ -116,6 +116,26 @@ int xc_mem_paging_load(xc_interface *xch, domid_t domain_id,
     return rc;
 }
 
+int xc_mem_set_subpage(xc_interface *handle, domid_t domid,
+                             xen_pfn_t gfn, uint32_t access)
+{
+    int rc;
+    DECLARE_HYPERCALL_BUFFER(xen_hvm_subpage_t, arg);
+
+    arg = xc_hypercall_buffer_alloc(handle, arg, sizeof(*arg));
+    if ( arg == NULL )
+        return -1;
+
+    arg->domid = domid;
+    arg->access_map = access;
+    arg->gfn = gfn;
+
+    rc = xencall2(handle->xcall, __HYPERVISOR_hvm_op, HVMOP_set_subpage,
+		  HYPERCALL_BUFFER_AS_ARG(arg));
+
+    xc_hypercall_buffer_free(handle, arg);
+    return rc;
+}
 
 /*
  * Local variables:

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Zhang Yi

author

2017-10-19 08:15 UTC #25 lore raw

RFC PATCH 14/14 xen: tools: Added xen-subpage tool.

In reply to Zhang Yi

From: Zhang Yi Z <[email protected]>

It is a tool could set a 4K page corresponding a 32 bit bitmap:
xen-subpage -m [domid] set [gfn] [bitmap]

Signed-off-by Zhang Yi Z <[email protected]>

---
 tools/tests/xen-subpage/Makefile      |  30 ++++++++
 tools/tests/xen-subpage/xen-subpage.c | 125 ++++++++++++++++++++++++++++++++++
 2 files changed, 155 insertions(+)
 create mode 100644 tools/tests/xen-subpage/Makefile
 create mode 100644 tools/tests/xen-subpage/xen-subpage.c

tools/tests/xen-subpage/Makefile new file +30 -0

new file mode 100644
index 0000000..89b58f2
@@ -0,0 +1,30 @@
+XEN_ROOT=$(CURDIR)/../../..
+include $(XEN_ROOT)/tools/Rules.mk
+
+CFLAGS += -Werror
+CFLAGS += -DXC_WANT_COMPAT_DEVICEMODEL_API
+
+CFLAGS += $(CFLAGS_libxenctrl)
+CFLAGS += $(CFLAGS_libxenguest)
+CFLAGS += $(CFLAGS_xeninclude)
+
+TARGETS-y := xen-subpage
+TARGETS := $(TARGETS-y)
+
+.PHONY: all
+all: build
+
+.PHONY: build
+build: $(TARGETS)
+
+.PHONY: clean
+clean:
+	$(RM) *.o $(TARGETS) *~ $(DEPS)
+
+.PHONY: distclean
+distclean: clean
+
+xen-subpage: xen-subpage.o Makefile
+	$(CC) -o $@ $< $(LDFLAGS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenevtchn)
+
+-include $(DEPS)

tools/tests/xen-subpage/xen-subpage.c new file +125 -0

125 lines not on this page · raw message

Signature · 7 lines

2.7.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Razvan Cojocaru

2017-10-19 08:37 UTC #23 lore raw

Re: [PATCH RFC 13/14] xen: tools: Introduce the set-subpage into xenctrl

In reply to Zhang Yi

Quoted 19 lines · Zhang Yi

From: Zhang Yi Z <[email protected]>
Introduce the Xen Hypercall HVMOP_set_subpage into Xenctl. The API is defined as flowing.
int xc_mem_set_subpage(xc_interface *handle, domid_t domid,
                       xen_pfn_t gfn, uint32_t access);
Signed-off-by Zhang, Yi Z <[email protected]>
---
 tools/libxc/include/xenctrl.h |  2 ++
 tools/libxc/xc_mem_paging.c   | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+)
tools/libxc/include/xenctrl.h +0 -0
index bde8313..a13f2c7 100644

tools/libxc/include/xenctrl.h int xc_mem_paging_evict(xc_interface *xch, domid_t domain_id, uint64_t gfn); · line 1960

Quoted 5 lines

tools/libxc/include/xenctrl.h +2 -0

@@ -1956,6 +1956,8 @@ int xc_mem_paging_evict(xc_interface *xch, domid_t domain_id, uint64_t gfn);
 int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, uint64_t gfn);
 int xc_mem_paging_load(xc_interface *xch, domid_t domain_id,
                        uint64_t gfn, void *buffer);
+int xc_mem_set_subpage(xc_interface *handle, domid_t domid,
+                             xen_pfn_t gfn, uint32_t access);

Would you consider a small comment here explaining at least the access parameter?

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Razvan Cojocaru

2017-10-19 08:42 UTC #26 lore raw

Re: [PATCH RFC 14/14] xen: tools: Added xen-subpage tool.

In reply to Zhang Yi

Quoted 77 lines

tools/tests/xen-subpage/xen-subpage.c new file +77 -0

+#include <errno.h>
+#include <inttypes.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <string.h>
+#include <time.h>
+#include <signal.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <poll.h>
+
+#include <xenctrl.h>
+
+#define DPRINTF(a, b...) fprintf(stderr, a, ## b)
+#define ERROR(a, b...) fprintf(stderr, a "\n", ## b)
+#define PERROR(a, b...) fprintf(stderr, a ": %s\n", ## b, strerror(errno))
+
+void usage(char* progname)
+{
+    fprintf(stderr, "Usage: %s [-m] <domain_id> get|set [gfn] [bit_map]", progname);
+
+            fprintf(stderr,
+            "\n"
+            "set - set gfn bitmap.\n"
+            "\n"
+            "-m requires this program to run\n");
+}
+
+int main(int argc, char *argv[])
+{
+    domid_t domain_id;
+    xc_interface *xch;
+    xen_pfn_t gfn = 0;
+    uint32_t access = 0;
+    int required = 0;
+    int rc = 0;
+
+    char* progname = argv[0];
+    argv++;
+    argc--;
+
+    if ( argc == 5 && argv[0][0] == '-' )
+    {
+        if ( !strcmp(argv[0], "-m") )
+            required = 1;
+        else
+        {
+            usage(progname);
+            return -1;
+        }
+        argv++;
+        argc--;
+    }
+
+    if ( argc != 4 )
+    {
+        usage(progname);
+        return -1;
+    }
+
+    domain_id = atoi(argv[0]);
+    argv++;
+    argc--;
+
+    if ( !strcmp(argv[0], "set") )
+    {
+        gfn = strtoul(argv[1], 0, 0);
+        access = strtoul(argv[2], 0, 0);
+        DPRINTF("set subpage gfn:0x%lx -- map:0x%x\n", gfn, access);
+        xch = xc_interface_open(NULL, NULL, 0);
+        if ( !xch )
+        {
+            ERROR("get interface error\n");
+            return -1;
+        }
+        xc_mem_set_subpage(xch, domain_id, gfn, access);

tools/tests/xen-subpage/xen-subpage.c line 115

Quoted 10 lines

tools/tests/xen-subpage/xen-subpage.c new file +10 -0

+        xc_interface_close(xch);
+    }
+    else
+    {
+        usage(argv[0]);
+        return -1;
+    }
+
+    return rc;
+}

As far as I understand, this example just calls the new hypercall and exits.

Should there be another vm_event-subscribed application that will be affected by the changes? If so, doesn't this rather belong in the xen-access.c test?

Also, no explanation is given in comments in the source code or the displayed help for useful values of the access parameter, and what it stands for.

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Razvan Cojocaru

2017-10-19 09:07 UTC #28 lore raw

In reply to Zhang Yi

Quoted 7 lines · Zhang Yi

From: Zhang Yi Z <[email protected]>
Hi All,
Here is a patch-series which adding EPT-Based Sub-page Write Protection Support. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

Has this been tested in any way with altp2m also enabled?

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Tamas K Lengyel

2017-10-19 18:17 UTC #6 lore raw

Re: [PATCH RFC 05/14] xen: vmx: Disable the 2M/1G superpage when SPP enabled

In reply to Zhang Yi

Quoted 15 lines · Zhang Yi

From: Zhang Yi Z <[email protected]>
Current we only support Sub-page Protection on the 4k page table.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index 04ae0d6..a4c24bb 100644

xen/arch/x86/hvm/vmx/vmx.c const struct hvm_function_table * __init start_vmx(void) · line 2500

Quoted 4 lines

xen/arch/x86/hvm/vmx/vmx.c +1 -0

@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 
+    if ( cpu_has_vmx_ept_spp )

I think this really only ought to happen if the command-line option has also been enabled.

Quoted 10 lines

xen/arch/x86/hvm/vmx/vmx.c +5 -1

+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();
 
     lbr_tsx_fixup_check();
--

2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Tamas K Lengyel

2017-10-19 18:27 UTC #14 lore raw

Re: [PATCH RFC 08/14] xen: vmx: Added setup spp page structure.

In reply to Zhang Yi

Quoted 49 lines · Zhang Yi

From: Zhang Yi Z <[email protected]>
The hardware uses the guest-physical address and bits 11:7 of the address accessed to lookup the SPPT to fetch a write permission bit for the 128 byte wide sub-page region being accessed within the 4K guest-physical page. If the sub-page region write permission bit is set, the write is allowed; otherwise the write is disallowed and results in an EPT violation.
Guest-physical pages mapped via leaf EPT-paging-structures for which the accumulated write-access bit and the SPP bits are both clear (0) generate
EPT violations on memory writes accesses. Guest-physical pages mapped via EPT-paging-structure for which the accumulated write-access bit is set (1) allow writes, effectively ignoring the SPP bit on the leaf EPT-paging structure.
Software will setup the spp page table level4,3,2 as well as EPT page structure, and fill the level1 via the 32 bit bitmap per a single 4K page.
Now it could be divided to 32 x 128 sub-pages.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/mm/mem_access.c      | 35 +++++++++++++++
 xen/arch/x86/mm/p2m-ept.c         | 94 +++++++++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/vmx/vmx.h | 10 +++++
 xen/include/asm-x86/p2m.h         |  3 ++
 4 files changed, 142 insertions(+)
xen/arch/x86/mm/mem_access.c +11 -0
index a471c74..1b97469 100644
@@ -490,6 +490,41 @@ unlock_exit:
     return rc;
 }
 
+static u64 format_spp_spte(u32 spp_wp_bitmap)
+{
+       u64 new_spte = 0;
+       int i = 0;
+
+       /*
+        * One 4K page contains 32 sub-pages, in SPP table L4E, old bits
+        * are reserved, so we need to transfer u32 subpage write
+        * protect bitmap to u64 SPP L4E format.
+        */
+       while ( i < 32 ) {

xen/arch/x86/mm/mem_access.c unlock_exit: · line 513

Quoted 10 lines

xen/arch/x86/mm/mem_access.c +10 -0

+               if ( spp_wp_bitmap & (1ULL << i) )
+                       new_spte |= 1ULL << (i * 2);
+
+               i++;
+       }
+
+       return new_spte;
+}
+
+int p2m_set_spp_page_st(struct domain *d, gfn_t gfn, uint32_t access_map)

So nothing in this patch makes use of this function. Could you please re-organize the patchset so this is included with the patch that starts using it?

Quoted 7 lines

xen/arch/x86/mm/mem_access.c +7 -0

+{
+    struct p2m_domain *p2m = p2m_get_hostp2m(d);
+    u64 access = format_spp_spte(access_map);
+    unsigned long gfn_l = gfn_x(gfn);
+    int ret = -1;
+
+    p2m_lock(p2m);

xen/arch/x86/mm/mem_access.c unlock_exit: · line 530

Quoted 10 lines

xen/arch/x86/mm/mem_access.c +7 -0

+    if ( p2m->spp_set_entry )
+        ret = p2m->spp_set_entry(p2m, gfn_l, access);
+    p2m_unlock(p2m);
+
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Tamas K Lengyel

2017-10-19 18:34 UTC #17 lore raw

Re: [PATCH RFC 09/14] xen: vmx: Introduce a Hyper call to set subpage

In reply to Zhang Yi

Quoted 3 lines · Zhang Yi

From: Zhang Yi Z <[email protected]>
The Hypercall is defined as HVMOP_set_subpage

Are there any expected use-cases where a HVM guest would need access to this hypercall? Is spp compatible with #VE? If not, I think it would be better to integrate this with the existing xen_mem_access_op.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:36 UTC #29 lore raw

In reply to Razvan Cojocaru

Quoted 10 lines · Razvan Cojocaru

Quoted 7 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Hi All,
Here is a patch-series which adding EPT-Based Sub-page Write Protection Support. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
Has this been tested in any way with altp2m also enabled?

Thanks for your review Razvan. Haven't test it on altp2m mode, We plan to add the altp2m compatibility in next part of this feature.

Quoted 4 lines

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:38 UTC #27 lore raw

Re: [PATCH RFC 14/14] xen: tools: Added xen-subpage tool.

In reply to Razvan Cojocaru

Quoted 90 lines · Razvan Cojocaru

Quoted 77 lines

tools/tests/xen-subpage/xen-subpage.c new file +77 -0

+#include <errno.h>
+#include <inttypes.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <string.h>
+#include <time.h>
+#include <signal.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <poll.h>
+
+#include <xenctrl.h>
+
+#define DPRINTF(a, b...) fprintf(stderr, a, ## b)
+#define ERROR(a, b...) fprintf(stderr, a "\n", ## b)
+#define PERROR(a, b...) fprintf(stderr, a ": %s\n", ## b, strerror(errno))
+
+void usage(char* progname)
+{
+    fprintf(stderr, "Usage: %s [-m] <domain_id> get|set [gfn] [bit_map]", progname);
+
+            fprintf(stderr,
+            "\n"
+            "set - set gfn bitmap.\n"
+            "\n"
+            "-m requires this program to run\n");
+}
+
+int main(int argc, char *argv[])
+{
+    domid_t domain_id;
+    xc_interface *xch;
+    xen_pfn_t gfn = 0;
+    uint32_t access = 0;
+    int required = 0;
+    int rc = 0;
+
+    char* progname = argv[0];
+    argv++;
+    argc--;
+
+    if ( argc == 5 && argv[0][0] == '-' )
+    {
+        if ( !strcmp(argv[0], "-m") )
+            required = 1;
+        else
+        {
+            usage(progname);
+            return -1;
+        }
+        argv++;
+        argc--;
+    }
+
+    if ( argc != 4 )
+    {
+        usage(progname);
+        return -1;
+    }
+
+    domain_id = atoi(argv[0]);
+    argv++;
+    argc--;
+
+    if ( !strcmp(argv[0], "set") )
+    {
+        gfn = strtoul(argv[1], 0, 0);
+        access = strtoul(argv[2], 0, 0);
+        DPRINTF("set subpage gfn:0x%lx -- map:0x%x\n", gfn, access);
+        xch = xc_interface_open(NULL, NULL, 0);
+        if ( !xch )
+        {
+            ERROR("get interface error\n");
+            return -1;
+        }
+        xc_mem_set_subpage(xch, domain_id, gfn, access);

tools/tests/xen-subpage/xen-subpage.c line 115

Quoted 10 lines

tools/tests/xen-subpage/xen-subpage.c new file +10 -0

+        xc_interface_close(xch);
+    }
+    else
+    {
+        usage(argv[0]);
+        return -1;
+    }
+
+    return rc;
+}

As far as I understand, this example just calls the new hypercall and exits.

Quoted 3 lines

Should there be another vm_event-subscribed application that will be affected by the changes? If so, doesn't this rather belong in the xen-access.c test?

There will not be other related app on domain0 at that time, the hypercall will prepare everything we need.
We will run a write test on a HVM guest, it will tell us if a sub page will be write-protected or not.

Quoted 4 lines

Also, no explanation is given in comments in the source code or the displayed help for useful values of the access parameter, and what it stands for.

Yes, Thanks Rezvan, will add some explanation.

Quoted 4 lines

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Razvan Cojocaru

2017-10-20 08:39 UTC #30 lore raw

In reply to Yi Zhang

Quoted 6 lines · Yi Zhang

Quoted 5 lines · Razvan Cojocaru
Quoted 4 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Hi All,

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Here is a patch-series which adding EPT-Based Sub-page Write Protection Support. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
Has this been tested in any way with altp2m also enabled?
Thanks for your review Razvan. Haven't test it on altp2m mode, We plan to add the altp2m compatibility in next part of this feature.

Fair enough. Thanks for adding this feature!

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Razvan Cojocaru

2017-10-20 08:40 UTC #31 lore raw

In reply to Yi Zhang

Quoted 6 lines · Yi Zhang

Quoted 5 lines · Razvan Cojocaru
Quoted 4 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Hi All,

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Here is a patch-series which adding EPT-Based Sub-page Write Protection Support. You can get It's software developer manuals from:
https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf
Has this been tested in any way with altp2m also enabled?
Thanks for your review Razvan. Haven't test it on altp2m mode, We plan to add the altp2m compatibility in next part of this feature.

Fair enough. Thanks for adding this feature!

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:41 UTC #24 lore raw

Re: [PATCH RFC 13/14] xen: tools: Introduce the set-subpage into xenctrl

In reply to Razvan Cojocaru

Quoted 2 lines · Razvan Cojocaru

Would you consider a small comment here explaining at least the access parameter?

Yes, Thanks Rezvan, will add some explanation.

Quoted 4 lines

Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:41 UTC #18 lore raw

Re: [PATCH RFC 09/14] xen: vmx: Introduce a Hyper call to set subpage

In reply to Tamas K Lengyel

Quoted 8 lines · Tamas K Lengyel

Quoted 3 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
The Hypercall is defined as HVMOP_set_subpage
Are there any expected use-cases where a HVM guest would need access to this hypercall? Is spp compatible with #VE? If not, I think it would be better to integrate this with the existing xen_mem_access_op.

That a good point Tamas, I'm considering to use this hypercall from a HVM guest, then it would be simple and useful if a HVM guest can control it by itself, there already somebody are investigating on it. So let us wait a while for this hypercall.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:42 UTC #15 lore raw

Re: [PATCH RFC 08/14] xen: vmx: Added setup spp page structure.

In reply to Tamas K Lengyel

Quoted 5 lines · Tamas K Lengyel

So nothing in this patch makes use of this function. Could you please re-organize the patchset so this is included with the patch that starts using it?

Yes, will follow up, Thanks for your review. Tamas.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-20 08:44 UTC #7 lore raw

Re: [PATCH RFC 05/14] xen: vmx: Disable the 2M/1G superpage when SPP enabled

In reply to Tamas K Lengyel

Quoted 19 lines · Tamas K Lengyel

Quoted 18 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Current we only support Sub-page Protection on the 4k page table.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index 04ae0d6..a4c24bb 100644
@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 

Quoted 4 lines

xen/arch/x86/hvm/vmx/vmx.c const struct hvm_function_table * __init start_vmx(void) · line 2500
Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+    if ( cpu_has_vmx_ept_spp )
I think this really only ought to happen if the command-line option has also been enabled.

Sorry, didn't catch your point, the command line option opt_hap_2m and opt_hap_1G was enable by default, I need to disable the supper page when spp feature enabled. Did you mean that if we enable 2M/1G by command-line we couldn't disable it here? yes, it is, I will improve this logic. Thank you Tamas.

Quoted 11 lines

xen/arch/x86/hvm/vmx/vmx.c +5 -1

+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();
 
     lbr_tsx_fixup_check();
--

2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Tamas K Lengyel

2017-10-24 17:44 UTC #8 lore raw

Re: [PATCH RFC 05/14] xen: vmx: Disable the 2M/1G superpage when SPP enabled

Quoted 25 lines · Yi Zhang

Quoted 19 lines · Tamas K Lengyel
Quoted 18 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Current we only support Sub-page Protection on the 4k page table.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index 04ae0d6..a4c24bb 100644
@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 
Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c const struct hvm_function_table * __init start_vmx(void) · line 2500
Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+    if ( cpu_has_vmx_ept_spp )
I think this really only ought to happen if the command-line option has also been enabled.

Quoted 5 lines

Sorry, didn't catch your point, the command line option opt_hap_2m and opt_hap_1G was enable by default, I need to disable the supper page when spp feature enabled. Did you mean that if we enable 2M/1G by command-line we couldn't disable it here? yes, it is, I will improve this logic. Thank you Tamas.

I meant that right now "cpu_has_vmx_ept_spp" looks like just checks whether the CPU supports SPP, not whether the command-line option was set to enable it. If the command line option is not set (or specifically disables SPP) then the large pages shouldn't get disabled.

Quoted 9 lines

xen/arch/x86/hvm/vmx/vmx.c +5 -0

+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
xen/arch/x86/hvm/vmx/vmx.c +0 -1
     lbr_tsx_fixup_check();
--
2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Yi Zhang

author

2017-10-25 07:41 UTC #9 lore raw

Re: [PATCH RFC 05/14] xen: vmx: Disable the 2M/1G superpage when SPP enabled

Quoted 32 lines · Tamas K Lengyel

Quoted 25 lines · Yi Zhang
Quoted 19 lines · Tamas K Lengyel
Quoted 18 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Current we only support Sub-page Protection on the 4k page table.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index 04ae0d6..a4c24bb 100644
@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 
Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c const struct hvm_function_table * __init start_vmx(void) · line 2500
Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+    if ( cpu_has_vmx_ept_spp )
I think this really only ought to happen if the command-line option has also been enabled.
Quoted 5 lines
Sorry, didn't catch your point, the command line option opt_hap_2m and opt_hap_1G was enable by default, I need to disable the supper page when spp feature enabled. Did you mean that if we enable 2M/1G by command-line we couldn't disable it here? yes, it is, I will improve this logic. Thank you Tamas.

Quoted 6 lines

I meant that right now "cpu_has_vmx_ept_spp" looks like just checks whether the CPU supports SPP, not whether the command-line option was set to enable it. If the command line option is not set (or specifically disables SPP) then the large pages shouldn't get disabled.

In patch 02/14, if we didn't set spp_enable, we will not set the spp cap in vmx_secondary_exec_control, so cp_has_vmx_ept_spp flag will set when hardware has spp cap and xen cmdlline passed the parameter "spp_enable=1"

Quoted 9 lines

xen/arch/x86/hvm/vmx/vmx.c +5 -0

+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
xen/arch/x86/hvm/vmx/vmx.c +0 -1
     lbr_tsx_fixup_check();
--
2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Tamas K Lengyel

2017-10-25 15:12 UTC #10 lore raw

Re: [PATCH RFC 05/14] xen: vmx: Disable the 2M/1G superpage when SPP enabled

Quoted 40 lines · Yi Zhang

Quoted 32 lines · Tamas K Lengyel
Quoted 25 lines · Yi Zhang
Quoted 19 lines · Tamas K Lengyel
Quoted 18 lines · Zhang Yi
From: Zhang Yi Z <[email protected]>
Current we only support Sub-page Protection on the 4k page table.
Signed-off-by Zhang Yi Z <[email protected]>
---
 xen/arch/x86/hvm/vmx/vmx.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/hvm/vmx/vmx.c +0 -0
index 04ae0d6..a4c24bb 100644
@@ -2497,6 +2497,12 @@ const struct hvm_function_table * __init start_vmx(void)
         vmx_function_table.get_guest_bndcfgs = vmx_get_guest_bndcfgs;
     }
 
Quoted 4 lines
xen/arch/x86/hvm/vmx/vmx.c const struct hvm_function_table * __init start_vmx(void) · line 2500
Quoted 1 line
xen/arch/x86/hvm/vmx/vmx.c +1 -0
+    if ( cpu_has_vmx_ept_spp )
I think this really only ought to happen if the command-line option has also been enabled.
Quoted 5 lines
Sorry, didn't catch your point, the command line option opt_hap_2m and opt_hap_1G was enable by default, I need to disable the supper page when spp feature enabled. Did you mean that if we enable 2M/1G by command-line we couldn't disable it here? yes, it is, I will improve this logic. Thank you Tamas.
Quoted 6 lines
I meant that right now "cpu_has_vmx_ept_spp" looks like just checks whether the CPU supports SPP, not whether the command-line option was set to enable it. If the command line option is not set (or specifically disables SPP) then the large pages shouldn't get disabled.

Quoted 3 lines

In patch 02/14, if we didn't set spp_enable, we will not set the spp cap in vmx_secondary_exec_control, so cp_has_vmx_ept_spp flag will set when hardware has spp cap and xen cmdlline passed the parameter "spp_enable=1"

OK, thanks, I missed that.

Quoted 10 lines

xen/arch/x86/hvm/vmx/vmx.c +5 -0

+    {
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_2MB;
+        vmx_function_table.hap_capabilities &= ~HVM_HAP_SUPERPAGE_1GB;
+    }
+
     setup_vmcs_dump();

Quoted 3 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
xen/arch/x86/hvm/vmx/vmx.c +0 -1
     lbr_tsx_fixup_check();
--
2.7.4

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel