Skip to contentSkip to conversation

Intel(R) SGX Driver

RFC PATCH Superseded Jarkko Sakkinen 3 participants 40 messages Oct 10, 2017 – Nov 14, 2017 1 message not in the archive

Subsystems (4)

v3 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v3 cover Intel(R) SGX Driver

Intel(R) SGX is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the enclave is disallowed to access the memory inside the enclave by the CPU access control. In a way you can think that SGX provides inverted sandbox. It protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption Engine (MEE) starting from the Skylake microachitecture. BIOS can define one or many MEE regions that can hold enclave data by configuring them with PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to the MEE regions. The data is encrypted using a random key whose life-time is exactly one power cycle.

You can tell if your CPU supports SGX by looking into ``/proc/cpuinfo``:

``cat /proc/cpuinfo | grep ' sgx '``

The GIT repositoy for SGX driver resides in

https://github.com/jsakkine-intel/linux-sgx.git

'le' branch contains the upstream candidate patches.

'master' branch contains the same patches with the following differences:

* top-level patch modifies the ioctl API to be SDK compatible
* does not use flexible launch control but instead relies on SDK provided Intel launch enclave.

'master' is just maintained to allow people to test the code.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()

Haim Cohen (1):
x86: add SGX MSRs to msr-index.h

Jarkko Sakkinen (9):
intel_sgx: updated MAINTAINERS
x86: define the feature control MSR's SGX launch control bit fs/pipe.c: export create_pipe_files() and replace_fd()
intel_sgx: driver for Intel Software Guard Extensions
intel_sgx: ptrace() support
intel_sgx: driver documentation
intel_sgx: in-kernel launch enclave
intel_sgx: glue code for in-kernel LE
intel_sgx: update IA32_SGXLEPUBKEYHASH* MSRs

Kai Huang (1):
x86: add SGX definition to cpufeature

Sean Christopherson (1):
x86: define the feature control MSR's SGX enable bit

 Documentation/index.rst                            |    1 +
 Documentation/x86/intel_sgx.rst                    |  131 +++
 MAINTAINERS                                        |    5 +
 arch/x86/include/asm/cpufeatures.h                 |    2 +
 arch/x86/include/asm/msr-index.h                   |    8 +
 arch/x86/include/asm/sgx.h                         |  233 +++++
 arch/x86/include/asm/sgx_arch.h                    |  268 ++++++
 arch/x86/include/uapi/asm/sgx.h                    |  147 +++
 drivers/platform/x86/Kconfig                       |    2 +
 drivers/platform/x86/Makefile                      |    1 +
 drivers/platform/x86/intel_sgx/Kconfig             |   34 +
 drivers/platform/x86/intel_sgx/Makefile            |   32 +
 drivers/platform/x86/intel_sgx/le/Makefile         |   26 +
 drivers/platform/x86/intel_sgx/le/enclave/Makefile |   46 +
 .../x86/intel_sgx/le/enclave/aes_encrypt.c         |  191 ++++
 .../platform/x86/intel_sgx/le/enclave/cmac_mode.c  |  254 +++++
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S      |  163 ++++
 .../intel_sgx/le/enclave/include/tinycrypt/aes.h   |  133 +++
 .../le/enclave/include/tinycrypt/cmac_mode.h       |  194 ++++
 .../le/enclave/include/tinycrypt/constants.h       |   59 ++
 .../intel_sgx/le/enclave/include/tinycrypt/utils.h |   95 ++
 drivers/platform/x86/intel_sgx/le/enclave/main.c   |  203 ++++
 .../platform/x86/intel_sgx/le/enclave/sgx_le.lds   |   28 +
 .../platform/x86/intel_sgx/le/enclave/sgxsign.c    |  537 +++++++++++
 drivers/platform/x86/intel_sgx/le/enclave/utils.c  |   78 ++
 drivers/platform/x86/intel_sgx/le/entry.S          |  117 +++
 .../platform/x86/intel_sgx/le/include/sgx_asm.h    |   64 ++
 .../platform/x86/intel_sgx/le/include/sgx_encl.h   |  105 ++
 drivers/platform/x86/intel_sgx/le/main.c           |  219 +++++
 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S   |   15 +
 drivers/platform/x86/intel_sgx/sgx.h               |  264 +++++
 drivers/platform/x86/intel_sgx/sgx_encl.c          | 1009 ++++++++++++++++++++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c         |  282 ++++++
 drivers/platform/x86/intel_sgx/sgx_le.c            |  290 ++++++
 .../platform/x86/intel_sgx/sgx_le_proxy_piggy.S    |   15 +
 drivers/platform/x86/intel_sgx/sgx_main.c          |  475 +++++++++
 drivers/platform/x86/intel_sgx/sgx_page_cache.c    |  586 ++++++++++++
 drivers/platform/x86/intel_sgx/sgx_util.c          |  396 ++++++++
 drivers/platform/x86/intel_sgx/sgx_vma.c           |  230 +++++
 fs/file.c                                          |    1 +
 fs/pipe.c                                          |    1 +
 41 files changed, 6940 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/aes_encrypt.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/aes.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/constants.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/utils.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/utils.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_encl.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_page_cache.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_util.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c
Signature · 1 line
2.14.1
#1 lore raw

RFC PATCH v3 1/12 intel_sgx: updated MAINTAINERS

---
 MAINTAINERS | 5 +++++
 1 file changed, 5 insertions(+)
MAINTAINERS +5 -0
index 2d3d750b19c0..30a4b7f97a93 100644
@@ -14932,6 +14932,11 @@ L:	[email protected]
 S:	Maintained
 F:	mm/zswap.c
 
+INTEL SGX
+M:	Jarkko Sakkinen <[email protected]>
+L:	[email protected]
+Q:	https://patchwork.kernel.org/project/intel-sgx/list/
+
 THE REST
 M:	Linus Torvalds <[email protected]>
 L:	[email protected]
Signature · 1 line
2.14.1
#2 lore raw

RFC PATCH v3 2/12 x86: add SGX definition to cpufeature

In reply to Jarkko Sakkinen

From: Kai Huang <[email protected]>

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 2519c6c801c9..31a7d1c0f204 100644
@@ -219,6 +219,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (ebx), word 9 */
 #define X86_FEATURE_FSGSBASE	( 9*32+ 0) /* {RD/WR}{FS/GS}BASE instructions*/
 #define X86_FEATURE_TSC_ADJUST	( 9*32+ 1) /* TSC adjustment MSR 0x3b */
+#define X86_FEATURE_SGX		( 9*32+ 2) /* Software Guard Extensions */
 #define X86_FEATURE_BMI1	( 9*32+ 3) /* 1st group bit manipulation extensions */
 #define X86_FEATURE_HLE		( 9*32+ 4) /* Hardware Lock Elision */
 #define X86_FEATURE_AVX2	( 9*32+ 5) /* AVX2 instructions */
Signature · 1 line
2.14.1
#3 lore raw

RFC PATCH v3 3/12 x86: define the feature control MSR's SGX enable bit

In reply to Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

---
 arch/x86/include/asm/msr-index.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/msr-index.h +1 -0
index 17f5c12e1afd..b35cb98b5d60 100644
@@ -435,6 +435,7 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
 #define FEATURE_CONTROL_LMCE				(1<<20)
 
 #define MSR_IA32_APICBASE		0x0000001b
Signature · 1 line
2.14.1
#4 lore raw

RFC PATCH v3 4/12 x86: define the feature control MSR's SGX launch control bit

In reply to Jarkko Sakkinen
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 31a7d1c0f204..43130f3c18a1 100644
@@ -298,6 +298,7 @@
 #define X86_FEATURE_AVX512_VPOPCNTDQ (16*32+14) /* POPCNT for vectors of DW/QW */
 #define X86_FEATURE_LA57	(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID	(16*32+22) /* RDPID instruction */
+#define X86_FEATURE_SGX_LC	(16*32+30) /* supports SGX launch configuration */
 
 /* AMD-defined CPU features, CPUID level 0x80000007 (ebx), word 17 */
 #define X86_FEATURE_OVERFLOW_RECOV (17*32+0) /* MCA overflow recovery support */
Signature · 1 line
2.14.1
#5 lore raw

RFC PATCH v3 5/12 x86: add SGX MSRs to msr-index.h

In reply to Jarkko Sakkinen

From: Haim Cohen <[email protected]>

These MSRs hold the SHA256 checksum of the currently configured launch enclave public key.

---
 arch/x86/include/asm/msr-index.h | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/include/asm/msr-index.h +7 -0
index b35cb98b5d60..22e27d46d046 100644
@@ -436,6 +436,7 @@
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
 #define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE	(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)
 
 #define MSR_IA32_APICBASE		0x0000001b
@@ -502,6 +503,12 @@
 #define PACKAGE_THERM_INT_LOW_ENABLE		(1 << 1)
 #define PACKAGE_THERM_INT_PLN_ENABLE		(1 << 24)
 
+/* Intel SGX MSRs */
+#define MSR_IA32_SGXLEPUBKEYHASH0	0x0000008C
+#define MSR_IA32_SGXLEPUBKEYHASH1	0x0000008D
+#define MSR_IA32_SGXLEPUBKEYHASH2	0x0000008E
+#define MSR_IA32_SGXLEPUBKEYHASH3	0x0000008F
+
 /* Thermal Thresholds Support */
 #define THERM_INT_THRESHOLD0_ENABLE    (1 << 15)
 #define THERM_SHIFT_THRESHOLD0        8
Signature · 1 line
2.14.1
#6 lore raw

RFC PATCH v3 6/12 fs/pipe.c: export create_pipe_files() and replace_fd()

In reply to Jarkko Sakkinen

Export create_pipe_files() and replace_fd() so that the SGX driver is able to create stdin and stdout pipes.

---
 fs/file.c | 1 +
 fs/pipe.c | 1 +
 2 files changed, 2 insertions(+)
fs/file.c +1 -0
index 1fc7fbbb4510..b1fa28919b22 100644
@@ -871,6 +871,7 @@ int replace_fd(unsigned fd, struct file *file, unsigned flags)
 	spin_unlock(&files->file_lock);
 	return err;
 }
+EXPORT_SYMBOL_GPL(replace_fd);
 
 SYSCALL_DEFINE3(dup3, unsigned int, oldfd, unsigned int, newfd, int, flags)
 {
fs/pipe.c +1 -0
index 97e5be897753..ee33a84127e7 100644
@@ -784,6 +784,7 @@ int create_pipe_files(struct file **res, int flags)
 	iput(inode);
 	return err;
 }
+EXPORT_SYMBOL_GPL(create_pipe_files);
 
 static int __do_pipe_flags(int *fd, struct file **files, int flags)
 {
Signature · 1 line
2.14.1
#7 lore raw

RFC PATCH v3 8/12 intel_sgx: ptrace() support

In reply to Jarkko Sakkinen

Implemented VMA callbacks in order to ptrace() debug enclaves.

---
 drivers/platform/x86/intel_sgx/sgx_vma.c | 115 +++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)
drivers/platform/x86/intel_sgx/sgx_vma.c +115 -0
index 29dd09f142c5..f83c0b8c7445 100644
@@ -108,8 +108,123 @@ static int sgx_vma_fault(struct vm_fault *vmf)
 		return VM_FAULT_SIGBUS;
 }
 
+static inline int sgx_vma_access_word(struct sgx_encl *encl,
+				      unsigned long addr,
+				      void *buf,
+				      int len,
+				      int write,
+				      struct sgx_encl_page *encl_page,
+				      int i)
+{
+	char data[sizeof(unsigned long)];
+	int align, cnt, offset;
+	void *vaddr;
+	int ret;
+
+	offset = ((addr + i) & (PAGE_SIZE - 1)) & ~(sizeof(unsigned long) - 1);
+	align = (addr + i) & (sizeof(unsigned long) - 1);
+	cnt = sizeof(unsigned long) - align;
+	cnt = min(cnt, len - i);
+
+	if (write) {
+		if (encl_page->flags & SGX_ENCL_PAGE_TCS &&
+		    (offset < 8 || (offset + (len - i)) > 16))
+			return -ECANCELED;
+
+		if (align || (cnt != sizeof(unsigned long))) {
+			vaddr = sgx_get_page(encl_page->epc_page);
+			ret = __edbgrd((void *)((unsigned long)vaddr + offset),
+				       (unsigned long *)data);
+			sgx_put_page(vaddr);
+			if (ret) {
+				sgx_dbg(encl, "EDBGRD returned %d\n", ret);
+				return -EFAULT;
+			}
+		}
+
+		memcpy(data + align, buf + i, cnt);
+		vaddr = sgx_get_page(encl_page->epc_page);
+		ret = __edbgwr((void *)((unsigned long)vaddr + offset),
+			       (unsigned long *)data);
+		sgx_put_page(vaddr);
+		if (ret) {
+			sgx_dbg(encl, "EDBGWR returned %d\n", ret);
+			return -EFAULT;
+		}
+	} else {
+		if (encl_page->flags & SGX_ENCL_PAGE_TCS &&
+		    (offset + (len - i)) > 72)
+			return -ECANCELED;
+
+		vaddr = sgx_get_page(encl_page->epc_page);
+		ret = __edbgrd((void *)((unsigned long)vaddr + offset),
+			       (unsigned long *)data);
+		sgx_put_page(vaddr);
+		if (ret) {
+			sgx_dbg(encl, "EDBGRD returned %d\n", ret);
+			return -EFAULT;
+		}
+
+		memcpy(buf + i, data + align, cnt);
+	}
+
+	return cnt;
+}
+
+static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
+			  void *buf, int len, int write)
+{
+	struct sgx_encl *encl = vma->vm_private_data;
+	struct sgx_encl_page *entry = NULL;
+	const char *op_str = write ? "EDBGWR" : "EDBGRD";
+	int ret = 0;
+	int i;
+
+	/* If process was forked, VMA is still there but vm_private_data is set
+	 * to NULL.
+	 */
+	if (!encl)
+		return -EFAULT;
+
+	if (!(encl->flags & SGX_ENCL_DEBUG) ||
+	    !(encl->flags & SGX_ENCL_INITIALIZED) ||
+	    (encl->flags & SGX_ENCL_DEAD))
+		return -EFAULT;
+
+	sgx_dbg(encl, "%s addr=0x%lx, len=%d\n", op_str, addr, len);
+
+	for (i = 0; i < len; i += ret) {
+		if (!entry || !((addr + i) & (PAGE_SIZE - 1))) {
+			if (entry)
+				entry->flags &= ~SGX_ENCL_PAGE_RESERVED;
+
+			entry = sgx_fault_page(vma, (addr + i) & PAGE_MASK,
+					       SGX_FAULT_RESERVE);
+			if (IS_ERR(entry)) {
+				ret = PTR_ERR(entry);
+				entry = NULL;
+				break;
+			}
+		}
+
+		/* No locks are needed because used fields are immutable after
+		 * intialization.
+		 */
+		ret = sgx_vma_access_word(encl, addr, buf, len, write,
+					  entry, i);
+		if (ret < 0)
+			break;
+	}
+
+	if (entry)
+		entry->flags &= ~SGX_ENCL_PAGE_RESERVED;
+
+	return (ret < 0 && ret != -ECANCELED) ? ret : i;
+}
+
 const struct vm_operations_struct sgx_vm_ops = {
 	.close = sgx_vma_close,
 	.open = sgx_vma_open,
 	.fault = sgx_vma_fault,
+	.access = sgx_vma_access,
 };
Signature · 1 line
2.14.1
#8 lore raw

RFC PATCH v3 9/12 intel_sgx: driver documentation

In reply to Jarkko Sakkinen
---
 Documentation/index.rst         |   1 +
 Documentation/x86/intel_sgx.rst | 131 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst
Documentation/index.rst +1 -0
index cb7f1ba5b3b1..ccfebc260e04 100644
@@ -86,6 +86,7 @@ implementation.
    :maxdepth: 2
 
    sh/index
+   x86/index
 
 Korean translations
 -------------------
Documentation/x86/intel_sgx.rst new file +131 -0

131 lines not on this page · raw message

Signature · 1 line
2.14.1
#9 lore raw

RFC PATCH v3 10/12 intel_sgx: in-kernel launch enclave

In reply to Jarkko Sakkinen

This commits implements the in-kernel launch enclave. It is wrapped into a user space program that reads SIGSTRUCT instances from stdin and outputs launch tokens to stdout.

The commit also adds enclave signing tool that is used by kbuild to measure and sign the lauch enclave.

CONFIG_INTEL_SGX_SIGNING_KEY points to a PEM-file for the 3072-bit RSA key that is used as the LE public key pair. The default location is:

drivers/platform/x86/intel_sgx/intel_sgx_signing_key.pem

If the default key does not exist kbuild will generate a random key and place it to this location. KBUILD_SGX_SIGN_PIN can be used to specify the passphrase for the LE public key.

TinyCrypt (https://github.com/01org/tinycrypt) is used as AES implementation, which is not timing resistant. Eventually this needs to be replaced with AES-NI based implementation that could be either

- re-use existing AES-NI code in the kernel
- have its own hand written code

---
 drivers/platform/x86/intel_sgx/Kconfig             |  13 +
 drivers/platform/x86/intel_sgx/Makefile            |  18 +
 drivers/platform/x86/intel_sgx/le/Makefile         |  26 +
 drivers/platform/x86/intel_sgx/le/enclave/Makefile |  46 ++
 .../x86/intel_sgx/le/enclave/aes_encrypt.c         | 191 ++++++++
 .../platform/x86/intel_sgx/le/enclave/cmac_mode.c  | 254 ++++++++++
 .../x86/intel_sgx/le/enclave/encl_bootstrap.S      | 163 +++++++
 .../intel_sgx/le/enclave/include/tinycrypt/aes.h   | 133 +++++
 .../le/enclave/include/tinycrypt/cmac_mode.h       | 194 ++++++++
 .../le/enclave/include/tinycrypt/constants.h       |  59 +++
 .../intel_sgx/le/enclave/include/tinycrypt/utils.h |  95 ++++
 drivers/platform/x86/intel_sgx/le/enclave/main.c   | 203 ++++++++
 .../platform/x86/intel_sgx/le/enclave/sgx_le.lds   |  28 ++
 .../platform/x86/intel_sgx/le/enclave/sgxsign.c    | 537 +++++++++++++++++++++
 drivers/platform/x86/intel_sgx/le/enclave/utils.c  |  78 +++
 drivers/platform/x86/intel_sgx/le/entry.S          | 117 +++++
 .../platform/x86/intel_sgx/le/include/sgx_asm.h    |  64 +++
 .../platform/x86/intel_sgx/le/include/sgx_encl.h   | 105 ++++
 drivers/platform/x86/intel_sgx/le/main.c           | 219 +++++++++
 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S   |  15 +
 .../platform/x86/intel_sgx/sgx_le_proxy_piggy.S    |  11 +
 21 files changed, 2569 insertions(+)
 create mode 100644 drivers/platform/x86/intel_sgx/le/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/aes_encrypt.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/aes.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/cmac_mode.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/constants.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/utils.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/enclave/utils.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/entry.S
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_asm.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/include/sgx_encl.h
 create mode 100644 drivers/platform/x86/intel_sgx/le/main.c
 create mode 100644 drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S
drivers/platform/x86/intel_sgx/Kconfig +13 -0
index 5c7e61ecb524..9c445e9f89fa 100644
@@ -2,6 +2,8 @@
 # Intel SGX
 #
 
+menu "Intel SGX"
+
 config INTEL_SGX
 	tristate "Intel(R) SGX Driver"
 	default n
@@ -17,3 +19,14 @@ config INTEL_SGX
 	called Enclave Page Cache (EPC). There is a hardware unit in the
 	processor called Memory Encryption Engine. The MEE encrypts and decrypts
 	the EPC pages as they enter and leave the processor package.
+
+
+config INTEL_SGX_SIGNING_KEY
+	string "Path to the Intel SGX LE signing key"
+	default "drivers/platform/x86/intel_sgx/intel_sgx_signing_key.pem"
+	depends on INTEL_SGX
+	---help---
+	Provide a path to a 3072-bit RSA private key that will be used to
+	sign the launch enclave.
+
+endmenu
drivers/platform/x86/intel_sgx/Makefile +18 -0
index 92af94668508..39c0a5ad242c 100644
@@ -11,3 +11,21 @@ intel_sgx-$(CONFIG_INTEL_SGX) += \
 	sgx_page_cache.o \
 	sgx_util.o \
 	sgx_vma.o \
+	sgx_le_proxy_piggy.o
+
+$(eval $(call config_filename,INTEL_SGX_SIGNING_KEY))
+
+INTEL_SGX_SIGNING_KEY_PATH := \
+	$(INTEL_SGX_SIGNING_KEY_SRCPREFIX)$(INTEL_SGX_SIGNING_KEY_FILENAME)
+
+ifeq ($(CONFIG_INTEL_SGX_SIGNING_KEY),"drivers/platform/x86/intel_sgx/intel_sgx_signing_key.pem")
+$(INTEL_SGX_SIGNING_KEY_PATH):
+	$(Q)openssl genrsa -3 -out $(INTEL_SGX_SIGNING_KEY_PATH) 3072
+endif
+
+$(obj)/sgx_le_proxy_piggy.o: $(INTEL_SGX_SIGNING_KEY_PATH) \
+			     $(obj)/le/sgx_le_proxy
+$(obj)/le/sgx_le_proxy: FORCE
+	$(Q)$(MAKE) $(build)=$(obj)/le $@
+
+export INTEL_SGX_SIGNING_KEY_PATH
drivers/platform/x86/intel_sgx/le/Makefile new file +26 -0
new file mode 100644
index 000000000000..50fe1849f6ca
@@ -0,0 +1,26 @@
+KASAN_SANITIZE := n
+OBJECT_FILES_NON_STANDARD := y
+KCOV_INSTRUMENT := n
+KBUILD_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -I$(obj)/include
+KBUILD_AFLAGS += -I$(obj)/include
+
+subdir- := enclave
+
+always := sgx_le_proxy
+clean-files := sgx_le_proxy
+
+#
+# sgx_le_proxy
+#
+
+sgx_le_proxy-y += main.o entry.o sgx_le_piggy.o
+targets += $(sgx_le_proxy-y)
+SGX_LE_PROXY_OBJS = $(addprefix $(obj)/,$(sgx_le_proxy-y))
+
+$(obj)/sgx_le_piggy.o: $(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss
+$(obj)/enclave/sgx_le.bin $(obj)/enclave/sgx_le.ss: FORCE
+	$(Q)$(MAKE) $(build)=$(obj)/enclave $@
+
+targets += sgx_le_proxy
+$(obj)/sgx_le_proxy: $(SGX_LE_PROXY_OBJS)
+	$(call if_changed,ld)
drivers/platform/x86/intel_sgx/le/enclave/Makefile new file +46 -0

46 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/aes_encrypt.c new file +191 -0

191 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/cmac_mode.c new file +254 -0

254 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/encl_bootstrap.S new file +163 -0

163 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/aes.h new file +133 -0

133 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/cmac_mode.h new file +194 -0

194 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/constants.h new file +59 -0

59 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/include/tinycrypt/utils.h new file +95 -0

95 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/main.c new file +203 -0

203 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/sgx_le.lds new file +28 -0
new file mode 100644
index 000000000000..56b694c15ebe
@@ -0,0 +1,28 @@
+OUTPUT_FORMAT(elf64-x86-64)
+
+SECTIONS
+{
+	. = 0;
+	.tcs : {
+		*(.tcs*)
+	}
+
+	. = ALIGN(4096);
+	.text : {
+		*(.text*)
+		*(.rodata*)
+	}
+
+	. = ALIGN(4096);
+	.data : {
+		*(.data*)
+	}
+
+	/DISCARD/ : {
+		*(.data*)
+		*(.comment*)
+		*(.note*)
+		*(.debug*)
+		*(.eh_frame*)
+	}
+}
drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c new file +537 -0

537 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/enclave/utils.c new file +78 -0

78 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/entry.S new file +117 -0

117 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/include/sgx_asm.h new file +64 -0

64 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/include/sgx_encl.h new file +105 -0

105 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/main.c new file +219 -0

219 lines not on this page · raw message

drivers/platform/x86/intel_sgx/le/sgx_le_piggy.S new file +15 -0
new file mode 100644
index 000000000000..e5631e8a755d
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+	.balign PAGE_SIZE
+
+GLOBAL(sgx_le_blob)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.bin"
+END(sgx_le_blob)
+
+GLOBAL(sgx_le_blob_end);
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S new file +11 -0
new file mode 100644
index 000000000000..faced8a9a75a
@@ -0,0 +1,11 @@
+#include <linux/linkage.h>
+#include <asm/export.h>
+#include <asm/page_types.h>
+
+	.section ".rodata","a"
+
+GLOBAL(sgx_le_proxy)
+	.incbin	"drivers/platform/x86/intel_sgx/le/sgx_le_proxy"
+END(sgx_le_proxy)
+
+GLOBAL(sgx_le_proxy_end)
Signature · 1 line
2.14.1
#10 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 10/12] intel_sgx: in-kernel launch enclave

Quoted 6 lines · Jarkko Sakkinen
(quoted diff) +6 -0
+static RSA *load_sign_key(const char *path)
+{
+	FILE *f;
+	RSA *key;
+
+	f = fopen(path, "rb");

Review

Quoted 10 lines
(quoted diff) +10 -0
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return NULL;
+	}
+	key = RSA_new();
+	if (!PEM_read_RSAPrivateKey(f, &key, pem_passwd_cb, NULL))
+		return NULL;
+	fclose(f);
+
+	if (BN_num_bytes(key->n) != SGX_MODULUS_SIZE) {

Dereferencing the RSA pointer (key) breaks on OpenSSL 1.1.0 as RSA is now an opaque object.  It's relatively easy to fudge around the issue, patch below.

https://github.com/openssl/openssl/issues/1491
https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes

Review

Quoted 9 lines
(quoted diff) +1 -0
+		fprintf(stderr, "Invalid key size %d\n", BN_num_bytes(key-
Quoted 1 line

n));

(quoted diff) +7 -0
+		RSA_free(key);
+		return NULL;
+	}
+
+	return key;
+}
+
(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c index 27e8c61d033c..e454dc95f438 100644

platform/x86/intel_sgx/le/enclave/sgxsign.c +0 -0
@@ -110,6 +110,17 @@ static int pem_passwd_cb(char *buf, int size, int rwflag,

void *u)
        return strlen(buf) >= size ? size - 1 : strlen(buf);
 }

+static inline const BIGNUM *get_modulus(RSA *key)
+{
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+       return key->n;
+#else
+       const BIGNUM *n;
+       RSA_get0_key(key, &n, NULL, NULL);
+       return n;
+#endif
+}
+
 static RSA *load_sign_key(const char *path)
 {
        FILE *f;
@@ -125,8 +136,9 @@ static RSA *load_sign_key(const char *path) return NULL;
        fclose(f);

-       if (BN_num_bytes(key->n) != SGX_MODULUS_SIZE) {
-               fprintf(stderr, "Invalid key size %d\n", BN_num_bytes(key->n)); +       if (BN_num_bytes(get_modulus(key)) != SGX_MODULUS_SIZE) { +               fprintf(stderr, "Invalid key size %d\n",
+                       BN_num_bytes(get_modulus(key))); RSA_free(key);
                return NULL;
        }
@@ -511,7 +523,7 @@ int main(int argc, char **argv)
        if (!sign_key)
                goto out;

-       BN_bn2bin(sign_key->n, ss.modulus);
+       BN_bn2bin(get_modulus(sign_key), ss.modulus);

        if (!measure_encl(argv[1], ss.body.mrenclave))
                goto out;

#11 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 10/12] intel_sgx: in-kernel launch enclave

Quoted 73 lines · Sean Christopherson
Quoted 6 lines · Jarkko Sakkinen
(quoted diff) +6 -0
+static RSA *load_sign_key(const char *path)
+{
+	FILE *f;
+	RSA *key;
+
+	f = fopen(path, "rb");

Review

Quoted 10 lines
(quoted diff) +10 -0
+	if (!f) {
+		fprintf(stderr, "Unable to open %s\n", path);
+		return NULL;
+	}
+	key = RSA_new();
+	if (!PEM_read_RSAPrivateKey(f, &key, pem_passwd_cb, NULL))
+		return NULL;
+	fclose(f);
+
+	if (BN_num_bytes(key->n) != SGX_MODULUS_SIZE) {

Dereferencing the RSA pointer (key) breaks on OpenSSL 1.1.0 as RSA is now an opaque object.  It's relatively easy to fudge around the issue, patch below.

https://github.com/openssl/openssl/issues/1491
https://wiki.openssl.org/index.php/OpenSSL_1.1.0_Changes

Quoted 9 lines
(quoted diff) +1 -0
+		fprintf(stderr, "Invalid key size %d\n", BN_num_bytes(key-
Quoted 1 line

n));

(quoted diff) +7 -0
+		RSA_free(key);
+		return NULL;
+	}
+
+	return key;
+}
+
(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/le/enclave/sgxsign.c index 27e8c61d033c..e454dc95f438 100644

platform/x86/intel_sgx/le/enclave/sgxsign.c +0 -0
@@ -110,6 +110,17 @@ static int pem_passwd_cb(char *buf, int size, int rwflag,
 void *u)
         return strlen(buf) >= size ? size - 1 : strlen(buf);
  }

platform/x86/intel_sgx/le/enclave/sgxsign.c +11 -0
+static inline const BIGNUM *get_modulus(RSA *key)
+{
+#if OPENSSL_VERSION_NUMBER < 0x10100000L
+       return key->n;
+#else
+       const BIGNUM *n;
+       RSA_get0_key(key, &n, NULL, NULL);
+       return n;
+#endif
+}
+

 static RSA *load_sign_key(const char *path)
 {
        FILE *f;

platform/x86/intel_sgx/le/enclave/sgxsign.c +0 -0
@@ -125,8 +136,9 @@ static RSA *load_sign_key(const char *path)
                 return NULL;
         fclose(f);


-       if (BN_num_bytes(key->n) != SGX_MODULUS_SIZE) {
-               fprintf(stderr, "Invalid key size %d\n", BN_num_bytes(key->n)); +       if (BN_num_bytes(get_modulus(key)) != SGX_MODULUS_SIZE) { +               fprintf(stderr, "Invalid key size %d\n",
+                       BN_num_bytes(get_modulus(key))); RSA_free(key);
                return NULL;
        }

platform/x86/intel_sgx/le/enclave/sgxsign.c +0 -0
@@ -511,7 +523,7 @@ int main(int argc, char **argv)
         if (!sign_key)
                 goto out;
Quoted 7 lines


-       BN_bn2bin(sign_key->n, ss.modulus);
+       BN_bn2bin(get_modulus(sign_key), ss.modulus);
        if (!measure_encl(argv[1], ss.body.mrenclave)) goto out;

Already sent v5 but I'll put this to v6. Thanks.

/Jarkko

#12 lore raw

RFC PATCH v3 11/12 intel_sgx: glue code for in-kernel LE

In reply to Jarkko Sakkinen

Implements a simple framework for hosting in-kernel LE by using the user space helper framework.

---
 drivers/platform/x86/intel_sgx/Kconfig     |   2 +
 drivers/platform/x86/intel_sgx/Makefile    |   1 +
 drivers/platform/x86/intel_sgx/sgx.h       |  15 ++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c |  13 +-
 drivers/platform/x86/intel_sgx/sgx_le.c    | 290 +++++++++++++++++++++++++++++
 drivers/platform/x86/intel_sgx/sgx_main.c  |  12 +-
 drivers/platform/x86/intel_sgx/sgx_util.c  |  25 +++
 7 files changed, 352 insertions(+), 6 deletions(-)
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_le.c
drivers/platform/x86/intel_sgx/Kconfig +2 -0
index 9c445e9f89fa..1a80ca0ddca3 100644
@@ -9,6 +9,8 @@ config INTEL_SGX
 	default n
 	depends on X86_64 && CPU_SUP_INTEL
 	select MMU_NOTIFIER
+	select CRYPTO
+	select CRYPTO_SHA256
 	---help---
 	Intel(R) SGX is a set of CPU instructions that can be used by
 	applications to set aside private regions of code and data.  The code
drivers/platform/x86/intel_sgx/Makefile +1 -0
index 39c0a5ad242c..34abcdd8eb72 100644
@@ -11,6 +11,7 @@ intel_sgx-$(CONFIG_INTEL_SGX) += \
 	sgx_page_cache.o \
 	sgx_util.o \
 	sgx_vma.o \
+	sgx_le.o \
 	sgx_le_proxy_piggy.o
 
 $(eval $(call config_filename,INTEL_SGX_SIGNING_KEY))
drivers/platform/x86/intel_sgx/sgx.h +15 -0
index 74bb91fe6176..94b6855fdad0 100644
@@ -68,6 +68,7 @@
 #include <linux/workqueue.h>
 #include <linux/mmu_notifier.h>
 #include <linux/radix-tree.h>
+#include <crypto/hash.h>
 #include <asm/sgx.h>
 
 #define SGX_EINIT_SPIN_COUNT	20
@@ -178,6 +179,7 @@ extern u32 sgx_misc_reserved;
 extern u32 sgx_xsave_size_tbl[64];
 extern bool sgx_locked_msrs;
 
+extern const struct file_operations sgx_fops;
 extern const struct vm_operations_struct sgx_vm_ops;
 
 #define sgx_pr_ratelimited(level, encl, fmt, ...)			  \
@@ -231,6 +233,9 @@ struct sgx_encl_page *sgx_fault_page(struct vm_area_struct *vma,
 				     unsigned int flags);
 
 
+int sgx_get_key_hash(struct crypto_shash *tfm, const void *modulus, void *hash);
+int sgx_get_key_hash_simple(const void *modulus, void *hash);
+
 extern struct mutex sgx_tgid_ctx_mutex;
 extern struct list_head sgx_tgid_ctx_list;
 extern atomic_t sgx_va_pages_cnt;
@@ -245,4 +250,14 @@ void sgx_put_page(void *epc_page_vaddr);
 void sgx_eblock(struct sgx_encl *encl, struct sgx_epc_page *epc_page);
 void sgx_etrack(struct sgx_encl *encl);
 
+extern struct sgx_le_ctx sgx_le_ctx;
+
+int sgx_le_init(struct sgx_le_ctx *ctx);
+void sgx_le_exit(struct sgx_le_ctx *ctx);
+
+int sgx_le_get_token(struct sgx_le_ctx *ctx,
+		     const struct sgx_encl *encl,
+		     const struct sgx_sigstruct *sigstruct,
+		     struct sgx_einittoken *token);
+
 #endif /* __ARCH_X86_INTEL_SGX_H__ */
drivers/platform/x86/intel_sgx/sgx_ioctl.c +9 -4
index 7a3af4863356..e4860d75e6b8 100644
@@ -69,7 +69,7 @@
 #include <linux/hashtable.h>
 #include <linux/shmem_fs.h>
 
-static int sgx_get_encl(unsigned long addr, struct sgx_encl **encl)
+static int sgx_encl_get(unsigned long addr, struct sgx_encl **encl)
 {
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;
@@ -156,7 +156,7 @@ static long sgx_ioc_enclave_add_page(struct file *filep, unsigned int cmd,
 	void *data;
 	int ret;
 
-	ret = sgx_get_encl(addp->addr, &encl);
+	ret = sgx_encl_get(addp->addr, &encl);
 	if (ret)
 		return ret;
 
@@ -227,11 +227,16 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,
 	if (ret)
 		goto out;
 
-	ret = sgx_get_encl(encl_id, &encl);
+	ret = sgx_encl_get(encl_id, &encl);
 	if (ret)
 		goto out;
 
-	ret = sgx_encl_init(encl, sigstruct, einittoken);
+	if (!(initp->flags && SGX_ENCLAVE_INIT_ARCH))
+		ret = sgx_le_get_token(&sgx_le_ctx, encl, sigstruct,
+				       einittoken);
+
+	if (!ret)
+		ret = sgx_encl_init(encl, sigstruct, einittoken);
 
 	kref_put(&encl->refcount, sgx_encl_release);
 
drivers/platform/x86/intel_sgx/sgx_le.c new file +290 -0

290 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_main.c +10 -2
index 2ad2dc88edff..0747be390570 100644
@@ -143,7 +143,7 @@ static unsigned long sgx_get_unmapped_area(struct file *file,
 	return addr;
 }
 
-static const struct file_operations sgx_fops = {
+const struct file_operations sgx_fops = {
 	.owner			= THIS_MODULE,
 	.unlocked_ioctl		= sgx_ioctl,
 #ifdef CONFIG_COMPAT
@@ -315,11 +315,17 @@ static int sgx_dev_init(struct device *parent, bool locked_msrs)
 		goto out_iounmap;
 	}
 
-	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	ret = sgx_le_init(&sgx_le_ctx);
 	if (ret)
 		goto out_workqueue;
 
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_le;
+
 	return 0;
+out_le:
+	sgx_le_exit(&sgx_le_ctx);
 out_workqueue:
 	destroy_workqueue(sgx_add_page_wq);
 out_iounmap:
@@ -387,6 +393,8 @@ static int sgx_drv_remove(struct platform_device *pdev)
 
 	cdev_device_del(&ctx->cdev, &ctx->dev);
 
+	sgx_le_exit(&sgx_le_ctx);
+
 	destroy_workqueue(sgx_add_page_wq);
 #ifdef CONFIG_X86_64
 	for (i = 0; i < sgx_nr_epc_banks; i++)
drivers/platform/x86/intel_sgx/sgx_util.c +25 -0
index 0ea5670f65a4..1df60ade7829 100644
@@ -369,3 +369,28 @@ void sgx_etrack(struct sgx_encl *encl)
 		sgx_invalidate(encl, true);
 	}
 }
+
+int sgx_get_key_hash(struct crypto_shash *tfm, const void *modulus, void *hash)
+{
+	SHASH_DESC_ON_STACK(shash, tfm);
+
+	shash->tfm = tfm;
+	shash->flags = CRYPTO_TFM_REQ_MAY_SLEEP;
+
+	return crypto_shash_digest(shash, modulus, SGX_MODULUS_SIZE, hash);
+}
+
+int sgx_get_key_hash_simple(const void *modulus, void *hash)
+{
+	struct crypto_shash *tfm;
+	int ret;
+
+	tfm = crypto_alloc_shash("sha256", 0, CRYPTO_ALG_ASYNC);
+	if (IS_ERR(tfm))
+		return PTR_ERR(tfm);
+
+	ret = sgx_get_key_hash(tfm, modulus, hash);
+
+	crypto_free_shash(tfm);
+	return ret;
+}
Signature · 1 line
2.14.1
#13 lore raw

RFC PATCH v3 12/12 intel_sgx: update IA32_SGXLEPUBKEYHASH* MSRs

In reply to Jarkko Sakkinen

Check if IA32_SGXLEPUBKEYHASH* MSRs match. If they do not match, allow the driver initialization to continue only if they are writable. In this case update them with the MRSIGNER of the launch enclave.

---
 drivers/platform/x86/intel_sgx/sgx.h                |  1 +
 drivers/platform/x86/intel_sgx/sgx_encl.c           | 20 ++++++++++++++++++++
 drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S |  4 ++++
 drivers/platform/x86/intel_sgx/sgx_main.c           |  7 ++++++-
 4 files changed, 31 insertions(+), 1 deletion(-)
drivers/platform/x86/intel_sgx/sgx.h +1 -0
index 94b6855fdad0..cf66bda37c1f 100644
@@ -178,6 +178,7 @@ extern u64 sgx_xfrm_mask;
 extern u32 sgx_misc_reserved;
 extern u32 sgx_xsave_size_tbl[64];
 extern bool sgx_locked_msrs;
+extern u64 sgx_le_pubkeyhash[4];
 
 extern const struct file_operations sgx_fops;
 extern const struct vm_operations_struct sgx_vm_ops;
drivers/platform/x86/intel_sgx/sgx_encl.c +20 -0
index aa0deed08cee..ac4c28c527c2 100644
@@ -68,6 +68,7 @@
 #include <linux/slab.h>
 #include <linux/hashtable.h>
 #include <linux/shmem_fs.h>
+#include <linux/percpu.h>
 
 struct sgx_add_page_req {
 	struct sgx_encl *encl;
@@ -874,6 +875,17 @@ static int sgx_einit(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
 	return ret;
 }
 
+static void sgx_update_pubkeyhash(void)
+{
+	if (sgx_locked_msrs)
+		return;
+
+	wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0, sgx_le_pubkeyhash[0]);
+	wrmsrl(MSR_IA32_SGXLEPUBKEYHASH1, sgx_le_pubkeyhash[1]);
+	wrmsrl(MSR_IA32_SGXLEPUBKEYHASH2, sgx_le_pubkeyhash[2]);
+	wrmsrl(MSR_IA32_SGXLEPUBKEYHASH3, sgx_le_pubkeyhash[3]);
+}
+
 /**
  * sgx_encl_init - perform EINIT for the given enclave
  *
@@ -909,6 +921,14 @@ int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
 		for (j = 0; j < SGX_EINIT_SPIN_COUNT; j++) {
 			ret = sgx_einit(encl, sigstruct, token);
 
+			if (ret == SGX_INVALID_ATTRIBUTE ||
+			    ret == SGX_INVALID_EINITTOKEN) {
+				preempt_disable();
+				sgx_update_pubkeyhash();
+				ret = sgx_einit(encl, sigstruct, token);
+				preempt_enable();
+			}
+
 			if (ret == SGX_UNMASKED_EVENT)
 				continue;
 			else
drivers/platform/x86/intel_sgx/sgx_le_proxy_piggy.S +4 -0
index faced8a9a75a..e1e3742a0c93 100644
@@ -9,3 +9,7 @@ GLOBAL(sgx_le_proxy)
 END(sgx_le_proxy)
 
 GLOBAL(sgx_le_proxy_end)
+
+GLOBAL(sgx_le_ss)
+	.incbin	"drivers/platform/x86/intel_sgx/le/enclave/sgx_le.ss"
+END(sgx_le_ss)
drivers/platform/x86/intel_sgx/sgx_main.c +6 -1
index 0747be390570..8380ad56f6a0 100644
@@ -80,6 +80,7 @@ MODULE_VERSION(DRV_VERSION);
  * Global data.
  */
 
+extern struct sgx_sigstruct sgx_le_ss;
 struct workqueue_struct *sgx_add_page_wq;
 #define SGX_MAX_EPC_BANKS 8
 struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
@@ -90,6 +91,7 @@ u64 sgx_xfrm_mask = 0x3;
 u32 sgx_misc_reserved;
 u32 sgx_xsave_size_tbl[64];
 bool sgx_locked_msrs;
+u64 sgx_le_pubkeyhash[4];
 
 #ifdef CONFIG_COMPAT
 long sgx_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
@@ -267,6 +269,10 @@ static int sgx_dev_init(struct device *parent, bool locked_msrs)
 		}
 	}
 
+	ret = sgx_get_key_hash_simple(sgx_le_ss.modulus, sgx_le_pubkeyhash);
+	if (ret)
+		return ret;
+
 	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
 		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
 			    &ecx, &edx);
@@ -356,7 +362,6 @@ static int sgx_drv_probe(struct platform_device *pdev)
 	}
 
 	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
-
 	if (!(fc & FEATURE_CONTROL_LOCKED)) {
 		pr_err("intel_sgx: the feature control MSR is not locked\n");
 		return -ENODEV;
Signature · 1 line
2.14.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 42 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +36 -0
new file mode 100644
index 000000000000..1089b563e07b
 
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Before returning
+ * the low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return: an EPC page or a system error code
+ */
+struct sgx_epc_page *sgx_alloc_page(unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_alloc_page_fast();
+		if (entry)
+			break;
+
+		/* We need at minimum two pages for the #PF handler. */
+		if (atomic_read(&sgx_va_pages_cnt) >
+		    (sgx_nr_total_epc_pages - 2))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+

drivers/platform/x86/intel_sgx/sgx_page_cache.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +10 -0
+		sgx_swap_pages(SGX_NR_SWAP_CLUSTER_MAX);
+		schedule();
+	}
+
+	if (sgx_nr_free_pages < sgx_nr_low_pages)
+		wake_up(&ksgxswapd_waitq);
+
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);

I think it makes sense to remove the exports from sgx_page_cache.c for the initial upstreaming given that the only consumer is the pre-release/out-of-tree KVM module, which generally requires recompiling the entire kernel anyways.

#16 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 50 lines · Sean Christopherson
Quoted 49 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +43 -0
new file mode 100644
index 000000000000..1089b563e07b
 
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Before returning
+ * the low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return: an EPC page or a system error code
+ */
+struct sgx_epc_page *sgx_alloc_page(unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_alloc_page_fast();
+		if (entry)
+			break;
+
+		/* We need at minimum two pages for the #PF handler. */
+		if (atomic_read(&sgx_va_pages_cnt) >
+		    (sgx_nr_total_epc_pages - 2))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
+		sgx_swap_pages(SGX_NR_SWAP_CLUSTER_MAX);
+		schedule();
+	}
+
+	if (sgx_nr_free_pages < sgx_nr_low_pages)
+		wake_up(&ksgxswapd_waitq);
+
Quoted 8 lines

drivers/platform/x86/intel_sgx/sgx_page_cache.c

Quoted 3 lines
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +3 -0
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);

I think it makes sense to remove the exports from sgx_page_cache.c for the initial upstreaming given that the only consumer is the pre-release/out-of-tree KVM module, which generally requires recompiling the entire kernel anyways.

Forgot them. Thanks.

For the same reason as you described I removed them from arch/x86/include/asm/sgx.h

/Jarkko

#17 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 60 lines · Jarkko Sakkinen
Quoted 50 lines · Sean Christopherson
Quoted 49 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +43 -0
new file mode 100644
index 000000000000..1089b563e07b
 
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Before returning
+ * the low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return: an EPC page or a system error code
+ */
+struct sgx_epc_page *sgx_alloc_page(unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_alloc_page_fast();
+		if (entry)
+			break;
+
+		/* We need at minimum two pages for the #PF handler. */
+		if (atomic_read(&sgx_va_pages_cnt) >
+		    (sgx_nr_total_epc_pages - 2))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
+		sgx_swap_pages(SGX_NR_SWAP_CLUSTER_MAX);
+		schedule();
+	}
+
+	if (sgx_nr_free_pages < sgx_nr_low_pages)
+		wake_up(&ksgxswapd_waitq);
+
Quoted 8 lines

drivers/platform/x86/intel_sgx/sgx_page_cache.c

Quoted 3 lines
drivers/platform/x86/intel_sgx/sgx_page_cache.c new file +3 -0
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);

I think it makes sense to remove the exports from sgx_page_cache.c for the initial upstreaming given that the only consumer is the pre-release/out-of-tree KVM module, which generally requires recompiling the entire kernel anyways.

Quoted 4 lines

Forgot them. Thanks.

For the same reason as you described I removed them from arch/x86/include/asm/sgx.h

sgx_free_page can also be cleaned up a bit if it's no longer exported, proposed patch below.

intel_sgx: make encl a required param for sgx_free_page

sgx_free_page is no longer exported, and so encl is no longer an optional param as all driver usage must specify the encl. Making encl required also eliminates the path that returns a value, i.e. modify sgx_free_page to have a void return.

---
 drivers/platform/x86/intel_sgx/sgx.h            |  2 +-
 drivers/platform/x86/intel_sgx/sgx_page_cache.c | 29 ++++++-------------------
 2 files changed, 8 insertions(+), 23 deletions(-)
drivers/platform/x86/intel_sgx/sgx.h +1 -1
index cf66bda37c1f..17eb75604184 100644
@@ -245,7 +245,7 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank);
 int sgx_page_cache_init(void);
 void sgx_page_cache_teardown(void);
 struct sgx_epc_page *sgx_alloc_page(unsigned int flags);
-int sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl);
+void sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl);
 void *sgx_get_page(struct sgx_epc_page *entry);
 void sgx_put_page(void *epc_page_vaddr);
 void sgx_eblock(struct sgx_encl *encl, struct sgx_epc_page *epc_page);
drivers/platform/x86/intel_sgx/sgx_page_cache.c +7 -22
index f8252a8b3893..bbd8ca630d9c 100644
@@ -522,23 +522,14 @@ struct sgx_epc_page *sgx_alloc_page(unsigned int flags)
 /**
  * sgx_free_page - free an EPC page
  *
- * EREMOVE an EPC page and insert it back to the list of free pages. Optionally,
- * an enclave can be given as a parameter. If the enclave is given, the
- * resulting error is printed out loud as a critical error. It is an indicator
- * of a driver bug if that would happen.
- *
- * If the enclave is not given as a parameter (like in the case when VMM uses
- * this function)), it is fully up to the caller to deal with the return value,
- * including printing it to the klog if it wants to do such a thing.
+ * EREMOVE an EPC page and insert it back to the list of free pages.
+ * If EREMOVE fails, the error is printed out loud as a critical error.
+ * It is an indicator of a driver bug if that would happen.
  *
  * @entry:	any EPC page
- * @encl:	enclave that owns the given EPC page (optional)
- *
- * Return:
- * 0 on success,
- * SGX error code if an enclave is not defined
+ * @encl:	enclave that owns the given EPC page
  */
-int sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl)
+void sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl)
 {
 	void *epc;
 	int ret;
@@ -547,19 +538,13 @@ int sgx_free_page(struct sgx_epc_page *entry, struct sgx_encl *encl)
 	ret = __eremove(epc);
 	sgx_put_page(epc);
 
-	if (ret) {
-		if (encl)
-			sgx_crit(encl, "EREMOVE returned %d\n", ret);
-		else
-			return ret;
-	}
+	if (ret)
+		sgx_crit(encl, "EREMOVE returned %d\n", ret);
 
 	spin_lock(&sgx_free_list_lock);
 	list_add(&entry->list, &sgx_free_list);
 	sgx_nr_free_pages++;
 	spin_unlock(&sgx_free_list_lock);
-
-	return 0;
 }
 
 void *sgx_get_page(struct sgx_epc_page *entry)
Signature · 1 line
2.14.2
#18 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to a message not in the archive
Quoted 29 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_encl.c new file +22 -0
new file mode 100644
index 000000000000..aa0deed08cee
@@ -0,0 +1,989 @@
 
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @created	is the enclave already created?
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that is
+ * part of the enclave, located in that address.
+ *
+ * Return:
+ * 0 on success,
+ * -EINVAL if not found,
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr, bool created,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)

drivers/platform/x86/intel_sgx/sgx_encl.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +10 -0
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	if (created) {
+		if (!encl)
+			return -EINVAL;
+	} else {
+		if (encl)
+			return -EINVAL;
+	}

What about removing @created and returning -ENOENT (or -ENXIO?) if result->vm_private_data is NULL? Removing @created will eliminate any potential confusion for the common case of @created=true. For @created=false, which should be limited to sgx_encl_create, I think that explicitly checking for "ret != -ENOENT" is more intuitive than checking whether or not sgx_encl_find succeeded, e.g. I knew the intent of the check in sgx_encl_create ahead of time and I still had to walk through sgx_encl_find to verify the behavior.

Quoted 5 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +5 -0
+
+	*vma = result;
+	return 0;
+}
+
#19 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 41 lines · Sean Christopherson
Quoted 29 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_encl.c new file +22 -0
new file mode 100644
index 000000000000..aa0deed08cee
@@ -0,0 +1,989 @@
 
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @created	is the enclave already created?
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that is
+ * part of the enclave, located in that address.
+ *
+ * Return:
+ * 0 on success,
+ * -EINVAL if not found,
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr, bool created,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)

drivers/platform/x86/intel_sgx/sgx_encl.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +10 -0
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	if (created) {
+		if (!encl)
+			return -EINVAL;
+	} else {
+		if (encl)
+			return -EINVAL;
+	}
Quoted 8 lines

What about removing @created and returning -ENOENT (or -ENXIO?) if result->vm_private_data is NULL? Removing @created will eliminate any potential confusion for the common case of @created=true. For @created=false, which should be limited to sgx_encl_create, I think that explicitly checking for "ret != -ENOENT" is more intuitive than checking whether or not sgx_encl_find succeeded, e.g. I knew the intent of the check in sgx_encl_create ahead of time and I still had to walk through sgx_encl_find to verify the behavior.

Would make sense. Thank you.

/Jarkko

#20 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 51 lines · Jarkko Sakkinen
Quoted 41 lines · Sean Christopherson
Quoted 29 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_encl.c new file +22 -0
new file mode 100644
index 000000000000..aa0deed08cee
@@ -0,0 +1,989 @@
 
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @created	is the enclave already created?
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that is
+ * part of the enclave, located in that address.
+ *
+ * Return:
+ * 0 on success,
+ * -EINVAL if not found,
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr, bool created,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)

drivers/platform/x86/intel_sgx/sgx_encl.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +10 -0
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	if (created) {
+		if (!encl)
+			return -EINVAL;
+	} else {
+		if (encl)
+			return -EINVAL;
+	}
Quoted 8 lines

What about removing @created and returning -ENOENT (or -ENXIO?) if result->vm_private_data is NULL? Removing @created will eliminate any potential confusion for the common case of @created=true. For @created=false, which should be limited to sgx_encl_create, I think that explicitly checking for "ret != -ENOENT" is more intuitive than checking whether or not sgx_encl_find succeeded, e.g. I knew the intent of the check in sgx_encl_create ahead of time and I still had to walk through sgx_encl_find to verify the behavior.

Quoted 3 lines

Would make sense. Thank you.

/Jarkko

And in the case of ioctls (sgx_encl_get()) it would probably make sense to deliver -ENOENT back to the user space instead of -EINVAL, wouldn't it?

/Jarkko

#21 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 52 lines · Jarkko Sakkinen
Quoted 51 lines · Jarkko Sakkinen
Quoted 49 lines · Sean Christopherson
Quoted 29 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_encl.c new file +22 -0
new file mode 100644
index 000000000000..aa0deed08cee
@@ -0,0 +1,989 @@
 
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @created	is the enclave already created?
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that is
+ * part of the enclave, located in that address.
+ *
+ * Return:
+ * 0 on success,
+ * -EINVAL if not found,
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr, bool created,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)

drivers/platform/x86/intel_sgx/sgx_encl.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +10 -0
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	if (created) {
+		if (!encl)
+			return -EINVAL;
+	} else {
+		if (encl)
+			return -EINVAL;
+	}

What about removing @created and returning -ENOENT (or -ENXIO?) if result->vm_private_data is NULL? Removing @created will eliminate any potential confusion for the common case of @created=true. For @created=false, which should be limited to sgx_encl_create, I think that explicitly checking for "ret != -ENOENT" is more intuitive than checking whether or not sgx_encl_find succeeded, e.g. I knew the intent of the check in sgx_encl_create ahead of time and I still had to walk through sgx_encl_find to verify the behavior.

Quoted 8 lines
Quoted 3 lines

Would make sense. Thank you.

/Jarkko

And in the case of ioctls (sgx_encl_get()) it would probably make sense to deliver -ENOENT back to the user space instead of -EINVAL, wouldn't it?

/Jarkko

Please ignore this response! :-)

/Jarkko

#22 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 57 lines · Jarkko Sakkinen
Quoted 56 lines · Jarkko Sakkinen
Quoted 54 lines · Jarkko Sakkinen
Quoted 41 lines · Sean Christopherson
Quoted 29 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_encl.c new file +22 -0
new file mode 100644
index 000000000000..aa0deed08cee
@@ -0,0 +1,989 @@
 
+/**
+ * sgx_encl_find - find an enclave
+ * @mm:		mm struct of the current process
+ * @addr:	address in the ELRANGE
+ * @created	is the enclave already created?
+ * @vma:	the resulting VMA
+ *
+ * Finds an enclave identified by the given address. Gives back the VMA, that is
+ * part of the enclave, located in that address.
+ *
+ * Return:
+ * 0 on success,
+ * -EINVAL if not found,
+ */
+int sgx_encl_find(struct mm_struct *mm, unsigned long addr, bool created,
+		  struct vm_area_struct **vma)
+{
+	struct vm_area_struct *result;
+	struct sgx_encl *encl;
+
+	result = find_vma(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)

drivers/platform/x86/intel_sgx/sgx_encl.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_encl.c new file +10 -0
+		return -EINVAL;
+
+	encl = result->vm_private_data;
+	if (created) {
+		if (!encl)
+			return -EINVAL;
+	} else {
+		if (encl)
+			return -EINVAL;
+	}
Quoted 8 lines

What about removing @created and returning -ENOENT (or -ENXIO?) if result->vm_private_data is NULL? Removing @created will eliminate any potential confusion for the common case of @created=true. For @created=false, which should be limited to sgx_encl_create, I think that explicitly checking for "ret != -ENOENT" is more intuitive than checking whether or not sgx_encl_find succeeded, e.g. I knew the intent of the check in sgx_encl_create ahead of time and I still had to walk through sgx_encl_find to verify the behavior.

Would make sense. Thank you.

/Jarkko

Quoted 8 lines
Quoted 4 lines

And in the case of ioctls (sgx_encl_get()) it would probably make sense to deliver -ENOENT back to the user space instead of -EINVAL, wouldn't it?

/Jarkko

Please ignore this response! :-)

/Jarkko

Or actually why not. Does not harm to tell user space that it is an enclave VMA but not yet created.

/Jarkko

#23 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to a message not in the archive
Quoted 20 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_main.c new file +14 -0
new file mode 100644
index 000000000000..2ad2dc88edff
 
+static dev_t sgx_devt;
+
+static void sgx_dev_release(struct device *dev)
+{
+	struct sgx_context *ctx = container_of(dev, struct sgx_context, dev);
+	kfree(ctx);
+}
+
+static struct sgx_context *sgx_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)

drivers/platform/x86/intel_sgx/sgx_main.c

Quoted 10 lines
drivers/platform/x86/intel_sgx/sgx_main.c new file +10 -0
+		return ERR_PTR(-ENOMEM);
+
+	device_initialize(&ctx->dev);
+
+	ctx->dev.bus = &sgx_subsys;
+	ctx->dev.parent = parent;
+	ctx->dev.devt = MKDEV(MAJOR(sgx_devt), 0);
+	ctx->dev.release = sgx_dev_release;
+
+	dev_set_name(&ctx->dev, "sgx");

/dev/sgx is created with 0600 permissions, does access to SGX really need to be restricted to root by default?

Quoted 9 lines
drivers/platform/x86/intel_sgx/sgx_main.c new file +9 -0
+
+	cdev_init(&ctx->cdev, &sgx_fops);
+	ctx->cdev.owner = THIS_MODULE;
+
+	dev_set_drvdata(parent, ctx);
+
+	return ctx;
+}
+
#24 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 26 lines · Sean Christopherson
Quoted 25 lines · Jarkko Sakkinen
drivers/platform/x86/intel_sgx/sgx_main.c new file +19 -0
new file mode 100644
index 000000000000..2ad2dc88edff
 
+static dev_t sgx_devt;
+
+static void sgx_dev_release(struct device *dev)
+{
+	struct sgx_context *ctx = container_of(dev, struct sgx_context, dev);
+	kfree(ctx);
+}
+
+static struct sgx_context *sgx_ctx_alloc(struct device *parent)
+{
+	struct sgx_context *ctx;
+
+	ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+	if (!ctx)
+		return ERR_PTR(-ENOMEM);
+
+	device_initialize(&ctx->dev);
+
+	ctx->dev.bus = &sgx_subsys;
Quoted 8 lines

drivers/platform/x86/intel_sgx/sgx_main.c

Quoted 5 lines
drivers/platform/x86/intel_sgx/sgx_main.c new file +5 -0
+	ctx->dev.parent = parent;
+	ctx->dev.devt = MKDEV(MAJOR(sgx_devt), 0);
+	ctx->dev.release = sgx_dev_release;
+
+	dev_set_name(&ctx->dev, "sgx");

/dev/sgx is created with 0600 permissions, does access to SGX really need to be restricted to root by default?

I personally am just a fan of whitelisting. It's only matter of a udev rule.

/Jarkko

#25 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to a message not in the archive

Review

Quoted 5 lines · Jarkko Sakkinen
(quoted diff) +5 -0
+static int __init sgx_drv_init(void)
+{
+	int ret;
+
+	ret = sgx_drv_subsys_init();

The return value of sgx_drv_subsys_init is being ignored.

Quoted 15 lines
(quoted diff) +15 -0
+
+	ret = platform_driver_register(&sgx_drv);
+	if (ret)
+		sgx_drv_subsys_exit();
+
+	return ret;
+}
+module_init(sgx_drv_init);
+
+static void __exit sgx_drv_exit(void)
+{
+	platform_driver_unregister(&sgx_drv);
+	sgx_drv_subsys_exit();
+}
+module_exit(sgx_drv_exit);
#26 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 9 lines · Sean Christopherson

Review

Quoted 5 lines · Jarkko Sakkinen
(quoted diff) +5 -0
+static int __init sgx_drv_init(void)
+{
+	int ret;
+
+	ret = sgx_drv_subsys_init();

The return value of sgx_drv_subsys_init is being ignored.

Quoted 15 lines
Quoted 5 lines
(quoted diff) +5 -0
+
+	ret = platform_driver_register(&sgx_drv);
+	if (ret)
+		sgx_drv_subsys_exit();
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+	return ret;
+}
+module_init(sgx_drv_init);
+
+static void __exit sgx_drv_exit(void)
+{
+	platform_driver_unregister(&sgx_drv);
+	sgx_drv_subsys_exit();
+}
+module_exit(sgx_drv_exit);

Holds still for v4. the latest version under review. Thanks.

/Jarkko

#27 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to a message not in the archive
Quoted 84 lines · Jarkko Sakkinen
(quoted diff) +16 -0
+static int sgx_dev_init(struct device *parent, bool locked_msrs)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax, ebx, ecx, edx;
+	unsigned long pa;
+	unsigned long size;
+	int ret;
+	int i;
+
+	pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n");
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	sgx_locked_msrs = locked_msrs;
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx,

&edx);

(quoted diff) +9 -0
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax, &ebx,

&ecx,

(quoted diff) +18 -0
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx &

0xfffff000);

(quoted diff) +38 -0
+
+		dev_info(parent, "EPC bank 0x%lx-0x%lx\n", pa, pa + size);
+
+		sgx_epc_banks[i].pa = pa;
+		sgx_epc_banks[i].size = size;
+	}
+
+	sgx_nr_epc_banks = i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+#ifdef CONFIG_X86_64
+		sgx_epc_banks[i].va = (unsigned long)
+			ioremap_cache(sgx_epc_banks[i].pa,
+				      sgx_epc_banks[i].size);
+		if (!sgx_epc_banks[i].va) {
+			sgx_nr_epc_banks = i;
+			ret = -ENOMEM;
+			goto out_iounmap;
+		}
+#endif
+		ret = sgx_add_epc_bank(sgx_epc_banks[i].pa,
+				       sgx_epc_banks[i].size, i);
+		if (ret) {
+			sgx_nr_epc_banks = i + 1;
+			goto out_iounmap;
+		}
+	}
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		goto out_iounmap;
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE, 1);
+	if (!sgx_add_page_wq) {
+		pr_err("intel_sgx: alloc_workqueue() failed\n");
+		ret = -ENOMEM;
+		goto out_iounmap;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	}
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+out_iounmap:

sgx_page_cache_teardown() should be called here, else ksgxswapd and the list of EPC pages will leak.

Quoted 6 lines
(quoted diff) +6 -0
+#ifdef CONFIG_X86_64
+	for (i = 0; i < sgx_nr_epc_banks; i++)
+		iounmap((void *)sgx_epc_banks[i].va);
+#endif
+	return ret;
+}

...

Quoted 20 lines
(quoted diff) +20 -0
+int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)
+{
+	unsigned long i;
+	struct sgx_epc_page *new_epc_page, *entry;
+	struct list_head *parser, *temp;
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);
+		if (!new_epc_page)
+			goto err_freelist;
+		new_epc_page->pa = (start + i) | bank;
+
+		spin_lock(&sgx_free_list_lock);
+		list_add_tail(&new_epc_page->list, &sgx_free_list);
+		sgx_nr_total_epc_pages++;
+		sgx_nr_free_pages++;
+		spin_unlock(&sgx_free_list_lock);
+	}
+
+	return 0;

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_freelist:
+	list_for_each_safe(parser, temp, &sgx_free_list) {
+		spin_lock(&sgx_free_list_lock);
+		entry = list_entry(parser, struct sgx_epc_page, list);
+		list_del(&entry->list);
+		spin_unlock(&sgx_free_list_lock);
+		kfree(entry);
+	}
+	return -ENOMEM;
+}

Freeing the entire list on failure does not seem like the appropriate behavior for this helper func, e.g. the list should be purged by sgx_page_cache_teardown. Buffering the new pages into a local list and only splicing said list into the global list upon success is more inline with what I would expect from a helper func, and also only requires a single lock/unlock.

(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/sgx_page_cache.c
index f8883d24692a..38496e6296f1 100644

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -397,7 +397,8 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long

size, int bank)
 {
        unsigned long i;
        struct sgx_epc_page *new_epc_page, *entry;
-       struct list_head *parser, *temp;
+       unsigned int nr_pages = 0;
+       LIST_HEAD(epc_pages);

        for (i = 0; i < size; i += PAGE_SIZE) {
                new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL); @@ -405,22 +406,19 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)
                        goto err_freelist;
                new_epc_page->pa = (start + i) | bank;

-               spin_lock(&sgx_free_list_lock);
-               list_add_tail(&new_epc_page->list, &sgx_free_list); -               sgx_nr_total_epc_pages++;
-               sgx_nr_free_pages++;
-               spin_unlock(&sgx_free_list_lock);
+               list_add_tail(&new_epc_page->list, &epc_pages);
+               nr_pages++;
        }

+       spin_lock(&sgx_free_list_lock);
+       list_splice_tail(&epc_pages, &sgx_free_list);
+       sgx_nr_total_epc_pages += nr_pages;
+       sgx_nr_free_pages += nr_pages;
+       spin_unlock(&sgx_free_list_lock);
        return 0;
 err_freelist:
-       list_for_each_safe(parser, temp, &sgx_free_list) {
-               spin_lock(&sgx_free_list_lock);
-               entry = list_entry(parser, struct sgx_epc_page, list); -               list_del(&entry->list);
-               spin_unlock(&sgx_free_list_lock);
+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }
        return -ENOMEM;
 }

#28 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 90 lines · Sean Christopherson
Quoted 89 lines · Jarkko Sakkinen
(quoted diff) +16 -0
+static int sgx_dev_init(struct device *parent, bool locked_msrs)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax, ebx, ecx, edx;
+	unsigned long pa;
+	unsigned long size;
+	int ret;
+	int i;
+
+	pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n");
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	sgx_locked_msrs = locked_msrs;
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx,

&edx);

(quoted diff) +9 -0
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax, &ebx,

&ecx,

(quoted diff) +18 -0
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx &

0xfffff000);

(quoted diff) +43 -0
+
+		dev_info(parent, "EPC bank 0x%lx-0x%lx\n", pa, pa + size);
+
+		sgx_epc_banks[i].pa = pa;
+		sgx_epc_banks[i].size = size;
+	}
+
+	sgx_nr_epc_banks = i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+#ifdef CONFIG_X86_64
+		sgx_epc_banks[i].va = (unsigned long)
+			ioremap_cache(sgx_epc_banks[i].pa,
+				      sgx_epc_banks[i].size);
+		if (!sgx_epc_banks[i].va) {
+			sgx_nr_epc_banks = i;
+			ret = -ENOMEM;
+			goto out_iounmap;
+		}
+#endif
+		ret = sgx_add_epc_bank(sgx_epc_banks[i].pa,
+				       sgx_epc_banks[i].size, i);
+		if (ret) {
+			sgx_nr_epc_banks = i + 1;
+			goto out_iounmap;
+		}
+	}
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		goto out_iounmap;
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE, 1);
+	if (!sgx_add_page_wq) {
+		pr_err("intel_sgx: alloc_workqueue() failed\n");
+		ret = -ENOMEM;
+		goto out_iounmap;
+	}
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+out_iounmap:

sgx_page_cache_teardown() should be called here, else ksgxswapd and the list of EPC pages will leak.

Thanks.

Quoted 84 lines
Quoted 6 lines
(quoted diff) +6 -0
+#ifdef CONFIG_X86_64
+	for (i = 0; i < sgx_nr_epc_banks; i++)
+		iounmap((void *)sgx_epc_banks[i].va);
+#endif
+	return ret;
+}

...

Quoted 20 lines
(quoted diff) +20 -0
+int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)
+{
+	unsigned long i;
+	struct sgx_epc_page *new_epc_page, *entry;
+	struct list_head *parser, *temp;
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);
+		if (!new_epc_page)
+			goto err_freelist;
+		new_epc_page->pa = (start + i) | bank;
+
+		spin_lock(&sgx_free_list_lock);
+		list_add_tail(&new_epc_page->list, &sgx_free_list);
+		sgx_nr_total_epc_pages++;
+		sgx_nr_free_pages++;
+		spin_unlock(&sgx_free_list_lock);
+	}
+
+	return 0;

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_freelist:
+	list_for_each_safe(parser, temp, &sgx_free_list) {
+		spin_lock(&sgx_free_list_lock);
+		entry = list_entry(parser, struct sgx_epc_page, list);
+		list_del(&entry->list);
+		spin_unlock(&sgx_free_list_lock);
+		kfree(entry);
+	}
+	return -ENOMEM;
+}

Freeing the entire list on failure does not seem like the appropriate behavior for this helper func, e.g. the list should be purged by sgx_page_cache_teardown. Buffering the new pages into a local list and only splicing said list into the global list upon success is more inline with what I would expect from a helper func, and also only requires a single lock/unlock.

(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/sgx_page_cache.c
index f8883d24692a..38496e6296f1 100644

platform/x86/intel_sgx/sgx_page_cache.c +1 -1
@@ -397,7 +397,8 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long
 size, int bank)
  {
         unsigned long i;
         struct sgx_epc_page *new_epc_page, *entry;
-       struct list_head *parser, *temp;
+       LIST_HEAD(epc_pages);


        for (i = 0; i < size; i += PAGE_SIZE) {
                new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -405,22 +406,19 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long
 size, int bank)
                         goto err_freelist;
                 new_epc_page->pa = (start + i) | bank;


-               spin_lock(&sgx_free_list_lock);
-               list_add_tail(&new_epc_page->list, &sgx_free_list); -               sgx_nr_total_epc_pages++;
-               sgx_nr_free_pages++;
-               spin_unlock(&sgx_free_list_lock);
+               list_add_tail(&new_epc_page->list, &epc_pages); +               nr_pages++;
        }

+       spin_lock(&sgx_free_list_lock);
+       list_splice_tail(&epc_pages, &sgx_free_list);
+       sgx_nr_total_epc_pages += nr_pages;
+       sgx_nr_free_pages += nr_pages;
+       spin_unlock(&sgx_free_list_lock);
        return 0;
 err_freelist:
-       list_for_each_safe(parser, temp, &sgx_free_list) { -               spin_lock(&sgx_free_list_lock);

Quoted 8 lines

-               entry = list_entry(parser, struct sgx_epc_page, list); -               list_del(&entry->list);
-               spin_unlock(&sgx_free_list_lock);
+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }
        return -ENOMEM;
 }

This makes sense to me and I think it makes also sense to move bunch of stuff from sgx_dev_init() to sgx_page_cache_init().

/Jarkko

#29 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 199 lines · Jarkko Sakkinen
Quoted 90 lines · Sean Christopherson
Quoted 89 lines · Jarkko Sakkinen
(quoted diff) +16 -0
+static int sgx_dev_init(struct device *parent, bool locked_msrs)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax, ebx, ecx, edx;
+	unsigned long pa;
+	unsigned long size;
+	int ret;
+	int i;
+
+	pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n");
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	sgx_locked_msrs = locked_msrs;
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx, &ecx,

&edx);

(quoted diff) +9 -0
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax, &ebx,

&ecx,

(quoted diff) +18 -0
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx &

0xfffff000);

(quoted diff) +43 -0
+
+		dev_info(parent, "EPC bank 0x%lx-0x%lx\n", pa, pa + size);
+
+		sgx_epc_banks[i].pa = pa;
+		sgx_epc_banks[i].size = size;
+	}
+
+	sgx_nr_epc_banks = i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+#ifdef CONFIG_X86_64
+		sgx_epc_banks[i].va = (unsigned long)
+			ioremap_cache(sgx_epc_banks[i].pa,
+				      sgx_epc_banks[i].size);
+		if (!sgx_epc_banks[i].va) {
+			sgx_nr_epc_banks = i;
+			ret = -ENOMEM;
+			goto out_iounmap;
+		}
+#endif
+		ret = sgx_add_epc_bank(sgx_epc_banks[i].pa,
+				       sgx_epc_banks[i].size, i);
+		if (ret) {
+			sgx_nr_epc_banks = i + 1;
+			goto out_iounmap;
+		}
+	}
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		goto out_iounmap;
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE, 1);
+	if (!sgx_add_page_wq) {
+		pr_err("intel_sgx: alloc_workqueue() failed\n");
+		ret = -ENOMEM;
+		goto out_iounmap;
+	}
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+out_iounmap:

sgx_page_cache_teardown() should be called here, else ksgxswapd and the list of EPC pages will leak.

Thanks.

Quoted 84 lines
Quoted 6 lines
(quoted diff) +6 -0
+#ifdef CONFIG_X86_64
+	for (i = 0; i < sgx_nr_epc_banks; i++)
+		iounmap((void *)sgx_epc_banks[i].va);
+#endif
+	return ret;
+}

...

Quoted 20 lines
(quoted diff) +20 -0
+int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)
+{
+	unsigned long i;
+	struct sgx_epc_page *new_epc_page, *entry;
+	struct list_head *parser, *temp;
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);
+		if (!new_epc_page)
+			goto err_freelist;
+		new_epc_page->pa = (start + i) | bank;
+
+		spin_lock(&sgx_free_list_lock);
+		list_add_tail(&new_epc_page->list, &sgx_free_list);
+		sgx_nr_total_epc_pages++;
+		sgx_nr_free_pages++;
+		spin_unlock(&sgx_free_list_lock);
+	}
+
+	return 0;

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_freelist:
+	list_for_each_safe(parser, temp, &sgx_free_list) {
+		spin_lock(&sgx_free_list_lock);
+		entry = list_entry(parser, struct sgx_epc_page, list);
+		list_del(&entry->list);
+		spin_unlock(&sgx_free_list_lock);
+		kfree(entry);
+	}
+	return -ENOMEM;
+}

Freeing the entire list on failure does not seem like the appropriate behavior for this helper func, e.g. the list should be purged by sgx_page_cache_teardown. Buffering the new pages into a local list and only splicing said list into the global list upon success is more inline with what I would expect from a helper func, and also only requires a single lock/unlock.

(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/sgx_page_cache.c
index f8883d24692a..38496e6296f1 100644

platform/x86/intel_sgx/sgx_page_cache.c +1 -1
@@ -397,7 +397,8 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long
 size, int bank)
  {
         unsigned long i;
         struct sgx_epc_page *new_epc_page, *entry;
-       struct list_head *parser, *temp;
+       LIST_HEAD(epc_pages);


        for (i = 0; i < size; i += PAGE_SIZE) {
                new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -405,22 +406,19 @@ int sgx_add_epc_bank(resource_size_t start, unsigned long
 size, int bank)
                         goto err_freelist;
                 new_epc_page->pa = (start + i) | bank;


-               spin_lock(&sgx_free_list_lock);
-               list_add_tail(&new_epc_page->list, &sgx_free_list); -               sgx_nr_total_epc_pages++;
-               sgx_nr_free_pages++;
-               spin_unlock(&sgx_free_list_lock);
+               list_add_tail(&new_epc_page->list, &epc_pages); +               nr_pages++;
        }

+       spin_lock(&sgx_free_list_lock);
+       list_splice_tail(&epc_pages, &sgx_free_list);
+       sgx_nr_total_epc_pages += nr_pages;
+       sgx_nr_free_pages += nr_pages;
+       spin_unlock(&sgx_free_list_lock);
        return 0;
 err_freelist:
-       list_for_each_safe(parser, temp, &sgx_free_list) { -               spin_lock(&sgx_free_list_lock);

Quoted 8 lines

-               entry = list_entry(parser, struct sgx_epc_page, list); -               list_del(&entry->list);
-               spin_unlock(&sgx_free_list_lock);
+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }
        return -ENOMEM;
 }

This makes sense to me and I think it makes also sense to move bunch of stuff from sgx_dev_init() to sgx_page_cache_init().

Quoted 1 line

/Jarkko

You can get away without nr_pages as you can get from i or am I misobserving something?

/Jarkko

#30 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 216 lines · Jarkko Sakkinen
Quoted 207 lines · Jarkko Sakkinen
Quoted 98 lines · Sean Christopherson
Quoted 96 lines · Jarkko Sakkinen
(quoted diff) +16 -0
+static int sgx_dev_init(struct device *parent, bool locked_msrs)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax, ebx, ecx, edx;
+	unsigned long pa;
+	unsigned long size;
+	int ret;
+	int i;
+
+	pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n");
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	sgx_locked_msrs = locked_msrs;
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx,

&ecx,
&edx);

(quoted diff) +9 -0
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax,

&ebx,
&ecx,

(quoted diff) +12 -0
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax,

&ebx,

(quoted diff) +5 -0
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax &

0xfffff000);

(quoted diff) +1 -0
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx &

0xfffff000);

(quoted diff) +2 -0
+
+		dev_info(parent, "EPC bank 0x%lx-0x%lx\n", pa, pa +

size);

(quoted diff) +32 -0
+
+		sgx_epc_banks[i].pa = pa;
+		sgx_epc_banks[i].size = size;
+	}
+
+	sgx_nr_epc_banks = i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+#ifdef CONFIG_X86_64
+		sgx_epc_banks[i].va = (unsigned long)
+			ioremap_cache(sgx_epc_banks[i].pa,
+				      sgx_epc_banks[i].size);
+		if (!sgx_epc_banks[i].va) {
+			sgx_nr_epc_banks = i;
+			ret = -ENOMEM;
+			goto out_iounmap;
+		}
+#endif
+		ret = sgx_add_epc_bank(sgx_epc_banks[i].pa,
+				       sgx_epc_banks[i].size, i);
+		if (ret) {
+			sgx_nr_epc_banks = i + 1;
+			goto out_iounmap;
+		}
+	}
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		goto out_iounmap;
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE,

1);

(quoted diff) +9 -0
+	if (!sgx_add_page_wq) {
+		pr_err("intel_sgx: alloc_workqueue() failed\n");
+		ret = -ENOMEM;
+		goto out_iounmap;
+	}
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+out_iounmap:

sgx_page_cache_teardown() should be called here, else ksgxswapd and the list of
EPC pages will leak.

Thanks.

Quoted 96 lines
Quoted 8 lines
(quoted diff) +6 -0
+#ifdef CONFIG_X86_64
+	for (i = 0; i < sgx_nr_epc_banks; i++)
+		iounmap((void *)sgx_epc_banks[i].va);
+#endif
+	return ret;
+}

...

Quoted 23 lines

+int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)

(quoted diff) +7 -0
+{
+	unsigned long i;
+	struct sgx_epc_page *new_epc_page, *entry;
+	struct list_head *parser, *temp;
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		new_epc_page = kzalloc(sizeof(*new_epc_page),

GFP_KERNEL);

(quoted diff) +12 -0
+		if (!new_epc_page)
+			goto err_freelist;
+		new_epc_page->pa = (start + i) | bank;
+
+		spin_lock(&sgx_free_list_lock);
+		list_add_tail(&new_epc_page->list, &sgx_free_list);
+		sgx_nr_total_epc_pages++;
+		sgx_nr_free_pages++;
+		spin_unlock(&sgx_free_list_lock);
+	}
+
+	return 0;

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_freelist:
+	list_for_each_safe(parser, temp, &sgx_free_list) {
+		spin_lock(&sgx_free_list_lock);
+		entry = list_entry(parser, struct sgx_epc_page, list);
+		list_del(&entry->list);
+		spin_unlock(&sgx_free_list_lock);
+		kfree(entry);
+	}
+	return -ENOMEM;
+}

Freeing the entire list on failure does not seem like the appropriate behavior
for this helper func, e.g. the list should be purged by sgx_page_cache_teardown.
Buffering the new pages into a local list and only splicing said list into the
global list upon success is more inline with what I would expect from a helper
func, and also only requires a single lock/unlock.

(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/sgx_page_cache.c
index f8883d24692a..38496e6296f1 100644

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -397,7 +397,8 @@ int sgx_add_epc_bank(resource_size_t start, unsigned

long
size, int bank)
 {
        unsigned long i;
        struct sgx_epc_page *new_epc_page, *entry;
-       struct list_head *parser, *temp;
+       LIST_HEAD(epc_pages);

        for (i = 0; i < size; i += PAGE_SIZE) {
                new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -405,22 +406,19 @@ int sgx_add_epc_bank(resource_size_t start, unsigned

long
size, int bank)
                        goto err_freelist;
                new_epc_page->pa = (start + i) | bank;

-               spin_lock(&sgx_free_list_lock);
-               list_add_tail(&new_epc_page->list, &sgx_free_list); -               sgx_nr_total_epc_pages++;
-               sgx_nr_free_pages++;
-               spin_unlock(&sgx_free_list_lock);
+               list_add_tail(&new_epc_page->list, &epc_pages); +               nr_pages++;
        }

+       spin_lock(&sgx_free_list_lock);
+       list_splice_tail(&epc_pages, &sgx_free_list);
+       sgx_nr_total_epc_pages += nr_pages;
+       sgx_nr_free_pages += nr_pages;
+       spin_unlock(&sgx_free_list_lock);
        return 0;
 err_freelist:
-       list_for_each_safe(parser, temp, &sgx_free_list) { -               spin_lock(&sgx_free_list_lock);
-               entry = list_entry(parser, struct sgx_epc_page, list); -               list_del(&entry->list);
-               spin_unlock(&sgx_free_list_lock);

Quoted 8 lines
Quoted 5 lines

+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }
        return -ENOMEM;
 }

This makes sense to me and I think it makes also sense to move bunch of stuff from sgx_dev_init() to sgx_page_cache_init().
/Jarkko

Quoted 2 lines

You can get away without nr_pages as you can get from i or am I misobserving something?

i is incremented by PAGE_SIZE.  I suppose you could do "+= i / PAGE_SIZE", but I would opt for clarity in this case.

#31 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 217 lines · Sean Christopherson
Quoted 216 lines · Jarkko Sakkinen
Quoted 215 lines · Jarkko Sakkinen
Quoted 98 lines · Sean Christopherson
Quoted 96 lines · Jarkko Sakkinen
(quoted diff) +16 -0
+static int sgx_dev_init(struct device *parent, bool locked_msrs)
+{
+	struct sgx_context *sgx_dev;
+	unsigned int eax, ebx, ecx, edx;
+	unsigned long pa;
+	unsigned long size;
+	int ret;
+	int i;
+
+	pr_info("intel_sgx: " DRV_DESCRIPTION " v" DRV_VERSION "\n");
+
+	sgx_dev = sgxm_ctx_alloc(parent);
+
+	sgx_locked_msrs = locked_msrs;
+
+	cpuid_count(SGX_CPUID, SGX_CPUID_CAPABILITIES, &eax, &ebx,

&ecx,
&edx);

(quoted diff) +9 -0
+	/* Only allow misc bits supported by the driver. */
+	sgx_misc_reserved = ~ebx | SGX_MISC_RESERVED_MASK;
+#ifdef CONFIG_X86_64
+	sgx_encl_size_max_64 = 1ULL << ((edx >> 8) & 0xFF);
+#endif
+	sgx_encl_size_max_32 = 1ULL << (edx & 0xFF);
+
+	if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+		cpuid_count(SGX_CPUID, SGX_CPUID_ATTRIBUTES, &eax,

&ebx,
&ecx,

(quoted diff) +12 -0
+			    &edx);
+		sgx_xfrm_mask = (((u64)edx) << 32) + (u64)ecx;
+
+		for (i = 2; i < 64; i++) {
+			cpuid_count(0x0D, i, &eax, &ebx, &ecx, &edx);
+			if ((1 << i) & sgx_xfrm_mask)
+				sgx_xsave_size_tbl[i] = eax + ebx;
+		}
+	}
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax,

&ebx,

(quoted diff) +5 -0
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax &

0xfffff000);

(quoted diff) +1 -0
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx &

0xfffff000);

(quoted diff) +2 -0
+
+		dev_info(parent, "EPC bank 0x%lx-0x%lx\n", pa, pa +

size);

(quoted diff) +32 -0
+
+		sgx_epc_banks[i].pa = pa;
+		sgx_epc_banks[i].size = size;
+	}
+
+	sgx_nr_epc_banks = i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+#ifdef CONFIG_X86_64
+		sgx_epc_banks[i].va = (unsigned long)
+			ioremap_cache(sgx_epc_banks[i].pa,
+				      sgx_epc_banks[i].size);
+		if (!sgx_epc_banks[i].va) {
+			sgx_nr_epc_banks = i;
+			ret = -ENOMEM;
+			goto out_iounmap;
+		}
+#endif
+		ret = sgx_add_epc_bank(sgx_epc_banks[i].pa,
+				       sgx_epc_banks[i].size, i);
+		if (ret) {
+			sgx_nr_epc_banks = i + 1;
+			goto out_iounmap;
+		}
+	}
+
+	ret = sgx_page_cache_init();
+	if (ret)
+		goto out_iounmap;
+
+	sgx_add_page_wq = alloc_workqueue("intel_sgx-add-page-wq",
+					  WQ_UNBOUND | WQ_FREEZABLE,

1);

(quoted diff) +9 -0
+	if (!sgx_add_page_wq) {
+		pr_err("intel_sgx: alloc_workqueue() failed\n");
+		ret = -ENOMEM;
+		goto out_iounmap;
+	}
+
+	ret = cdev_device_add(&sgx_dev->cdev, &sgx_dev->dev);
+	if (ret)
+		goto out_workqueue;
Quoted 9 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+	return 0;
+out_workqueue:
+	destroy_workqueue(sgx_add_page_wq);
+out_iounmap:

sgx_page_cache_teardown() should be called here, else ksgxswapd and the list of
EPC pages will leak.

Thanks.

Quoted 93 lines
Quoted 8 lines
(quoted diff) +6 -0
+#ifdef CONFIG_X86_64
+	for (i = 0; i < sgx_nr_epc_banks; i++)
+		iounmap((void *)sgx_epc_banks[i].va);
+#endif
+	return ret;
+}

...

Quoted 23 lines

+int sgx_add_epc_bank(resource_size_t start, unsigned long size, int bank)

(quoted diff) +7 -0
+{
+	unsigned long i;
+	struct sgx_epc_page *new_epc_page, *entry;
+	struct list_head *parser, *temp;
+
+	for (i = 0; i < size; i += PAGE_SIZE) {
+		new_epc_page = kzalloc(sizeof(*new_epc_page),

GFP_KERNEL);

(quoted diff) +12 -0
+		if (!new_epc_page)
+			goto err_freelist;
+		new_epc_page->pa = (start + i) | bank;
+
+		spin_lock(&sgx_free_list_lock);
+		list_add_tail(&new_epc_page->list, &sgx_free_list);
+		sgx_nr_total_epc_pages++;
+		sgx_nr_free_pages++;
+		spin_unlock(&sgx_free_list_lock);
+	}
+
+	return 0;

Review

Quoted 10 lines
(quoted diff) +10 -0
+err_freelist:
+	list_for_each_safe(parser, temp, &sgx_free_list) {
+		spin_lock(&sgx_free_list_lock);
+		entry = list_entry(parser, struct sgx_epc_page, list);
+		list_del(&entry->list);
+		spin_unlock(&sgx_free_list_lock);
+		kfree(entry);
+	}
+	return -ENOMEM;
+}

Freeing the entire list on failure does not seem like the appropriate behavior
for this helper func, e.g. the list should be purged by sgx_page_cache_teardown.
Buffering the new pages into a local list and only splicing said list into the
global list upon success is more inline with what I would expect from a helper
func, and also only requires a single lock/unlock.

(quoted diff) +0 -0

drivers/platform/x86/intel_sgx/sgx_page_cache.c
index f8883d24692a..38496e6296f1 100644

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -397,7 +397,8 @@ int sgx_add_epc_bank(resource_size_t start, unsigned

long
size, int bank)
 {
        unsigned long i;
        struct sgx_epc_page *new_epc_page, *entry;
-       struct list_head *parser, *temp;
+       LIST_HEAD(epc_pages);

        for (i = 0; i < size; i += PAGE_SIZE) {
                new_epc_page = kzalloc(sizeof(*new_epc_page), GFP_KERNEL);

platform/x86/intel_sgx/sgx_page_cache.c +0 -0
@@ -405,22 +406,19 @@ int sgx_add_epc_bank(resource_size_t start, unsigned

long
size, int bank)
                        goto err_freelist; new_epc_page->pa = (start + i) | bank;

-               spin_lock(&sgx_free_list_lock);
-               list_add_tail(&new_epc_page->list, &sgx_free_list); -               sgx_nr_total_epc_pages++;
-               sgx_nr_free_pages++;
-               spin_unlock(&sgx_free_list_lock);
+               list_add_tail(&new_epc_page->list, &epc_pages); +               nr_pages++;
        }

+       spin_lock(&sgx_free_list_lock);
+       list_splice_tail(&epc_pages, &sgx_free_list);
+       sgx_nr_total_epc_pages += nr_pages;
+       sgx_nr_free_pages += nr_pages;
+       spin_unlock(&sgx_free_list_lock);
        return 0;
 err_freelist:
-       list_for_each_safe(parser, temp, &sgx_free_list) { -               spin_lock(&sgx_free_list_lock);

Quoted 8 lines

-               entry = list_entry(parser, struct sgx_epc_page, list); -               list_del(&entry->list);
-               spin_unlock(&sgx_free_list_lock);
+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }
        return -ENOMEM;
 }

This makes sense to me and I think it makes also sense to move bunch of stuff from sgx_dev_init() to sgx_page_cache_init().
/Jarkko

Quoted 5 lines
Quoted 2 lines

You can get away without nr_pages as you can get from i or am I misobserving something?

i is incremented by PAGE_SIZE.  I suppose you could do "+= i / PAGE_SIZE", but I would opt for clarity in this case.

Redundant calculations do not improve clarity.

/Jarkko

#32 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to Sean Christopherson
Quoted 3 lines · Sean Christopherson

+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }

Shouldn't you iterate 'epc_pages' in the rollback now?

/Jarkko

#33 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 6 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson

+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }

Shouldn't you iterate 'epc_pages' in the rollback now?

Yes, that's the intent, nice catch.

#34 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 9 lines · Sean Christopherson
Quoted 6 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson

+       list_for_each_entry(entry, &sgx_free_list, list) kfree(entry);
-       }

Shouldn't you iterate 'epc_pages' in the rollback now?

Yes, that's the intent, nice catch.

Yeah, just a sanity check that I'm misinterpreting what I'm seeing :-) Thanks.

/Jarkko

#35 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

In reply to a message not in the archive

Review

Quoted 2 lines · Jarkko Sakkinen
(quoted diff) +2 -0
+static LIST_HEAD(sgx_free_list);
+static DEFINE_SPINLOCK(sgx_free_list_lock);

Is this a global list? Will this be a scalability problem on larger systems?

#36 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 6 lines · Dave Hansen

Review

Quoted 2 lines · Jarkko Sakkinen
(quoted diff) +2 -0
+static LIST_HEAD(sgx_free_list);
+static DEFINE_SPINLOCK(sgx_free_list_lock);

Is this a global list? Will this be a scalability problem on larger systems?

It will be need to be refined for NUMA.

In addition, per-CPU caches would probably make sense.

For simplicity, I would keep it as it is up until the driver is in the mainline.

/Jarkko

#37 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 8 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen

Review

Quoted 2 lines · Jarkko Sakkinen
(quoted diff) +2 -0
+static LIST_HEAD(sgx_free_list);
+static DEFINE_SPINLOCK(sgx_free_list_lock);

Is this a global list? Will this be a scalability problem on larger systems?

Quoted 6 lines

It will be need to be refined for NUMA.

In addition, per-CPU caches would probably make sense.

For simplicity, I would keep it as it is up until the driver is in the mainline.

FWIW, I don't think we should merge things that aren't performant. Global locks like this are just intolerable. You can add this as a later patch, but please don't merge stuff like this.

#38 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 11 lines · Dave Hansen
Quoted 10 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen

Review

Quoted 2 lines · Jarkko Sakkinen
(quoted diff) +2 -0
+static LIST_HEAD(sgx_free_list);
+static DEFINE_SPINLOCK(sgx_free_list_lock);

Is this a global list? Will this be a scalability problem on larger systems?

It will be need to be refined for NUMA.

Quoted 8 lines
Quoted 4 lines

In addition, per-CPU caches would probably make sense.

For simplicity, I would keep it as it is up until the driver is in the mainline.

FWIW, I don't think we should merge things that aren't performant. Global locks like this are just intolerable. You can add this as a later patch, but please don't merge stuff like this.

The back pointer to struct sgx_encl_page from struct sgx_epc_page is useless. I've had this in backlog already long time ago but had forgot it as I've been mostly working with the launch infrastructure lately.

Your comment worked as kind of a reminder of that. Thank you.

Once that field is removed the whole struct is useless and EPC bank converges to an array. With an array the driver should be able reserve pages without a global lock.

I've started writing a patch to make all this happen and it is progressing really well. I'm planning to include this change to v6. As it simplifies code I'm going to squash it as part of the initial driver patch.

How does this sound?

/Jarkko

#39 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 31 lines · Jarkko Sakkinen
Quoted 11 lines · Dave Hansen
Quoted 10 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen

Review

Quoted 2 lines · Jarkko Sakkinen
(quoted diff) +2 -0
+static LIST_HEAD(sgx_free_list);
+static DEFINE_SPINLOCK(sgx_free_list_lock);

Is this a global list? Will this be a scalability problem on larger systems?

It will be need to be refined for NUMA.

Quoted 8 lines
Quoted 4 lines

In addition, per-CPU caches would probably make sense.

For simplicity, I would keep it as it is up until the driver is in the mainline.

FWIW, I don't think we should merge things that aren't performant. Global locks like this are just intolerable. You can add this as a later patch, but please don't merge stuff like this.

The back pointer to struct sgx_encl_page from struct sgx_epc_page is useless. I've had this in backlog already long time ago but had forgot it as I've been mostly working with the launch infrastructure lately.

Your comment worked as kind of a reminder of that. Thank you.

Once that field is removed the whole struct is useless and EPC bank converges to an array. With an array the driver should be able reserve pages without a global lock.

Quoted 6 lines

I've started writing a patch to make all this happen and it is progressing really well. I'm planning to include this change to v6. As it simplifies code I'm going to squash it as part of the initial driver patch.

How does this sound?

Every sgx_epc_bank will have a bitmap array for reserved EPC.

Every unswapped sgx_encl_page will have a pointer containing physical address of the EPC page in the upper bits and bank number in the lower bits (like sgx_epc_page has now in the 'pa' field).

This layout does not require a global lock.

/Jarkko

#40 lore raw

Re: [intel-sgx-kernel-dev] [PATCH RFC v3 07/12] intel_sgx: driver for Intel Software Guard Extensions

Quoted 6 lines · Jarkko Sakkinen

I've started writing a patch to make all this happen and it is progressing really well. I'm planning to include this change to v6. As it simplifies code I'm going to squash it as part of the initial driver patch.

How does this sound?

Sounds sane to me.