Skip to contentSkip to conversation

Intel SGX1 support

PATCH Superseded Jarkko Sakkinen 8 participants 54 messages Jul 3, 2018 – Jul 28, 2018

Subsystems (6)
Documentation x86 KVM KVM x86 +2 more subsystems

v12 of 40 Superseded Latest is v40 →

lore
Share & tools
lore
#0 lore raw

v12 cover Intel SGX1 support

Intel(R) SGX is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the enclave is disallowed to access the memory inside the enclave by the CPU access control. In a way you can think that SGX provides inverted sandbox. It protects the application from a malicious host.

There is a new hardware unit in the processor called Memory Encryption Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define one or many MEE regions that can hold enclave data by configuring them with PRMRR registers.

The MEE automatically encrypts the data leaving the processor package to the MEE regions. The data is encrypted using a random key whose life-time is exactly one power cycle.

The current implementation requires that the firmware sets IA32_SGXLEPUBKEYHASH* MSRs as writable so that ultimately the kernel can decide what enclaves it wants run. The implementation does not create any bottlenecks to support read-only MSRs later on.

You can tell if your CPU supports SGX by looking into /proc/cpuinfo:

cat /proc/cpuinfo | grep sgx

v12:
* Split to more narrow scoped commits in order to ease the review process and use co-developed-by tag for co-authors of commits instead of listing them in the source files.
* Removed cruft EXPORT_SYMBOL() declarations and converted to static variables.
* Removed in-kernel LE i.e. this version of the SGX software stack only supports unlocked IA32_SGXLEPUBKEYHASHx MSRs.
* Refined documentation on launching enclaves, swapping and enclave construction.
* Refined sgx_arch.h to include alignment information for every struct that requires it and removed structs that are not needed without an LE.
* Got rid of SGX_CPUID.
* SGX detection now prints log messages about firmware configuration issues.

v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in sgx_page_cache_teardown(), which causes a leaked kthread after driver deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be invalidated when resuming, which will cause it not function properly anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols to LE even if “KASAN_SANITIZE := false” was set in the makefile.
* Fixed a regression in the #PF handler. If a page has SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail. It did not, which caused weird races when trying to change other parts of swapping code.
* EPC management has been refactored to a flat LRU cache and moved to arch/x86. The swapper thread reads a cluster of EPC pages and swaps all of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL when an enclave is already initialized or dead instead of zero.

v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.

v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode. The driver does not require anymore new exports.

v8:
* Check that public key MSRs match the LE public key hash in the driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch enclave and user enclaves by having a separate lock for hash calculation.

v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
a bit in order to better align with kernel conventions.

v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of struct sgx_epc_page instances there is an array of integers that encodes address and bank of an EPC page (the same data as 'pa' field earlier). The locking has been moved to the EPC bank level instead of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in every function.
* Style fixes based on Darren's comments to sgx_le.c.

v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.

v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.

v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.

v2:
* get_rand_uint32() changed the value of the pointer instead of value where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()

Jarkko Sakkinen (8):
x86/sgx: updated MAINTAINERS
x86/sgx: architectural structures
x86/sgx: data structures for tracking available EPC pages x86/sgx: wrappers for ENCLS opcode leaf functions x86/sgx: EPC page allocation routines platform/x86: Intel SGX driver
platform/x86: ptrace() support for the SGX driver x86/sgx: driver documentation

Kai Huang (1):
x86/sgx: add SGX definitions to cpufeature

Sean Christopherson (4):
x86/sgx: add SGX definitions to msr-index.h x86/cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX x86/sgx: detect Intel SGX
x86/sgx: sgx_einit()

 Documentation/index.rst                       |    1 +
 Documentation/x86/intel_sgx.rst               |  185 +++
 MAINTAINERS                                   |    7 +
 arch/x86/Kconfig                              |   19 +
 arch/x86/include/asm/cpufeature.h             |    7 +-
 arch/x86/include/asm/cpufeatures.h            |   10 +-
 arch/x86/include/asm/disabled-features.h      |    3 +-
 arch/x86/include/asm/msr-index.h              |    8 +
 arch/x86/include/asm/required-features.h      |    3 +-
 arch/x86/include/asm/sgx.h                    |  246 ++++
 arch/x86/include/asm/sgx_arch.h               |  202 ++++
 arch/x86/include/asm/sgx_pr.h                 |   29 +
 arch/x86/include/uapi/asm/sgx.h               |  138 +++
 arch/x86/kernel/cpu/Makefile                  |    1 +
 arch/x86/kernel/cpu/common.c                  |    7 +
 arch/x86/kernel/cpu/intel_sgx.c               |  460 ++++++++
 arch/x86/kvm/cpuid.h                          |    1 +
 drivers/platform/x86/Kconfig                  |    2 +
 drivers/platform/x86/Makefile                 |    1 +
 drivers/platform/x86/intel_sgx/Kconfig        |   22 +
 drivers/platform/x86/intel_sgx/Makefile       |   13 +
 drivers/platform/x86/intel_sgx/sgx.h          |  157 +++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 1011 +++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    |  286 +++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    |  151 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |  231 ++++
 drivers/platform/x86/intel_sgx/sgx_main.c     |  298 +++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      |  174 +++
 28 files changed, 3668 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/x86/intel_sgx.rst
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_arch.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c
Signature · 1 line
2.17.1
#1 lore raw

PATCH v12 1/13 x86/sgx: updated MAINTAINERS

Updated the MAINTAINERS file to point to the Intel SGX subsystem tree.

---
 MAINTAINERS | 7 +++++++
 1 file changed, 7 insertions(+)
MAINTAINERS +7 -0
index ca4afd68530c..12f7c68378e8 100644
@@ -7329,6 +7329,13 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/gpio/gpio-intel-mid.c
 
+INTEL SGX
+M:	Jarkko Sakkinen <[email protected]>
+L:	[email protected]
+Q:	https://patchwork.kernel.org/project/intel-sgx/list/
+F:	drivers/platform/x86/intel_sgx/
+K:	\bSGX_
+
 INVENSENSE MPU-3050 GYROSCOPE DRIVER
 M:	Linus Walleij <[email protected]>
 L:	[email protected]
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v12 01/13] x86/sgx: updated MAINTAINERS

Quoted 1 line · Jarkko Sakkinen

Updated the MAINTAINERS file to point to the Intel SGX subsystem tree.

Update not Updated. Documentation says:

Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour.

Applies to later patches as well.

Thanks,

tglx

#3 lore raw

Re: [PATCH v12 01/13] x86/sgx: updated MAINTAINERS

Quoted 11 lines · Thomas Gleixner
Quoted 1 line · Jarkko Sakkinen

Updated the MAINTAINERS file to point to the Intel SGX subsystem tree.

Update not Updated. Documentation says:

Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour.

Quoted 5 lines

Applies to later patches as well.

Thanks,

tglx

Thanks I'll refine to follow this form.

/Jarkko

#4 lore raw

PATCH v12 2/13 x86/sgx: add SGX definitions to cpufeature

In reply to Jarkko Sakkinen

From: Kai Huang <[email protected]>

Added X86_FEATURE_SGX and X86_FEATURE_SGX_LC definitions that define the bits CPUID level 7 bits for determining whether the CPU supports SGX and launch configuration other than the Intel proprietary key. If this the case, IA32_SGXLEPUBKEYHASHn MSRs (0 < n < 4) are available for defining the root key for enclaves.

---
 arch/x86/include/asm/cpufeatures.h | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/include/asm/cpufeatures.h +2 -0
index fb00a2fca990..54d5269e1b86 100644
@@ -234,6 +234,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */
 #define X86_FEATURE_FSGSBASE		( 9*32+ 0) /* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/
 #define X86_FEATURE_TSC_ADJUST		( 9*32+ 1) /* TSC adjustment MSR 0x3B */
+#define X86_FEATURE_SGX			( 9*32+ 2) /* Software Guard Extensions */
 #define X86_FEATURE_BMI1		( 9*32+ 3) /* 1st group bit manipulation extensions */
 #define X86_FEATURE_HLE			( 9*32+ 4) /* Hardware Lock Elision */
 #define X86_FEATURE_AVX2		( 9*32+ 5) /* AVX2 instructions */
@@ -327,6 +328,7 @@
 #define X86_FEATURE_LA57		(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID		(16*32+22) /* RDPID instruction */
 #define X86_FEATURE_CLDEMOTE		(16*32+25) /* CLDEMOTE instruction */
+#define X86_FEATURE_SGX_LC		(16*32+30) /* supports SGX launch configuration */
 
 /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */
 #define X86_FEATURE_OVERFLOW_RECOV	(17*32+ 0) /* MCA overflow recovery support */
Signature · 1 line
2.17.1
#5 lore raw

Re: [PATCH v12 02/13] x86/sgx: add SGX definitions to cpufeature

On Tue, 3 Jul 2018, Jarkko Sakkinen wrote:

Subject: x86/sgx: add SGX definitions to cpufeature

The prefix is bogus. Not everything you touch is SGX just because.

The proper prefix is 'x86/cpufeatures:'
The following sentence starts with an uppercase letter.

And with the proper prefix the subject wants to be something like:

x86/cpufeatures: Add SGX and SGX_LC CPU features

Quoted 3 lines

From: Kai Huang <[email protected]>

Added X86_FEATURE_SGX and X86_FEATURE_SGX_LC definitions that define the

Add the ...

-> definitions that define

That's redundant.

Quoted 1 line

bits CPUID level 7 bits for determining whether the CPU supports SGX and

bits ... bits ??

Quoted 5 lines

launch configuration other than the Intel proprietary key. If this the case, IA32_SGXLEPUBKEYHASHn MSRs (0 < n < 4) are available for defining the root key for enclaves.

Lacks your SOB

Thanks,

tglx

#6 lore raw

Re: [PATCH v12 02/13] x86/sgx: add SGX definitions to cpufeature

Quoted 31 lines · Thomas Gleixner

On Tue, 3 Jul 2018, Jarkko Sakkinen wrote:

Subject: x86/sgx: add SGX definitions to cpufeature

The prefix is bogus. Not everything you touch is SGX just because.

The proper prefix is 'x86/cpufeatures:'
The following sentence starts with an uppercase letter.

And with the proper prefix the subject wants to be something like:

x86/cpufeatures: Add SGX and SGX_LC CPU features

Quoted 3 lines

From: Kai Huang <[email protected]>

Added X86_FEATURE_SGX and X86_FEATURE_SGX_LC definitions that define the

Add the ...

-> definitions that define

That's redundant.

Quoted 1 line

bits CPUID level 7 bits for determining whether the CPU supports SGX and

bits ... bits ??

Quoted 4 lines

launch configuration other than the Intel proprietary key. If this the case, IA32_SGXLEPUBKEYHASHn MSRs (0 < n < 4) are available for defining the root key for enclaves.

Quoted 7 lines
Quoted 1 line

Lacks your SOB

Thanks,

tglx

Thanks, will refine!

/Jarkko

#7 lore raw

PATCH v12 3/13 x86/sgx: add SGX definitions to msr-index.h

In reply to Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

ENCLS and ENCLU are usable if and only if SGX_ENABLE is set and After SGX is activated the IA32_SGXLEPUBKEYHASHn MSRs are writable if SGX_LC_WR is set and the feature control is locked.

SGX related bits in IA32_FEATURE_CONTROL cannot be set before SGX is activated by the pre-boot firmware. SGX activation is triggered by setting bit 0 in the MSR 0x7a. Until SGX is activated, the LE hash MSRs are writable to allow pre-boot firmware to lock down the LE root key with a non-Intel value.

---
 arch/x86/include/asm/msr-index.h | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/include/asm/msr-index.h +8 -0
index fda2114197b3..a7355fb7344f 100644
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)
 
 #define MSR_IA32_APICBASE		0x0000001b
@@ -545,6 +547,12 @@
 #define PACKAGE_THERM_INT_LOW_ENABLE		(1 << 1)
 #define PACKAGE_THERM_INT_PLN_ENABLE		(1 << 24)
 
+/* Intel SGX MSRs */
+#define MSR_IA32_SGXLEPUBKEYHASH0	0x0000008C
+#define MSR_IA32_SGXLEPUBKEYHASH1	0x0000008D
+#define MSR_IA32_SGXLEPUBKEYHASH2	0x0000008E
+#define MSR_IA32_SGXLEPUBKEYHASH3	0x0000008F
+
 /* Thermal Thresholds Support */
 #define THERM_INT_THRESHOLD0_ENABLE    (1 << 15)
 #define THERM_SHIFT_THRESHOLD0        8
Signature · 1 line
2.17.1
#8 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

arch/x86/include/asm/msr-index.h line 484

Quoted 9 lines · Jarkko Sakkinen
arch/x86/include/asm/msr-index.h +2 -0
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)

Have you run checkpatch.pl on these? There's a tabs-vs-spaces issue here.

#9 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

Quoted 12 lines · Dave Hansen

arch/x86/include/asm/msr-index.h line 484

Quoted 9 lines · Jarkko Sakkinen
arch/x86/include/asm/msr-index.h +2 -0
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)

Have you run checkpatch.pl on these? There's a tabs-vs-spaces issue here.

Thought I did. Do not know how that has slipped there but I will fix it. Thanks.

/Jarkko

#10 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

In reply to Dave Hansen
Quoted 12 lines · Dave Hansen

arch/x86/include/asm/msr-index.h line 484

Quoted 9 lines · Jarkko Sakkinen
arch/x86/include/asm/msr-index.h +2 -0
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)

Have you run checkpatch.pl on these? There's a tabs-vs-spaces issue here.

High portion of lines in this file are over 80 characters and have this same incosistent spacing. I'm not sure if this should be fixed bacause it would be consistent with the formatting that this file uses...

/Jarkko

#11 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

Quoted 14 lines · Jarkko Sakkinen
Quoted 12 lines · Dave Hansen

arch/x86/include/asm/msr-index.h line 484

Quoted 9 lines · Jarkko Sakkinen
arch/x86/include/asm/msr-index.h +2 -0
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)

Have you run checkpatch.pl on these? There's a tabs-vs-spaces issue here.

Quoted 3 lines

High portion of lines in this file are over 80 characters and have this same incosistent spacing. I'm not sure if this should be fixed bacause it would be consistent with the formatting that this file uses...

Ignore! I was looking at the cpufeature patch, this one really has a formatting issue (apologies).

/Jarkko

#12 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

In reply to Jarkko Sakkinen
Quoted 12 lines · Jarkko Sakkinen
Quoted 11 lines · Dave Hansen

arch/x86/include/asm/msr-index.h line 484

Quoted 9 lines · Jarkko Sakkinen
arch/x86/include/asm/msr-index.h +2 -0
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
 #define FEATURE_CONTROL_LMCE				(1<<20)

Have you run checkpatch.pl on these? There's a tabs-vs-spaces issue here.

Quoted 3 lines

High portion of lines in this file are over 80 characters and have this same incosistent spacing. I'm not sure if this should be fixed bacause it would be consistent with the formatting that this file uses...

Why does that matter?

Just make sure your patch doesn't introduce any new instances of bad form unless it's *required* to fit into the file.

#13 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

In reply to Jarkko Sakkinen

On Tue, 3 Jul 2018, Jarkko Sakkinen wrote:

x86/msr: Add ....

Quoted 6 lines

From: Sean Christopherson <[email protected]>

ENCLS and ENCLU are usable if and only if SGX_ENABLE is set and After SGX is activated the IA32_SGXLEPUBKEYHASHn MSRs are writable if SGX_LC_WR is set and the feature control is locked.

Quoted 8 lines

SGX related bits in IA32_FEATURE_CONTROL cannot be set before SGX is activated by the pre-boot firmware. SGX activation is triggered by setting bit 0 in the MSR 0x7a. Until SGX is activated, the LE hash MSRs are writable to allow pre-boot firmware to lock down the LE root key with a non-Intel value.

Lacks SOB

arch/x86/include/asm/msr-index.h line 482

Quoted 13 lines
---
 arch/x86/include/asm/msr-index.h | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index fda2114197b3..a7355fb7344f 100644
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)

Tabs not spaces please. checkpatch.pl would have told you

arch/x86/include/asm/msr-index.h line 483

Quoted 1 line
arch/x86/include/asm/msr-index.h +1 -0
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)

Thanks,

tglx

#14 lore raw

Re: [PATCH v12 03/13] x86/sgx: add SGX definitions to msr-index.h

Quoted 37 lines · Thomas Gleixner

On Tue, 3 Jul 2018, Jarkko Sakkinen wrote:

x86/msr: Add ....

Quoted 6 lines

From: Sean Christopherson <[email protected]>

ENCLS and ENCLU are usable if and only if SGX_ENABLE is set and After SGX is activated the IA32_SGXLEPUBKEYHASHn MSRs are writable if SGX_LC_WR is set and the feature control is locked.

Quoted 8 lines

SGX related bits in IA32_FEATURE_CONTROL cannot be set before SGX is activated by the pre-boot firmware. SGX activation is triggered by setting bit 0 in the MSR 0x7a. Until SGX is activated, the LE hash MSRs are writable to allow pre-boot firmware to lock down the LE root key with a non-Intel value.

Lacks SOB

arch/x86/include/asm/msr-index.h line 482

Quoted 13 lines
---
 arch/x86/include/asm/msr-index.h | 8 ++++++++
 1 file changed, 8 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index fda2114197b3..a7355fb7344f 100644
@@ -479,6 +479,8 @@
 #define FEATURE_CONTROL_LOCKED				(1<<0)
 #define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
 #define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
+#define FEATURE_CONTROL_SGX_ENABLE                      (1<<18)

Tabs not spaces please. checkpatch.pl would have told you

Quoted 5 lines

arch/x86/include/asm/msr-index.h line 483

Quoted 1 line
arch/x86/include/asm/msr-index.h +1 -0
+#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)

Thanks,

tglx

Thanks, will refine!

/Jarkko

#15 lore raw

PATCH v12 4/13 x86/cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX

In reply to Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

CPUID_12_EAX is an Intel-defined feature bits leaf dedicated for SGX. There are currently four documented feature bits, with more expected in the not-too-distant future.

---
 arch/x86/include/asm/cpufeature.h        | 7 +++++--
 arch/x86/include/asm/cpufeatures.h       | 8 +++++++-
 arch/x86/include/asm/disabled-features.h | 3 ++-
 arch/x86/include/asm/required-features.h | 3 ++-
 arch/x86/kernel/cpu/common.c             | 7 +++++++
 arch/x86/kvm/cpuid.h                     | 1 +
 6 files changed, 24 insertions(+), 5 deletions(-)
arch/x86/include/asm/cpufeature.h +5 -2
index aced6c9290d6..940f0c01d5f8 100644
@@ -30,6 +30,7 @@ enum cpuid_leafs
 	CPUID_7_ECX,
 	CPUID_8000_0007_EBX,
 	CPUID_7_EDX,
+	CPUID_12_EAX,
 };
 
 #ifdef CONFIG_X86_FEATURE_NAMES
@@ -81,8 +82,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 19, feature_bit) ||	\
 	   REQUIRED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define DISABLED_MASK_BIT_SET(feature_bit)				\
 	 ( CHECK_BIT_IN_MASK_WORD(DISABLED_MASK,  0, feature_bit) ||	\
@@ -104,8 +106,9 @@ extern const char * const x86_bug_flags[NBUGINTS*32];
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 16, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 17, feature_bit) ||	\
 	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 18, feature_bit) ||	\
+	   CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 19, feature_bit) ||	\
 	   DISABLED_MASK_CHECK					  ||	\
-	   BUILD_BUG_ON_ZERO(NCAPINTS != 19))
+	   BUILD_BUG_ON_ZERO(NCAPINTS != 20))
 
 #define cpu_has(c, bit)							\
 	(__builtin_constant_p(bit) && REQUIRED_MASK_BIT_SET(bit) ? 1 :	\
arch/x86/include/asm/cpufeatures.h +7 -1
index 54d5269e1b86..60ebf79062f0 100644
@@ -13,7 +13,7 @@
 /*
  * Defines x86 CPU feature bits
  */
-#define NCAPINTS			19	   /* N 32-bit words worth of info */
+#define NCAPINTS			20	   /* N 32-bit words worth of info */
 #define NBUGINTS			1	   /* N 32-bit bug flags */
 
 /*
@@ -344,6 +344,12 @@
 #define X86_FEATURE_ARCH_CAPABILITIES	(18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */
 #define X86_FEATURE_SPEC_CTRL_SSBD	(18*32+31) /* "" Speculative Store Bypass Disable */
 
+/* Intel SGX CPU features, CPUID level 0x000000012:0 (EAX), word 19 */
+#define X86_FEATURE_SGX1		(19*32+ 0) /* SGX1 leaf functions */
+#define X86_FEATURE_SGX2		(19*32+ 1) /* SGX2 leaf functions */
+#define X86_FEATURE_SGX_ENCLV		(19*32+ 5) /* SGX ENCLV instruction, leafs E[INC|DEC]VIRTCHILD, ESETCONTEXT */
+#define X86_FEATURE_SGX_ENCLS_C		(19*32+ 6) /* SGX ENCLS leafs ERDINFO, ETRACK, ELDBC and ELDUC */
+
 /*
  * BUG word(s)
  */
arch/x86/include/asm/disabled-features.h +2 -1
index 33833d1909af..c81b5d446a3e 100644
@@ -78,6 +78,7 @@
 #define DISABLED_MASK16	(DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP)
 #define DISABLED_MASK17	0
 #define DISABLED_MASK18	0
-#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define DISABLED_MASK19	0
+#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/required-features.h +2 -1
index 6847d85400a8..fa5700097f64 100644
@@ -101,6 +101,7 @@
 #define REQUIRED_MASK16	0
 #define REQUIRED_MASK17	0
 #define REQUIRED_MASK18	0
-#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
+#define REQUIRED_MASK19	0
+#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)
 
 #endif /* _ASM_X86_REQUIRED_FEATURES_H */
arch/x86/kernel/cpu/common.c +7 -0
index 38276f58d3bf..5e712fa6bf4f 100644
@@ -843,6 +843,13 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
 		}
 	}
 
+	/* Intel SGX features: level 0x00000012 */
+	if (c->cpuid_level >= 0x00000012) {
+		cpuid(0x00000012, &eax, &ebx, &ecx, &edx);
+
+		c->x86_capability[CPUID_12_EAX] = eax;
+	}
+
 	/* AMD-defined flags: level 0x80000001 */
 	eax = cpuid_eax(0x80000000);
 	c->extended_cpuid_level = eax;
arch/x86/kvm/cpuid.h +1 -0
index 9a327d5b6d1f..669c1774afdb 100644
@@ -55,6 +55,7 @@ static const struct cpuid_reg reverse_cpuid[] = {
 	[CPUID_7_ECX]         = {         7, 0, CPUID_ECX},
 	[CPUID_8000_0007_EBX] = {0x80000007, 0, CPUID_EBX},
 	[CPUID_7_EDX]         = {         7, 0, CPUID_EDX},
+	[CPUID_12_EAX]        = {        12, 0, CPUID_EAX},
 };
 
 static __always_inline struct cpuid_reg x86_feature_cpuid(unsigned x86_feature)
Signature · 1 line
2.17.1
#16 lore raw

Re: [PATCH v12 04/13] x86/cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX

Quoted 7 lines · Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

CPUID_12_EAX is an Intel-defined feature bits leaf dedicated for SGX. There are currently four documented feature bits, with more expected in the not-too-distant future.

Lacks SOB

Thanks,

tglx

#17 lore raw

Re: [PATCH v12 04/13] x86/cpufeatures: add Intel-defined SGX leaf CPUID_12_EAX

Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

CPUID_12_EAX is an Intel-defined feature bits leaf dedicated for SGX. There are currently four documented feature bits, with more expected in the not-too-distant future.

Quoted 7 lines
Quoted 1 line

Lacks SOB

Thanks,

tglx

Thanks, will refine!

/Jarkko

#18 lore raw

PATCH v12 5/13 x86/sgx: architectural structures

In reply to Jarkko Sakkinen

This commit adds arch/x86/include/asm/sgx_arch.h that contains definitions for data structures used by the SGX.

---
 arch/x86/include/asm/sgx_arch.h | 183 ++++++++++++++++++++++++++++++++
 1 file changed, 183 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx_arch.h
arch/x86/include/asm/sgx_arch.h new file +183 -0
new file mode 100644
index 000000000000..41a37eaa3f51
@@ -0,0 +1,183 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+
+#ifndef _ASM_X86_SGX_ARCH_H
+#define _ASM_X86_SGX_ARCH_H
+
+#include <linux/types.h>
+
+#define SGX_CPUID 0x12
+
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};
+
+#define SGX_SSA_GPRS_SIZE		182
+#define SGX_SSA_MISC_EXINFO_SIZE	16
+
+enum sgx_misc {
+	SGX_MISC_EXINFO		= 0x01,
+};
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+enum sgx_attribute {
+	SGX_ATTR_DEBUG		= 0x02,
+	SGX_ATTR_MODE64BIT	= 0x04,
+	SGX_ATTR_PROVISIONKEY	= 0x10,
+	SGX_ATTR_EINITTOKENKEY	= 0x20,
+};
+
+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L
+
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);
+
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+struct sgx_tcs {
+	uint64_t state;
+	uint64_t flags;
+	uint64_t ossa;
+	uint32_t cssa;
+	uint32_t nssa;
+	uint64_t oentry;
+	uint64_t aep;
+	uint64_t ofsbase;
+	uint64_t ogsbase;
+	uint32_t fslimit;
+	uint32_t gslimit;
+	uint64_t reserved[503];
+} __packed __aligned(4096);
+
+struct sgx_pageinfo {
+	uint64_t linaddr;
+	uint64_t srcpge;
+	union {
+		uint64_t secinfo;
+		uint64_t pcmd;
+	};
+	uint64_t secs;
+} __packed __aligned(32);
+
+
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L
+
+enum sgx_page_type {
+	SGX_PAGE_TYPE_SECS	= 0x00,
+	SGX_PAGE_TYPE_TCS	= 0x01,
+	SGX_PAGE_TYPE_REG	= 0x02,
+	SGX_PAGE_TYPE_VA	= 0x03,
+	SGX_PAGE_TYPE_TRIM	= 0x04,
+};
+
+enum sgx_secinfo_flags {
+	SGX_SECINFO_R		= 0x01,
+	SGX_SECINFO_W		= 0x02,
+	SGX_SECINFO_X		= 0x04,
+	SGX_SECINFO_SECS	= (SGX_PAGE_TYPE_SECS << 8),
+	SGX_SECINFO_TCS		= (SGX_PAGE_TYPE_TCS << 8),
+	SGX_SECINFO_REG		= (SGX_PAGE_TYPE_REG << 8),
+	SGX_SECINFO_VA          = (SGX_PAGE_TYPE_VA << 8),
+	SGX_SECINFO_TRIM	= (SGX_PAGE_TYPE_TRIM << 8),
+};
+
+struct sgx_secinfo {
+	uint64_t flags;
+	uint64_t reserved[7];
+} __packed __aligned(64);
+
+struct sgx_pcmd {
+	struct sgx_secinfo secinfo;
+	uint64_t enclave_id;
+	uint8_t reserved[40];
+	uint8_t mac[16];
+} __packed __aligned(128);
+
+#define SGX_MODULUS_SIZE 384
+
+struct sgx_sigstruct_header {
+	uint64_t header1[2];
+	uint32_t vendor;
+	uint32_t date;
+	uint64_t header2[2];
+	uint32_t swdefined;
+	uint8_t reserved1[84];
+} __packed;
+
+struct sgx_sigstruct_body {
+	uint32_t miscselect;
+	uint32_t miscmask;
+	uint8_t reserved2[20];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t attributemask[16];
+	uint8_t mrenclave[32];
+	uint8_t reserved3[32];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+} __packed;
+
+struct sgx_sigstruct {
+	struct sgx_sigstruct_header header;
+	uint8_t modulus[SGX_MODULUS_SIZE];
+	uint32_t exponent;
+	uint8_t signature[SGX_MODULUS_SIZE];
+	struct sgx_sigstruct_body body;
+	uint8_t reserved4[12];
+	uint8_t q1[SGX_MODULUS_SIZE];
+	uint8_t q2[SGX_MODULUS_SIZE];
+} __packed __aligned(4096);
+
+struct sgx_einittoken_payload {
+	uint32_t valid;
+	uint32_t reserved1[11];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[32];
+} __packed;
+
+struct sgx_einittoken {
+	struct sgx_einittoken_payload payload;
+	uint8_t cpusvnle[16];
+	uint16_t isvprodidle;
+	uint16_t isvsvnle;
+	uint8_t reserved2[24];
+	uint32_t maskedmiscselectle;
+	uint64_t maskedattributesle;
+	uint64_t maskedxfrmle;
+	uint8_t keyid[32];
+	uint8_t mac[16];
+} __packed __aligned(512);
+
+#endif /* _ASM_X86_SGX_ARCH_H */
Signature · 1 line
2.17.1
#19 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 16 lines · Jarkko Sakkinen

This commit adds arch/x86/include/asm/sgx_arch.h that contains definitions for data structures used by the SGX.

---
 arch/x86/include/asm/sgx_arch.h | 183 ++++++++++++++++++++++++++++++++
 1 file changed, 183 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx_arch.h
arch/x86/include/asm/sgx_arch.h new file +0 -0
new file mode 100644
index 000000000000..41a37eaa3f51

arch/x86/include/asm/sgx_arch.h line 2

Quoted 2 lines
arch/x86/include/asm/sgx_arch.h new file +2 -0
@@ -0,0 +1,183 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.

It would also be anice to have a description of what this contains. "Don't put software data structures in here, or else"

arch/x86/include/asm/sgx_arch.h line 9

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+#include <linux/types.h>
+
+#define SGX_CPUID 0x12

Shouldn't you be introducing this earlier and using it here?

+ /* Intel SGX features: level 0x00000012 */
+ if (c->cpuid_level >= 0x00000012) {
+ cpuid(0x00000012, &eax, &ebx, &ecx, &edx);
+
+ c->x86_capability[CPUID_12_EAX] = eax;
+ }

Quoted 35 lines
arch/x86/include/asm/sgx_arch.h new file +35 -0
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};
+
+#define SGX_SSA_GPRS_SIZE		182
+#define SGX_SSA_MISC_EXINFO_SIZE	16
+
+enum sgx_misc {
+	SGX_MISC_EXINFO		= 0x01,
+};
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+enum sgx_attribute {
+	SGX_ATTR_DEBUG		= 0x02,
+	SGX_ATTR_MODE64BIT	= 0x04,
+	SGX_ATTR_PROVISIONKEY	= 0x10,
+	SGX_ATTR_EINITTOKENKEY	= 0x20,
+};
+
+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L
+
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

It would be nice to align these a _bit_, like:

arch/x86/include/asm/sgx_arch.h line 54

Quoted 5 lines
arch/x86/include/asm/sgx_arch.h new file +5 -0
+	uint32_t mrsigner[8];
+	uint8_t	 reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t  reserved4[SGX_SECS_RESERVED4_SIZE];

Is the SECS *defined* to be a single page, or can it be bigger?

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+struct sgx_tcs {
+	uint64_t state;
+	uint64_t flags;
+	uint64_t ossa;

arch/x86/include/asm/sgx_arch.h line 76

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint32_t cssa;
+	uint32_t nssa;
+	uint64_t oentry;
+	uint64_t aep;
+	uint64_t ofsbase;
+	uint64_t ogsbase;
+	uint32_t fslimit;
+	uint32_t gslimit;
+	uint64_t reserved[503];
+} __packed __aligned(4096);

It's interesting that you defined a reserved[] array here with a hard-coded number and a (nicely-aligned, I'll point out) uint64_t, but the sgx_secs was defined using #defined reserve sizes and a uint8_t. Seems like we should just be consistent. No?

Also, are these truly structures that need alignment? Or are they just the *format* of a page (which is obviously aligned)? For instance, if we copied the hardware structure into a temporary buffer, we wouldn't need it aligned.

I sometimes prefer asking for an instance of a variable to be aligned rather than the type for things like this.

arch/x86/include/asm/sgx_arch.h line 86

Quoted 9 lines
arch/x86/include/asm/sgx_arch.h new file +9 -0
+struct sgx_pageinfo {
+	uint64_t linaddr;
+	uint64_t srcpge;
+	union {
+		uint64_t secinfo;
+		uint64_t pcmd;
+	};
+	uint64_t secs;
+} __packed __aligned(32);

I see these were verbatim names taken from the SDM. Could you also please add some small comments about them? "secs", for instance, is a pretty common abbreviation for "seconds". It would really help to have this at _least_ say:

	uint64_t sgx_secs;
Quoted 24 lines
arch/x86/include/asm/sgx_arch.h new file +24 -0
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L
+
+enum sgx_page_type {
+	SGX_PAGE_TYPE_SECS	= 0x00,
+	SGX_PAGE_TYPE_TCS	= 0x01,
+	SGX_PAGE_TYPE_REG	= 0x02,
+	SGX_PAGE_TYPE_VA	= 0x03,
+	SGX_PAGE_TYPE_TRIM	= 0x04,
+};
+
+enum sgx_secinfo_flags {
+	SGX_SECINFO_R		= 0x01,
+	SGX_SECINFO_W		= 0x02,
+	SGX_SECINFO_X		= 0x04,
+	SGX_SECINFO_SECS	= (SGX_PAGE_TYPE_SECS << 8),
+	SGX_SECINFO_TCS		= (SGX_PAGE_TYPE_TCS << 8),
+	SGX_SECINFO_REG		= (SGX_PAGE_TYPE_REG << 8),
+	SGX_SECINFO_VA          = (SGX_PAGE_TYPE_VA << 8),
+	SGX_SECINFO_TRIM	= (SGX_PAGE_TYPE_TRIM << 8),
+};
+
+struct sgx_secinfo {

arch/x86/include/asm/sgx_arch.h line 122

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t flags;
+	uint64_t reserved[7];
+} __packed __aligned(64);
+
+struct sgx_pcmd {
+	struct sgx_secinfo secinfo;
+	uint64_t enclave_id;
+	uint8_t reserved[40];
+	uint8_t mac[16];
+} __packed __aligned(128);

I don't see any sign of this alignment restriction in the SDM.

arch/x86/include/asm/sgx_arch.h line 124

Quoted 1 line
arch/x86/include/asm/sgx_arch.h new file +1 -0
+#define SGX_MODULUS_SIZE 384

This is #defined in a rather odd place. Why here? Also, please add a unit. SGX_MODULUS_SIZE_BYTES or SGX_MODULUS_BYTES is way better than _SIZE.

Quoted 6 lines
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_sigstruct_header {
+	uint64_t header1[2];
+	uint32_t vendor;
+	uint32_t date;
+	uint64_t header2[2];
+	uint32_t swdefined;

arch/x86/include/asm/sgx_arch.h line 141

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint8_t reserved1[84];
+} __packed;
+
+struct sgx_sigstruct_body {
+	uint32_t miscselect;
+	uint32_t miscmask;
+	uint8_t reserved2[20];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t attributemask[16];

I've hit my limit on silly SDM names. :)

Please make these human-readable: "attribute_mask". The divergence from the SDM naming is justified at this point.

Quoted 6 lines
arch/x86/include/asm/sgx_arch.h new file +6 -0
+	uint8_t mrenclave[32];
+	uint8_t reserved3[32];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+} __packed;
+

arch/x86/include/asm/sgx_arch.h line 157

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+struct sgx_sigstruct {
+	struct sgx_sigstruct_header header;
+	uint8_t modulus[SGX_MODULUS_SIZE];
+	uint32_t exponent;
+	uint8_t signature[SGX_MODULUS_SIZE];
+	struct sgx_sigstruct_body body;
+	uint8_t reserved4[12];
+	uint8_t q1[SGX_MODULUS_SIZE];
+	uint8_t q2[SGX_MODULUS_SIZE];
+} __packed __aligned(4096);

It's interesting that the SDM says "page aligned" in some places and "4096-byte aligned" in others. Oh well.

arch/x86/include/asm/sgx_arch.h line 161

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+struct sgx_einittoken_payload {
+	uint32_t valid;
+	uint32_t reserved1[11];

... and back to different types being used for padding. These really need to at least be consistent.

arch/x86/include/asm/sgx_arch.h line 168

Quoted 7 lines
arch/x86/include/asm/sgx_arch.h new file +7 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[32];
+} __packed;

Just curious, why is this broken out into its own structure? It doesn't appear architectural.

Quoted 15 lines
arch/x86/include/asm/sgx_arch.h new file +14 -0
+struct sgx_einittoken {
+	struct sgx_einittoken_payload payload;
+	uint8_t cpusvnle[16];
+	uint16_t isvprodidle;
+	uint16_t isvsvnle;
+	uint8_t reserved2[24];
+	uint32_t maskedmiscselectle;
+	uint64_t maskedattributesle;
+	uint64_t maskedxfrmle;
+	uint8_t keyid[32];
+	uint8_t mac[16];
+} __packed __aligned(512);
+
+#endif /* _ASM_X86_SGX_ARCH_H */
#20 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 11 lines · Dave Hansen
Quoted 10 lines · Jarkko Sakkinen

This commit adds arch/x86/include/asm/sgx_arch.h that contains definitions for data structures used by the SGX.

---
 arch/x86/include/asm/sgx_arch.h | 183 ++++++++++++++++++++++++++++++++
 1 file changed, 183 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx_arch.h
Quoted 11 lines

arch/x86/include/asm/sgx_arch.h line 2

Quoted 8 lines
arch/x86/include/asm/sgx_arch.h new file +2 -0
new file mode 100644
index 000000000000..41a37eaa3f51
@@ -0,0 +1,183 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.

It would also be anice to have a description of what this contains. "Don't put software data structures in here, or else"

Agreed.

Review

Quoted 13 lines

arch/x86/include/asm/sgx_arch.h line 9

Quoted 4 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+#include <linux/types.h>
+
+#define SGX_CPUID 0x12

Shouldn't you be introducing this earlier and using it here?

(quoted diff) +6 -0
+	/* Intel SGX features: level 0x00000012 */
+	if (c->cpuid_level >= 0x00000012) {
+		cpuid(0x00000012, &eax, &ebx, &ecx, &edx);
+
+		c->x86_capability[CPUID_12_EAX] = eax;
+	}

Definitely.

Quoted 49 lines
Quoted 36 lines
arch/x86/include/asm/sgx_arch.h new file +35 -0
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};
+
+#define SGX_SSA_GPRS_SIZE		182
+#define SGX_SSA_MISC_EXINFO_SIZE	16
+
+enum sgx_misc {
+	SGX_MISC_EXINFO		= 0x01,
+};
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+enum sgx_attribute {
+	SGX_ATTR_DEBUG		= 0x02,
+	SGX_ATTR_MODE64BIT	= 0x04,
+	SGX_ATTR_PROVISIONKEY	= 0x10,
+	SGX_ATTR_EINITTOKENKEY	= 0x20,
+};
+
+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L
+
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

It would be nice to align these a _bit_, like:

Quoted 5 lines

arch/x86/include/asm/sgx_arch.h line 54

Quoted 5 lines
arch/x86/include/asm/sgx_arch.h new file +5 -0
+	uint32_t mrsigner[8];
+	uint8_t	 reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t  reserved4[SGX_SECS_RESERVED4_SIZE];

My personal taste is not to align struct fields (I do align enums and such). Is this important? I would prefer to keep it how it is but I'm not going to argue about it too much...

Quoted 1 line

Is the SECS *defined* to be a single page, or can it be bigger?

It is always a single EPC page.

Quoted 17 lines
Quoted 17 lines
arch/x86/include/asm/sgx_arch.h new file +17 -0
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL
+
+struct sgx_tcs {
+	uint64_t state;
+	uint64_t flags;
+	uint64_t ossa;
+	uint32_t cssa;
+	uint32_t nssa;
+	uint64_t oentry;
+	uint64_t aep;
+	uint64_t ofsbase;
+	uint64_t ogsbase;
+	uint32_t fslimit;
Quoted 8 lines

arch/x86/include/asm/sgx_arch.h line 76

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+	uint32_t gslimit;
+	uint64_t reserved[503];
+} __packed __aligned(4096);

It's interesting that you defined a reserved[] array here with a hard-coded number and a (nicely-aligned, I'll point out) uint64_t, but the sgx_secs was defined using #defined reserve sizes and a uint8_t. Seems like we should just be consistent. No?

Yes :-)

Quoted 4 lines

Also, are these truly structures that need alignment? Or are they just the *format* of a page (which is obviously aligned)? For instance, if we copied the hardware structure into a temporary buffer, we wouldn't need it aligned.

You have good point here. For sgx_tcs and sgx_secs there should not be alignment at all as it is either in EPC or in a temporary buffer. I will remove alignments for those.

For structs such as secinfo it makes sense to have aligment info because you usually create them as local variables but CPU requires particular aligment (like 64 for secinfo).

Quoted 2 lines

I sometimes prefer asking for an instance of a variable to be aligned rather than the type for things like this.

This was a really good point, thanks.

Quoted 15 lines

arch/x86/include/asm/sgx_arch.h line 86

Quoted 9 lines
arch/x86/include/asm/sgx_arch.h new file +9 -0
+struct sgx_pageinfo {
+	uint64_t linaddr;
+	uint64_t srcpge;
+	union {
+		uint64_t secinfo;
+		uint64_t pcmd;
+	};
+	uint64_t secs;
+} __packed __aligned(32);

I see these were verbatim names taken from the SDM. Could you also please add some small comments about them? "secs", for instance, is a pretty common abbreviation for "seconds". It would really help to have this at _least_ say:

Quoted 1 line
	uint64_t sgx_secs;

I would go with comments as it has advantages when you maintain this code to use same fields names as SDM:

uint64_t secs; /* Enclave Control Structure */

I wonder why they had to put that 'S' in the front :-( One thing I would like to diverse from SDM is to use verbatim ECS as it is more consistent naming with TCS (Thread Control Structure) and pronouncing SECS has taken me into embarrassing situations in the past...

Quoted 28 lines
Quoted 28 lines
arch/x86/include/asm/sgx_arch.h new file +28 -0
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L
+
+enum sgx_page_type {
+	SGX_PAGE_TYPE_SECS	= 0x00,
+	SGX_PAGE_TYPE_TCS	= 0x01,
+	SGX_PAGE_TYPE_REG	= 0x02,
+	SGX_PAGE_TYPE_VA	= 0x03,
+	SGX_PAGE_TYPE_TRIM	= 0x04,
+};
+
+enum sgx_secinfo_flags {
+	SGX_SECINFO_R		= 0x01,
+	SGX_SECINFO_W		= 0x02,
+	SGX_SECINFO_X		= 0x04,
+	SGX_SECINFO_SECS	= (SGX_PAGE_TYPE_SECS << 8),
+	SGX_SECINFO_TCS		= (SGX_PAGE_TYPE_TCS << 8),
+	SGX_SECINFO_REG		= (SGX_PAGE_TYPE_REG << 8),
+	SGX_SECINFO_VA          = (SGX_PAGE_TYPE_VA << 8),
+	SGX_SECINFO_TRIM	= (SGX_PAGE_TYPE_TRIM << 8),
+};
+
+struct sgx_secinfo {
+	uint64_t flags;
+	uint64_t reserved[7];
+} __packed __aligned(64);
+
Quoted 8 lines

arch/x86/include/asm/sgx_arch.h line 122

Quoted 6 lines
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_pcmd {
+	struct sgx_secinfo secinfo;
+	uint64_t enclave_id;
+	uint8_t reserved[40];
+	uint8_t mac[16];
+} __packed __aligned(128);

I don't see any sign of this alignment restriction in the SDM.

Have to check whether it is needed or a missing item in the SDM.

Quoted 4 lines

arch/x86/include/asm/sgx_arch.h line 124

Quoted 1 line
arch/x86/include/asm/sgx_arch.h new file +1 -0
+#define SGX_MODULUS_SIZE 384

This is #defined in a rather odd place. Why here? Also, please add a unit. SGX_MODULUS_SIZE_BYTES or SGX_MODULUS_BYTES is way better than _SIZE.

Ok.

Quoted 19 lines
Quoted 6 lines
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_sigstruct_header {
+	uint64_t header1[2];
+	uint32_t vendor;
+	uint32_t date;
+	uint64_t header2[2];
+	uint32_t swdefined;

arch/x86/include/asm/sgx_arch.h line 141

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint8_t reserved1[84];
+} __packed;
+
+struct sgx_sigstruct_body {
+	uint32_t miscselect;
+	uint32_t miscmask;
+	uint8_t reserved2[20];
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t attributemask[16];

I've hit my limit on silly SDM names. :)

Quoted 2 lines

Please make these human-readable: "attribute_mask". The divergence from the SDM naming is justified at this point.

I can update my naming in a way that for these obvious things I use kernel naming style.

I would stick the SDM verbatim for some fields like MRENCLAVE because it is so widely used name also in research papers etc.

Quoted 27 lines
Quoted 6 lines
arch/x86/include/asm/sgx_arch.h new file +6 -0
+	uint8_t mrenclave[32];
+	uint8_t reserved3[32];
+	uint16_t isvprodid;
+	uint16_t isvsvn;
+} __packed;
+

arch/x86/include/asm/sgx_arch.h line 157

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+struct sgx_sigstruct {
+	struct sgx_sigstruct_header header;
+	uint8_t modulus[SGX_MODULUS_SIZE];
+	uint32_t exponent;
+	uint8_t signature[SGX_MODULUS_SIZE];
+	struct sgx_sigstruct_body body;
+	uint8_t reserved4[12];
+	uint8_t q1[SGX_MODULUS_SIZE];
+	uint8_t q2[SGX_MODULUS_SIZE];
+} __packed __aligned(4096);

It's interesting that the SDM says "page aligned" in some places and "4096-byte aligned" in others. Oh well.

arch/x86/include/asm/sgx_arch.h line 161

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+struct sgx_einittoken_payload {
+	uint32_t valid;
+	uint32_t reserved1[11];

... and back to different types being used for padding. These really need to at least be consistent.

Quoted 9 lines

arch/x86/include/asm/sgx_arch.h line 168

Quoted 7 lines
arch/x86/include/asm/sgx_arch.h new file +7 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint8_t mrenclave[32];
+	uint8_t reserved2[32];
+	uint8_t mrsigner[32];
+	uint8_t reserved3[32];
+} __packed;

Just curious, why is this broken out into its own structure? It doesn't appear architectural.

For token it does not make much sense. For sigstruct the token CMAC calculation is done over those inner structs if you run your own LE.

Quoted 7 lines
Quoted 7 lines
arch/x86/include/asm/sgx_arch.h new file +7 -0
+struct sgx_einittoken {
+	struct sgx_einittoken_payload payload;
+	uint8_t cpusvnle[16];
+	uint16_t isvprodidle;
+	uint16_t isvsvnle;
+	uint8_t reserved2[24];
+	uint32_t maskedmiscselectle;
Quoted 8 lines

arch/x86/include/asm/sgx_arch.h line 183

Quoted 8 lines
arch/x86/include/asm/sgx_arch.h new file +7 -0
+	uint64_t maskedattributesle;
+	uint64_t maskedxfrmle;
+	uint8_t keyid[32];
+	uint8_t mac[16];
+} __packed __aligned(512);
+
+#endif /* _ASM_X86_SGX_ARCH_H */

/Jarkko

#21 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

In reply to Jarkko Sakkinen
Quoted 1 line · Jarkko Sakkinen

This commit adds arch/x86/include/asm/sgx_arch.h that contains definitions

This is not a commit. Simply: Add .....

Quoted 4 lines

for data structures used by the SGX.

Documentaiton:

A Co-Developed-by: states that the patch was also created by another developer along with the original author. This is useful at times when multiple people work on a single patch. Note, this person also needs to have a Signed-off-by:
line in the patch as well.

Please fix all over the place.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL

L ? ULL I'd say...

Quoted 1 line

+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L

Ditto

arch/x86/include/asm/sgx_arch.h line 46

Quoted 12 lines
arch/x86/include/asm/sgx_arch.h new file +12 -0
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t attributes;

Please make these tabular aligned for readility sake

arch/x86/include/asm/sgx_arch.h line 46

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+	uint32_t	miscselect;
+	uint8_t		reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t	attributes;

Hmm? All over the place...

arch/x86/include/asm/sgx_arch.h line 58

Quoted 2 lines
arch/x86/include/asm/sgx_arch.h new file +2 -0
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */

Please do not use tail comments. Either put the comment above the define or use kernel doc.

arch/x86/include/asm/sgx_arch.h line 61

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL

See above.

arch/x86/include/asm/sgx_arch.h line 91

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L

Ditto

Thanks,

tglx

#22 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 11 lines · Thomas Gleixner
Quoted 1 line · Jarkko Sakkinen

This commit adds arch/x86/include/asm/sgx_arch.h that contains definitions

This is not a commit. Simply: Add .....

Quoted 4 lines

for data structures used by the SGX.

Quoted 8 lines

Documentaiton:

A Co-Developed-by: states that the patch was also created by another developer along with the original author. This is useful at times when multiple people work on a single patch. Note, this person also needs to have a Signed-off-by:
line in the patch as well.

Please fix all over the place.

I used this tag for the first time and thought that it was undocumented because checkpatch.pl complained about it. I will fix this.

Quoted 3 lines
Quoted 1 line

+#define SGX_MISC_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL

L ? ULL I'd say...

Yes, should be ULL.

Quoted 4 lines
Quoted 1 line

+#define SGX_ATTR_RESERVED_MASK 0xFFFFFFFFFFFFFFC9L

Ditto

Quoted 14 lines

arch/x86/include/asm/sgx_arch.h line 46

Quoted 12 lines
arch/x86/include/asm/sgx_arch.h new file +12 -0
+#define SGX_SECS_RESERVED1_SIZE 24
+#define SGX_SECS_RESERVED2_SIZE 32
+#define SGX_SECS_RESERVED3_SIZE 96
+#define SGX_SECS_RESERVED4_SIZE 3836
+
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t attributes;

Please make these tabular aligned for readility sake

Ok, so it is now +2 for this. I can do that if you really want that.

Quoted 24 lines

arch/x86/include/asm/sgx_arch.h line 46

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+	uint32_t	miscselect;
+	uint8_t		reserved1[SGX_SECS_RESERVED1_SIZE];
+	uint64_t	attributes;

Hmm? All over the place...

arch/x86/include/asm/sgx_arch.h line 58

Quoted 2 lines
arch/x86/include/asm/sgx_arch.h new file +2 -0
+enum sgx_tcs_flags {
+	SGX_TCS_DBGOPTIN	= 0x01, /* cleared on EADD */

Please do not use tail comments. Either put the comment above the define or use kernel doc.

arch/x86/include/asm/sgx_arch.h line 61

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+};
+
+#define SGX_TCS_RESERVED_MASK 0xFFFFFFFFFFFFFFFEL

See above.

arch/x86/include/asm/sgx_arch.h line 91

Quoted 3 lines
arch/x86/include/asm/sgx_arch.h new file +3 -0
+#define SGX_SECINFO_PERMISSION_MASK	0x0000000000000007L
+#define SGX_SECINFO_PAGE_TYPE_MASK	0x000000000000FF00L
+#define SGX_SECINFO_RESERVED_MASK	0xFFFFFFFFFFFF00F8L

Ditto

Quoted 3 lines

Thanks,

tglx

Thank you.

/Jarkko

#23 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

In reply to Jarkko Sakkinen

arch/x86/include/asm/sgx_arch.h line 15

Quoted 5 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +5 -0
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};

I made a review comment about these last time. I don't remember receiving any kind of response from you on it, or seeing it addressed in these patches in any kind of way.

I've also seen comments from Thomas that sound really familiar such as the lack of comments on the paging code or the use of a semaphore to protect things that don't sleep. I made very, very similar comments on the last posting.

Is there a reason that those comments were not addressed?

#24 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 11 lines · Dave Hansen

arch/x86/include/asm/sgx_arch.h line 15

Quoted 5 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +5 -0
+enum sgx_cpuid {
+	SGX_CPUID_CAPABILITIES	= 0,
+	SGX_CPUID_ATTRIBUTES	= 1,
+	SGX_CPUID_EPC_BANKS	= 2,
+};

I made a review comment about these last time. I don't remember receiving any kind of response from you on it, or seeing it addressed in these patches in any kind of way.

Quoted 6 lines

I've also seen comments from Thomas that sound really familiar such as the lack of comments on the paging code or the use of a semaphore to protect things that don't sleep. I made very, very similar comments on the last posting.

Is there a reason that those comments were not addressed?

My sloppiness is the reason.

The feedback has been awesome from you and Thomas for this version, which will help me to get these things right for the next version.

Thank you.

/Jarkko

#25 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

In reply to Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

Why are the uint* versions in use here? Those are for userspace ABI, but this is entirely for in-kernel-use, right?

We've used u8/16/32/64 in arch code in a bunch of places. They're at least a bit more compact and easier to read. It's this:

	u8	foo;
	u64	bar;

vs. this:

	uint8_t	 foo;
	uint64_t bar;
#26 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 24 lines · Dave Hansen
Quoted 6 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

Why are the uint* versions in use here? Those are for userspace ABI, but this is entirely for in-kernel-use, right?

We've used u8/16/32/64 in arch code in a bunch of places. They're at least a bit more compact and easier to read. It's this:

Quoted 7 lines
	u8	foo;
	u64	bar;

vs. this:

	uint8_t	 foo;
	uint64_t bar;

The reason was that with in-kernel LE these were in fact used by user space code. Now they can be changed to those that you suggested.

/Jarkko

#27 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 30 lines · Jarkko Sakkinen
Quoted 29 lines · Dave Hansen
Quoted 6 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_secs {
+	uint64_t size;
+	uint64_t base;
+	uint32_t ssaframesize;
+	uint32_t miscselect;
+	uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+	uint64_t attributes;
+	uint64_t xfrm;
+	uint32_t mrenclave[8];
+	uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+	uint32_t mrsigner[8];
+	uint8_t	reserved3[SGX_SECS_RESERVED3_SIZE];
+	uint16_t isvvprodid;
+	uint16_t isvsvn;
+	uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

Why are the uint* versions in use here? Those are for userspace ABI, but this is entirely for in-kernel-use, right?

We've used u8/16/32/64 in arch code in a bunch of places. They're at least a bit more compact and easier to read. It's this:

	u8	foo;
	u64	bar;

vs. this:

Quoted 8 lines
Quoted 2 lines
	uint8_t	 foo;
	uint64_t bar;

The reason was that with in-kernel LE these were in fact used by user space code. Now they can be changed to those that you suggested.

/Jarkko

For things exported to user space use __u* and __s* types... the _t types would actually violate the C standard with respect to namespace pollution.

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#28 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 31 lines · On Fri, Jul 6, 2018 at 12:50 AM, <[email protected]> wrote:
Quoted 30 lines · Jarkko Sakkinen
Quoted 29 lines · Dave Hansen
Quoted 6 lines · Jarkko Sakkinen
arch/x86/include/asm/sgx_arch.h new file +6 -0
+struct sgx_secs {
+  uint64_t size;
+  uint64_t base;
+  uint32_t ssaframesize;
+  uint32_t miscselect;
+  uint8_t reserved1[SGX_SECS_RESERVED1_SIZE];

arch/x86/include/asm/sgx_arch.h line 55

Quoted 10 lines
arch/x86/include/asm/sgx_arch.h new file +10 -0
+  uint64_t attributes;
+  uint64_t xfrm;
+  uint32_t mrenclave[8];
+  uint8_t reserved2[SGX_SECS_RESERVED2_SIZE];
+  uint32_t mrsigner[8];
+  uint8_t reserved3[SGX_SECS_RESERVED3_SIZE];
+  uint16_t isvvprodid;
+  uint16_t isvsvn;
+  uint8_t reserved4[SGX_SECS_RESERVED4_SIZE];
+} __packed __aligned(4096);

Why are the uint* versions in use here? Those are for userspace ABI, but this is entirely for in-kernel-use, right?

We've used u8/16/32/64 in arch code in a bunch of places. They're at least a bit more compact and easier to read. It's this:

     u8      foo;
     u64     bar;

vs. this:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
     uint8_t  foo;
     uint64_t bar;

The reason was that with in-kernel LE these were in fact used by user space code. Now they can be changed to those that you suggested.

For things exported to user space use __u* and __s* types... the _t types would actually violate the C standard with respect to namespace pollution.

Hmm... Coding style 5(d) allows to use uintNN_t in new code (as a variation of uNN choice).

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#29 lore raw

Re: [PATCH v12 05/13] x86/sgx: architectural structures

Quoted 6 lines · Andy Shevchenko
Quoted 1 line · On Fri, Jul 6, 2018 at 12:50 AM, <[email protected]> wrote:

For things exported to user space use __u* and __s* types... the _t types would actually violate the C standard with respect to namespace pollution.

Hmm... Coding style 5(d) allows to use uintNN_t in new code (as a variation of uNN choice).

But not for user space exposed headers. See 5(e) for that matter.

Thanks,

tglx

#30 lore raw

PATCH v12 6/13 x86/sgx: detect Intel SGX

In reply to Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

Intel(R) SGX is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the enclave is disallowed to access the memory inside the enclave by the CPU access control.

This commit adds the check for SGX to arch/x86 and a new config option, INTEL_SGX_CORE. Exposes a boolean variable 'sgx_enabled' to query whether or not the SGX support is available.

---
 arch/x86/Kconfig                | 19 ++++++++++++
 arch/x86/include/asm/sgx.h      | 12 ++++++++
 arch/x86/include/asm/sgx_pr.h   | 13 ++++++++
 arch/x86/kernel/cpu/Makefile    |  1 +
 arch/x86/kernel/cpu/intel_sgx.c | 54 +++++++++++++++++++++++++++++++++
 5 files changed, 99 insertions(+)
 create mode 100644 arch/x86/include/asm/sgx.h
 create mode 100644 arch/x86/include/asm/sgx_pr.h
 create mode 100644 arch/x86/kernel/cpu/intel_sgx.c
arch/x86/Kconfig +19 -0
index c07f492b871a..d2c7e7a2b2ac 100644
@@ -1925,6 +1925,25 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS
 
 	  If unsure, say y.
 
+config INTEL_SGX_CORE
+	prompt "Intel SGX core functionality"
+	def_bool n
+	depends on X86_64 && CPU_SUP_INTEL
+	help
+	Intel Software Guard eXtensions (SGX) is a set of CPU instructions
+	that allows ring 3 applications to create enclaves, private regions
+	of memory that are protected, by hardware, from unauthorized access
+	and/or modification.
+
+	This option enables kernel recognition of SGX, high-level management
+	of the Enclave Page Cache (EPC), tracking and writing of SGX Launch
+	Enclave Hash MSRs, and allows for virtualization of SGX via KVM. By
+	iteslf, this option does not provide SGX support to userspace.
+
+	For details, see Documentation/x86/intel_sgx.rst
+
+	If unsure, say N.
+
 config EFI
 	bool "EFI runtime service support"
 	depends on ACPI
arch/x86/include/asm/sgx.h new file +12 -0
new file mode 100644
index 000000000000..2130e639ab49
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-18 Intel Corporation.
+
+#ifndef _ASM_X86_SGX_H
+#define _ASM_X86_SGX_H
+
+#include <linux/types.h>
+
+extern bool sgx_enabled;
+extern bool sgx_lc_enabled;
+
+#endif /* _ASM_X86_SGX_H */
arch/x86/include/asm/sgx_pr.h new file +13 -0
new file mode 100644
index 000000000000..42a4185d4bc9
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+
+#ifndef _ASM_X86_SGX_PR_H
+#define _ASM_X86_SGX_PR_H
+
+#include <linux/printk.h>
+#include <linux/ratelimit.h>
+
+#undef pr_fmt
+#define pr_fmt(fmt) "intel_sgx: " fmt
+
+#endif /* _ASM_X86_SGX_PR_H */
arch/x86/kernel/cpu/Makefile +1 -0
index a66229f51b12..9552ff5b4ec3 100644
@@ -36,6 +36,7 @@ obj-$(CONFIG_CPU_SUP_TRANSMETA_32)	+= transmeta.o
 obj-$(CONFIG_CPU_SUP_UMC_32)		+= umc.o
 
 obj-$(CONFIG_INTEL_RDT)	+= intel_rdt.o intel_rdt_rdtgroup.o intel_rdt_monitor.o intel_rdt_ctrlmondata.o
+obj-$(CONFIG_INTEL_SGX_CORE)		+= intel_sgx.o
 
 obj-$(CONFIG_X86_MCE)			+= mcheck/
 obj-$(CONFIG_MTRR)			+= mtrr/
arch/x86/kernel/cpu/intel_sgx.c new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
2.17.1
#31 lore raw

Re: [PATCH v12 06/13] x86/sgx: detect Intel SGX

arch/x86/kernel/cpu/intel_sgx.c line 11

Quoted 12 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/intel_sgx.c new file +11 -0
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>

The common include ordering is

include linux/....

include asm/...

Quoted 31 lines
arch/x86/kernel/cpu/intel_sgx.c new file +29 -0
+
+bool sgx_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_enabled);
 
 
+bool sgx_lc_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_lc_enabled);
+
+static __init bool sgx_is_enabled(bool *lc_enabled)
+{
+	unsigned long fc;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX))
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX1))
+		return false;
+
+	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
+	if (!(fc & FEATURE_CONTROL_LOCKED)) {
+		pr_info("IA32_FEATURE_CONTROL MSR is not locked\n");
+		return false;
+	}
+
+	if (!(fc & FEATURE_CONTROL_SGX_ENABLE)) {
+		pr_info("disabled by the firmware\n");
+		return false;
+	}
+
+	if (!(fc & FEATURE_CONTROL_SGX_LE_WR)) {
+		pr_info("IA32_SGXLEPUBKEYHASHn MSRs are not writable\n");

arch/x86/kernel/cpu/intel_sgx.c line 50

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c new file +10 -0
+		return false;
+	}
+
+	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
+	return true;
+}
+
+static __init int sgx_init(void)
+{
+	sgx_enabled = sgx_is_enabled(&sgx_lc_enabled);

This is horrible, really. Both variables are global. So what the heck is wrong with assigning them directly in the function?

Thanks,

tglx

#32 lore raw

Re: [PATCH v12 06/13] x86/sgx: detect Intel SGX

Quoted 56 lines · Thomas Gleixner

arch/x86/kernel/cpu/intel_sgx.c line 11

Quoted 12 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/intel_sgx.c new file +11 -0
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2016-17 Intel Corporation.
+
+#include <asm/sgx.h>
+#include <asm/sgx_pr.h>
+#include <linux/freezer.h>
+#include <linux/highmem.h>
+#include <linux/kthread.h>
+#include <linux/ratelimit.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>

The common include ordering is

include linux/....

include asm/...

Quoted 36 lines
arch/x86/kernel/cpu/intel_sgx.c new file +34 -0
+
+bool sgx_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_enabled);
 
 
+bool sgx_lc_enabled __ro_after_init;
+EXPORT_SYMBOL(sgx_lc_enabled);
+
+static __init bool sgx_is_enabled(bool *lc_enabled)
+{
+	unsigned long fc;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX))
+		return false;
+
+	if (!boot_cpu_has(X86_FEATURE_SGX1))
+		return false;
+
+	rdmsrl(MSR_IA32_FEATURE_CONTROL, fc);
+	if (!(fc & FEATURE_CONTROL_LOCKED)) {
+		pr_info("IA32_FEATURE_CONTROL MSR is not locked\n");
+		return false;
+	}
+
+	if (!(fc & FEATURE_CONTROL_SGX_ENABLE)) {
+		pr_info("disabled by the firmware\n");
+		return false;
+	}
+
+	if (!(fc & FEATURE_CONTROL_SGX_LE_WR)) {
+		pr_info("IA32_SGXLEPUBKEYHASHn MSRs are not writable\n");
+		return false;
+	}
+
+	*lc_enabled = !!(fc & FEATURE_CONTROL_SGX_LE_WR);
+	return true;
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 50

Quoted 5 lines
arch/x86/kernel/cpu/intel_sgx.c new file +5 -0
+}
+
+static __init int sgx_init(void)
+{
+	sgx_enabled = sgx_is_enabled(&sgx_lc_enabled);

This is horrible, really. Both variables are global. So what the heck is wrong with assigning them directly in the function?

Fully agreed, does not make any sense.

Quoted 3 lines

Thanks,

tglx

/Jarkko

#33 lore raw

PATCH v12 7/13 x86/sgx: data structures for tracking available EPC pages

In reply to Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

This commit adds a database of EPC banks for kernel to easily access the available EPC pages. On UMA architectures there is a singe bank of EPC pages. On NUMA architectures there is an EPC bank for each node.

---
 arch/x86/include/asm/sgx.h      |  30 +++++++
 arch/x86/kernel/cpu/intel_sgx.c | 146 +++++++++++++++++++++++++++++++-
 2 files changed, 175 insertions(+), 1 deletion(-)
arch/x86/include/asm/sgx.h +30 -0
index 2130e639ab49..77b2294fcfb0 100644
@@ -4,9 +4,39 @@
 #ifndef _ASM_X86_SGX_H
 #define _ASM_X86_SGX_H
 
+#include <asm/sgx_arch.h>
+#include <asm/asm.h>
+#include <linux/bitops.h>
+#include <linux/err.h>
+#include <linux/rwsem.h>
 #include <linux/types.h>
 
+#define SGX_MAX_EPC_BANKS 8
+
+#define SGX_EPC_BANK(epc_page) \
+	(&sgx_epc_banks[(unsigned long)(epc_page->desc) & ~PAGE_MASK])
+#define SGX_EPC_PFN(epc_page) PFN_DOWN((unsigned long)(epc_page->desc))
+#define SGX_EPC_ADDR(epc_page) ((unsigned long)(epc_page->desc) & PAGE_MASK)
+
+struct sgx_epc_page {
+	unsigned long desc;
+	struct list_head list;
+};
+
+struct sgx_epc_bank {
+	unsigned long pa;
+	unsigned long va;
+	unsigned long size;
+	struct sgx_epc_page *pages_data;
+	struct sgx_epc_page **pages;
+	atomic_t free_cnt;
+	struct rw_semaphore lock;
+};
+
 extern bool sgx_enabled;
 extern bool sgx_lc_enabled;
 
+void *sgx_get_page(struct sgx_epc_page *ptr);
+void sgx_put_page(void *epc_page_ptr);
+
 #endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/intel_sgx.c +145 -1

166 lines not on this page · raw message

Signature · 1 line
2.17.1
#34 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

Quoted 3 lines · Jarkko Sakkinen

This commit adds a database of EPC banks for kernel to easily access the available EPC pages. On UMA architectures there is a singe bank of EPC pages. On NUMA architectures there is an EPC bank for each node.

Is this universally true? What about Sub-NUMA-Clustering/Cluster-on-Die systems?

#35 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

Quoted 7 lines · Dave Hansen
Quoted 3 lines · Jarkko Sakkinen

This commit adds a database of EPC banks for kernel to easily access the available EPC pages. On UMA architectures there is a singe bank of EPC pages. On NUMA architectures there is an EPC bank for each node.

Is this universally true? What about Sub-NUMA-Clustering/Cluster-on-Die systems?

I have to say that I do not have a good answer right now but I will study this area for the next version.

/Jarkko

#36 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

In reply to Jarkko Sakkinen
Quoted 7 lines · Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

This commit adds a database of EPC banks for kernel to easily access the

Add a ....

Quoted 1 line

available EPC pages. On UMA architectures there is a singe bank of EPC

singe? Spell checkers exist for a reason.

arch/x86/include/asm/sgx.h line 8

Quoted 12 lines

pages. On NUMA architectures there is an EPC bank for each node.

arch/x86/include/asm/sgx.h +2 -0
index 2130e639ab49..77b2294fcfb0 100644
@@ -4,9 +4,39 @@
 #ifndef _ASM_X86_SGX_H
 #define _ASM_X86_SGX_H
 
+#include <asm/sgx_arch.h>
+#include <asm/asm.h>

asm includes go below.

arch/x86/include/asm/sgx.h line 19

Quoted 11 lines
arch/x86/include/asm/sgx.h +9 -0
+#include <linux/bitops.h>
+#include <linux/err.h>
+#include <linux/rwsem.h>
 #include <linux/types.h>
 
+#define SGX_MAX_EPC_BANKS 8
+
+#define SGX_EPC_BANK(epc_page) \
+	(&sgx_epc_banks[(unsigned long)(epc_page->desc) & ~PAGE_MASK])
+#define SGX_EPC_PFN(epc_page) PFN_DOWN((unsigned long)(epc_page->desc))
+#define SGX_EPC_ADDR(epc_page) ((unsigned long)(epc_page->desc) & PAGE_MASK)

And these need to be macros because MACROS_ARE_SO_PROMINENT, while inline functions are so boring and type safe.

arch/x86/include/asm/sgx.h line 25

Quoted 6 lines
arch/x86/include/asm/sgx.h +6 -0
+static atomic_t sgx_nr_free_pages = ATOMIC_INIT(0);
+static struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS];
+static int sgx_nr_epc_banks;
+
+/**
+ * sgx_get_page - pin an EPC page

Description starts with an uppercase letter.

arch/x86/include/asm/sgx.h line 26

Quoted 1 line
arch/x86/include/asm/sgx.h +1 -0
+ * @page:	an EPC page

That's not an EPC page. It's a pointer to an EPC page. Please make these comments useful and do not add them just that they exist.

arch/x86/include/asm/sgx.h line 34

Quoted 8 lines
arch/x86/include/asm/sgx.h +8 -0
+ *
+ * Return: a pointer to the pinned EPC page
+ */
+void *sgx_get_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+
+	if (IS_ENABLED(CONFIG_X86_64))

And this is needed because the SGX config switch has:

       depends on X86_64 && CPU_SUP_INTEL

So what is this for and why do you need the 32bit implementation at all?

Quoted 4 lines
arch/x86/include/asm/sgx.h +4 -0
+		return (void *)(bank->va + SGX_EPC_ADDR(page) - bank->pa);
+
+	return kmap_atomic_pfn(SGX_EPC_PFN(page));
+}

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+EXPORT_SYMBOL(sgx_get_page);
+
+/**
+ * sgx_put_page - unpin an EPC page
+ * @ptr:	a pointer to the pinned EPC page
+ */
+void sgx_put_page(void *ptr)
+{
+	if (IS_ENABLED(CONFIG_X86_64))
+		return;

Ditto

Quoted 4 lines
arch/x86/include/asm/sgx.h +3 -0
+	kunmap_atomic(ptr);
+}
+EXPORT_SYMBOL(sgx_put_page);

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+
+static __init int sgx_init_epc_bank(unsigned long addr, unsigned long size,
+				    unsigned long index,
+				    struct sgx_epc_bank *bank)
+{
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	unsigned long i;
+	void *va;
+
+	if (IS_ENABLED(CONFIG_X86_64)) {

And more.

arch/x86/include/asm/sgx.h

Quoted 13 lines
arch/x86/include/asm/sgx.h +13 -0
+		va = ioremap_cache(addr, size);
+		if (!va)
+			return -ENOMEM;
+	}
+
+	bank->pages_data = kzalloc(nr_pages * sizeof(struct sgx_epc_page),
+				   GFP_KERNEL);
+	if (!bank->pages_data) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+
+		return -ENOMEM;
+	}

if (!bank->pages_data)

		goto out_iomap;

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+
+	bank->pages = kzalloc(nr_pages * sizeof(struct sgx_epc_page *),
+			      GFP_KERNEL);
+	if (!bank->pages) {
+		if (IS_ENABLED(CONFIG_X86_64))
+			iounmap(va);
+		kfree(bank->pages_data);
+		bank->pages_data = NULL;
+		return -ENOMEM;
+	}

if (!bank->pages)

		goto out_pdata;

arch/x86/include/asm/sgx.h

Quoted 9 lines
arch/x86/include/asm/sgx.h +9 -0
+	for (i = 0; i < nr_pages; i++) {
+		bank->pages[i] = &bank->pages_data[i];
+		bank->pages[i]->desc = (addr + (i << PAGE_SHIFT)) | index;
+	}
+
+	bank->pa = addr;
+	bank->size = size;
+	if (IS_ENABLED(CONFIG_X86_64))
+		bank->va = (unsigned long)va;

Why is bank->va unsigned long and not a void *? That's spare the type casts.

arch/x86/include/asm/sgx.h

Quoted 5 lines
arch/x86/include/asm/sgx.h +5 -0
+
+	atomic_set(&bank->free_cnt, nr_pages);
+	init_rwsem(&bank->lock);
+	atomic_add(nr_pages, &sgx_nr_free_pages);
+	return 0;

out_iomap:

	iounmap(va);
out_pdata:
	kfree(bank->pages_data);
	bank->pages_data = NULL;
	return -ENOMEM;

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+static __init int sgx_page_cache_init(void)
+{
+	unsigned long size;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	unsigned long pa;
+	int i;
+	int ret;

Please aggregate the declarations of the same type. No point in wasting all the screen space. And please use 'u32' for ea-dx

arch/x86/include/asm/sgx.h

Quoted 9 lines
arch/x86/include/asm/sgx.h +9 -0
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);

These magic constants should be 'U', using uppercase 'F' and defines. Plus this wants a comment how pa and size are cobbled together from the cpuid regs.

arch/x86/include/asm/sgx.h

Quoted 2 lines
arch/x86/include/asm/sgx.h +2 -0
+
+		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);

size - 1 because the bank does not reach into the next page.

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
+		if (ret) {
+			sgx_page_cache_teardown();
+			return ret;
+		}
+
+		sgx_nr_epc_banks++;
+	}
+
+	return 0;

This returns success even when the number of detected banks is zero.

Quoted 1 line

+ }

This whole thing can be written readable.

{
	int i, cnt = SGX_CPUID_EPC_BANKS;

	for (i = 0; i < SGX_MAX_EPC_BANKS; i++, cnt++) {
	     	u32 eax, ebx, ecx, edx;
		unsigned long pa, size;
		int ret;

		cpuid_count(SGX_CPUID, cnt, &eax, &ebx, &ecx, &edx);
		if (!(eax & 0xF))
			break;

		pa = combine_regs(ebx, eax);
		size = combine_regs(edx, ecx);

		pr_info("EPC bank 0x%lx-0x%lx\n", pa, pa + size);

		ret = sgx_init_epc_bank(pa, size, i, &sgx_epc_banks[i]);
		if (ret) {
			sgx_page_cache_teardown();
			return ret;
		}
		sgx_nr_epc_banks++;
	}

	return sgx_nr_epc_banks ? : -ENODEV;
}

You get the idea...

Thanks,

tglx

reviewer
#37 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

Quoted 9 lines · Thomas Gleixner
Quoted 7 lines · Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

This commit adds a database of EPC banks for kernel to easily access the

what kind of database? How does one query it?

Quoted 10 lines

arch/x86/include/asm/sgx.h

Quoted 8 lines
arch/x86/include/asm/sgx.h +2 -0
index 2130e639ab49..77b2294fcfb0 100644
 
 
+/**
+ * sgx_get_page - pin an EPC page

Description starts with an uppercase letter.

How would someone know that? It's not documented anywhere. I happen to disagree with it.

Quoted 14 lines

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+static __init int sgx_page_cache_init(void)
+{
+	unsigned long size;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	unsigned long pa;
+	int i;
+	int ret;

Please aggregate the declarations of the same type. No point in wasting all the screen space. And please use 'u32' for ea-dx

Quoted 13 lines

arch/x86/include/asm/sgx.h

Quoted 9 lines
arch/x86/include/asm/sgx.h +9 -0
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);

These magic constants should be 'U', using uppercase 'F' and defines. Plus this wants a comment how pa and size are cobbled together from the cpuid regs.

Why the uppercase 'F'?

thnx,

Signature · 1 line
~Randy
#38 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

Quoted 19 lines · Randy Dunlap
Quoted 9 lines · Thomas Gleixner
Quoted 7 lines · Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

This commit adds a database of EPC banks for kernel to easily access the

what kind of database? How does one query it?

Quoted 4 lines
Quoted 4 lines
arch/x86/include/asm/sgx.h +0 -0
index 2130e639ab49..77b2294fcfb0 100644
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

+/**
+ * sgx_get_page - pin an EPC page

Description starts with an uppercase letter.

How would someone know that? It's not documented anywhere.

It's a short sentence decribing the function and sentences start usually with uppercase letters, right?

Quoted 1 line

I happen to disagree with it.

Then we have to agree that we disagree.

Quoted 28 lines
Quoted 14 lines

arch/x86/include/asm/sgx.h

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+static __init int sgx_page_cache_init(void)
+{
+	unsigned long size;
+	unsigned int eax;
+	unsigned int ebx;
+	unsigned int ecx;
+	unsigned int edx;
+	unsigned long pa;
+	int i;
+	int ret;

Please aggregate the declarations of the same type. No point in wasting all the screen space. And please use 'u32' for ea-dx

Quoted 13 lines

arch/x86/include/asm/sgx.h

Quoted 9 lines
arch/x86/include/asm/sgx.h +9 -0
+
+	for (i = 0; i < SGX_MAX_EPC_BANKS; i++) {
+		cpuid_count(SGX_CPUID, i + SGX_CPUID_EPC_BANKS, &eax, &ebx,
+			    &ecx, &edx);
+		if (!(eax & 0xf))
+			break;
+
+		pa = ((u64)(ebx & 0xfffff) << 32) + (u64)(eax & 0xfffff000);
+		size = ((u64)(edx & 0xfffff) << 32) + (u64)(ecx & 0xfffff000);

These magic constants should be 'U', using uppercase 'F' and defines. Plus this wants a comment how pa and size are cobbled together from the cpuid regs.

Quoted 1 line

Why the uppercase 'F'?

Makes is more readable IMO, but I'm not religious about that.

Thanks,

tglx

#39 lore raw

Re: [PATCH v12 07/13] x86/sgx: data structures for tracking available EPC pages

In reply to Randy Dunlap
Quoted 9 lines · Randy Dunlap
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

Quoted 3 lines
Quoted 1 line
Quoted 1 line

This commit adds a database of EPC banks for kernel to easily access the

what kind of database? How does one query it?

Have to think of a better word. Thanks!

/Jarkko

#40 lore raw

PATCH v12 8/13 x86/sgx: wrappers for ENCLS opcode leaf functions

In reply to Jarkko Sakkinen

This commit adds wrappers for Intel(R) SGX ENCLS opcode leaf functions except for ENCLS(EINIT). The ENCLS instruction invokes the privileged functions for managing (creation, initialization and swapping) and debugging enclaves.

---
 arch/x86/include/asm/sgx.h      | 176 ++++++++++++++++++++++++++++++++
 arch/x86/include/asm/sgx_arch.h |  19 ++++
 2 files changed, 195 insertions(+)
arch/x86/include/asm/sgx.h +176 -0
index 77b2294fcfb0..a42c8ed10f7d 100644
@@ -11,6 +11,157 @@
 #include <linux/rwsem.h>
 #include <linux/types.h>
 
+#define IS_ENCLS_FAULT(r) ((r) & 0xffff0000)
+#define ENCLS_FAULT_VECTOR(r) ((r) >> 16)
+
+#define ENCLS_TO_ERR(r) (IS_ENCLS_FAULT(r) ? -EFAULT :		\
+			(r) == SGX_UNMASKED_EVENT ? -EINTR :	\
+			(r) == SGX_MAC_COMPARE_FAIL ? -EIO :	\
+			(r) == SGX_ENTRYEPOCH_LOCKED ? -EBUSY : -EPERM)
+
+#define __encls_ret_N(rax, inputs...)			\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)			\
+	: "=a"(ret)					\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_ret_1(rax, rcx)				\
+	({						\
+	__encls_ret_N(rax, "c"(rcx));			\
+	})
+
+#define __encls_ret_2(rax, rbx, rcx)			\
+	({						\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx));		\
+	})
+
+#define __encls_ret_3(rax, rbx, rcx, rdx)			\
+	({							\
+	__encls_ret_N(rax, "b"(rbx), "c"(rcx), "d"(rdx));	\
+	})
+
+#define __encls_N(rax, rbx_out, inputs...)		\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"   xor %%eax,%%eax;\n"				\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)				\
+	: "=a"(ret), "=b"(rbx_out)			\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})
+
+#define __encls_2(rax, rbx, rcx)				\
+	({							\
+	unsigned long ign_rbx_out;				\
+	__encls_N(rax, ign_rbx_out, "b"(rbx), "c"(rcx));	\
+	})
+
+#define __encls_1_1(rax, data, rcx)			\
+	({						\
+	unsigned long rbx_out;				\
+	int ret = __encls_N(rax, rbx_out, "c"(rcx));	\
+	if (!ret)					\
+		data = rbx_out;				\
+	ret;						\
+	})
+
+static inline int __ecreate(struct sgx_pageinfo *pginfo, void *secs)
+{
+	return __encls_2(ECREATE, pginfo, secs);
+}
+
+static inline int __eextend(void *secs, void *epc)
+{
+	return __encls_2(EEXTEND, secs, epc);
+}
+
+static inline int __eadd(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EADD, pginfo, epc);
+}
+
+static inline int __einit(void *sigstruct, struct sgx_einittoken *einittoken,
+			  void *secs)
+{
+	return __encls_ret_3(EINIT, sigstruct, secs, einittoken);
+}
+
+static inline int __eremove(void *epc)
+{
+	return __encls_ret_1(EREMOVE, epc);
+}
+
+static inline int __edbgwr(unsigned long addr, unsigned long *data)
+{
+	return __encls_2(EDGBWR, *data, addr);
+}
+
+static inline int __edbgrd(unsigned long addr, unsigned long *data)
+{
+	return __encls_1_1(EDGBRD, *data, addr);
+}
+
+static inline int __etrack(void *epc)
+{
+	return __encls_ret_1(ETRACK, epc);
+}
+
+static inline int __eldu(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(ELDU, pginfo, epc, va);
+}
+
+static inline int __eblock(void *epc)
+{
+	return __encls_ret_1(EBLOCK, epc);
+}
+
+static inline int __epa(void *epc)
+{
+	unsigned long rbx = SGX_PAGE_TYPE_VA;
+
+	return __encls_2(EPA, rbx, epc);
+}
+
+static inline int __ewb(struct sgx_pageinfo *pginfo, void *epc, void *va)
+{
+	return __encls_ret_3(EWB, pginfo, epc, va);
+}
+
+static inline int __eaug(struct sgx_pageinfo *pginfo, void *epc)
+{
+	return __encls_2(EAUG, pginfo, epc);
+}
+
+static inline int __emodpr(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODPR, secinfo, epc);
+}
+
+static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODT, secinfo, epc);
+}
+
 #define SGX_MAX_EPC_BANKS 8
 
 #define SGX_EPC_BANK(epc_page) \
@@ -39,4 +190,29 @@ extern bool sgx_lc_enabled;
 void *sgx_get_page(struct sgx_epc_page *ptr);
 void sgx_put_page(void *epc_page_ptr);
 
+#define SGX_FN(name, params...)		\
+{					\
+	void *epc;			\
+	int ret;			\
+	epc = sgx_get_page(epc_page);	\
+	ret = __##name(params);		\
+	sgx_put_page(epc);		\
+	return ret;			\
+}
+
+#define BUILD_SGX_FN(fn, name)				\
+static inline int fn(struct sgx_epc_page *epc_page)	\
+	SGX_FN(name, epc)
+BUILD_SGX_FN(sgx_eremove, eremove)
+BUILD_SGX_FN(sgx_eblock, eblock)
+BUILD_SGX_FN(sgx_etrack, etrack)
+BUILD_SGX_FN(sgx_epa, epa)
+
+static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
+			     struct sgx_epc_page *epc_page)
+	SGX_FN(emodpr, secinfo, epc)
+static inline int sgx_emodt(struct sgx_secinfo *secinfo,
+			    struct sgx_epc_page *epc_page)
+	SGX_FN(emodt, secinfo, epc)
+
 #endif /* _ASM_X86_SGX_H */
arch/x86/include/asm/sgx_arch.h +19 -0
index 41a37eaa3f51..85e6eb4c11a5 100644
@@ -14,6 +14,25 @@ enum sgx_cpuid {
 	SGX_CPUID_EPC_BANKS	= 2,
 };
 
+enum sgx_encls_leafs {
+	ECREATE	= 0x0,
+	EADD	= 0x1,
+	EINIT	= 0x2,
+	EREMOVE	= 0x3,
+	EDGBRD	= 0x4,
+	EDGBWR	= 0x5,
+	EEXTEND	= 0x6,
+	ELDB	= 0x7,
+	ELDU	= 0x8,
+	EBLOCK	= 0x9,
+	EPA	= 0xA,
+	EWB	= 0xB,
+	ETRACK	= 0xC,
+	EAUG	= 0xD,
+	EMODPR	= 0xE,
+	EMODT	= 0xF,
+};
+
 #define SGX_SSA_GPRS_SIZE		182
 #define SGX_SSA_MISC_EXINFO_SIZE	16
 
Signature · 1 line
2.17.1
#41 lore raw

Re: [PATCH v12 08/13] x86/sgx: wrappers for ENCLS opcode leaf functions

Quoted 1 line · Jarkko Sakkinen

This commit adds wrappers for Intel(R) SGX ENCLS opcode leaf functions

Add...

arch/x86/include/asm/sgx.h line 20

Quoted 11 lines

except for ENCLS(EINIT). The ENCLS instruction invokes the privileged functions for managing (creation, initialization and swapping) and debugging enclaves.

arch/x86/include/asm/sgx.h +7 -0
+#define IS_ENCLS_FAULT(r) ((r) & 0xffff0000)
+#define ENCLS_FAULT_VECTOR(r) ((r) >> 16)
+
+#define ENCLS_TO_ERR(r) (IS_ENCLS_FAULT(r) ? -EFAULT :		\
+			(r) == SGX_UNMASKED_EVENT ? -EINTR :	\
+			(r) == SGX_MAC_COMPARE_FAIL ? -EIO :	\
+			(r) == SGX_ENTRYEPOCH_LOCKED ? -EBUSY : -EPERM)

Inlines please along with proper comments.

arch/x86/include/asm/sgx.h line 29

Quoted 8 lines
arch/x86/include/asm/sgx.h +8 -0
+#define __encls_ret_N(rax, inputs...)			\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\

SHLL ??? _All_ the macro maze needs proper comments.

Quoted 8 lines
arch/x86/include/asm/sgx.h +8 -0
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)			\
+	: "=a"(ret)					\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})

....

Quoted 9 lines
arch/x86/include/asm/sgx.h +5 -0
+static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODT, secinfo, epc);
+}
+
 #define SGX_MAX_EPC_BANKS 8
 
 #define SGX_EPC_BANK(epc_page) \

arch/x86/include/asm/sgx.h extern bool sgx_lc_enabled; · line 199

Quoted 10 lines
arch/x86/include/asm/sgx.h +7 -0
@@ -39,4 +190,29 @@ extern bool sgx_lc_enabled;
 void *sgx_get_page(struct sgx_epc_page *ptr);
 void sgx_put_page(void *epc_page_ptr);
 
+#define SGX_FN(name, params...)		\
+{					\
+	void *epc;			\
+	int ret;			\
+	epc = sgx_get_page(epc_page);	\
+	ret = __##name(params);		\
+	sgx_put_page(epc);		\

This whole get/put magic is totally pointless. The stuff is 64bit only, so all it needs is the address, because 'put' is a noop on 64bit.

Quoted 7 lines
arch/x86/include/asm/sgx.h +7 -0
+	return ret;			\
+}
+
+#define BUILD_SGX_FN(fn, name)				\
+static inline int fn(struct sgx_epc_page *epc_page)	\
+	SGX_FN(name, epc)
+BUILD_SGX_FN(sgx_eremove, eremove)

arch/x86/include/asm/sgx.h extern bool sgx_lc_enabled; · line 216

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+BUILD_SGX_FN(sgx_eblock, eblock)
+BUILD_SGX_FN(sgx_etrack, etrack)
+BUILD_SGX_FN(sgx_epa, epa)
+
+static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
+			     struct sgx_epc_page *epc_page)
+	SGX_FN(emodpr, secinfo, epc)
+static inline int sgx_emodt(struct sgx_secinfo *secinfo,
+			    struct sgx_epc_page *epc_page)
+	SGX_FN(emodt, secinfo, epc)

Bah this is really unreadable crap. What's so horribly wrong with writing this simply as:

static inline int sgx_eremove(struct sgx_epc_page *epc_page)

{
	return __encls_ret_1(EREMOVE, epc_page_addr(epc_page));
}

static inline int sgx_emodt(struct sgx_secinfo *secinfo,
struct sgx_epc_page *epc_page)

{
	return __encls_ret_2(EREMOVE, secinfo, epc_page_addr(epc_page));
}

instead of all these completely pointless meta functions and build macro maze around it.

Why? Because then every function which is actually used in code has a proper prototype instead of nongrepable magic and a gazillion of wrappers.

Thanks,

tglx

#42 lore raw

Re: [PATCH v12 08/13] x86/sgx: wrappers for ENCLS opcode leaf functions

Quoted 20 lines · Thomas Gleixner
Quoted 1 line · Jarkko Sakkinen

This commit adds wrappers for Intel(R) SGX ENCLS opcode leaf functions

Add...

arch/x86/include/asm/sgx.h line 20

Quoted 11 lines

except for ENCLS(EINIT). The ENCLS instruction invokes the privileged functions for managing (creation, initialization and swapping) and debugging enclaves.

arch/x86/include/asm/sgx.h +7 -0
+#define IS_ENCLS_FAULT(r) ((r) & 0xffff0000)
+#define ENCLS_FAULT_VECTOR(r) ((r) >> 16)
+
+#define ENCLS_TO_ERR(r) (IS_ENCLS_FAULT(r) ? -EFAULT :		\
+			(r) == SGX_UNMASKED_EVENT ? -EINTR :	\
+			(r) == SGX_MAC_COMPARE_FAIL ? -EIO :	\
+			(r) == SGX_ENTRYEPOCH_LOCKED ? -EBUSY : -EPERM)

Inlines please along with proper comments.

Quoted 10 lines

arch/x86/include/asm/sgx.h line 29

Quoted 8 lines
arch/x86/include/asm/sgx.h +8 -0
+#define __encls_ret_N(rax, inputs...)			\
+	({						\
+	int ret;					\
+	asm volatile(					\
+	"1: .byte 0x0f, 0x01, 0xcf;\n\t"		\
+	"2:\n"						\
+	".section .fixup,\"ax\"\n"			\
+	"3: shll $16,%%eax\n"				\

SHLL ??? _All_ the macro maze needs proper comments.

Yeah, agreed.

Quoted 23 lines
Quoted 8 lines
arch/x86/include/asm/sgx.h +8 -0
+	"   jmp 2b\n"					\
+	".previous\n"					\
+	_ASM_EXTABLE_FAULT(1b, 3b)			\
+	: "=a"(ret)					\
+	: "a"(rax), inputs				\
+	: "memory");					\
+	ret;						\
+	})

....

Quoted 11 lines
arch/x86/include/asm/sgx.h +5 -0
+static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
+{
+	return __encls_ret_2(EMODT, secinfo, epc);
+}
+
 #define SGX_MAX_EPC_BANKS 8
 
 #define SGX_EPC_BANK(epc_page) \
@@ -39,4 +190,29 @@ extern bool sgx_lc_enabled;
 void *sgx_get_page(struct sgx_epc_page *ptr);
 void sgx_put_page(void *epc_page_ptr);
Quoted 10 lines

arch/x86/include/asm/sgx.h extern bool sgx_lc_enabled; · line 199

Quoted 7 lines
arch/x86/include/asm/sgx.h +7 -0
+#define SGX_FN(name, params...)		\
+{					\
+	void *epc;			\
+	int ret;			\
+	epc = sgx_get_page(epc_page);	\
+	ret = __##name(params);		\
+	sgx_put_page(epc);		\

This whole get/put magic is totally pointless. The stuff is 64bit only, so all it needs is the address, because 'put' is a noop on 64bit.

I did some early/proto versions of SGX code with 32-bit environment. I guess it is better to strip this stuff simply off.

Quoted 33 lines
Quoted 8 lines
arch/x86/include/asm/sgx.h +7 -0
+	return ret;			\
+}
+
+#define BUILD_SGX_FN(fn, name)				\
+static inline int fn(struct sgx_epc_page *epc_page)	\
+	SGX_FN(name, epc)
+BUILD_SGX_FN(sgx_eremove, eremove)

arch/x86/include/asm/sgx.h extern bool sgx_lc_enabled; · line 216

Quoted 10 lines
arch/x86/include/asm/sgx.h +10 -0
+BUILD_SGX_FN(sgx_eblock, eblock)
+BUILD_SGX_FN(sgx_etrack, etrack)
+BUILD_SGX_FN(sgx_epa, epa)
+
+static inline int sgx_emodpr(struct sgx_secinfo *secinfo,
+			     struct sgx_epc_page *epc_page)
+	SGX_FN(emodpr, secinfo, epc)
+static inline int sgx_emodt(struct sgx_secinfo *secinfo,
+			    struct sgx_epc_page *epc_page)
+	SGX_FN(emodt, secinfo, epc)

Bah this is really unreadable crap. What's so horribly wrong with writing this simply as:

static inline int sgx_eremove(struct sgx_epc_page *epc_page)

{
	return __encls_ret_1(EREMOVE, epc_page_addr(epc_page));
}

static inline int sgx_emodt(struct sgx_secinfo *secinfo,
struct sgx_epc_page *epc_page)

{
	return __encls_ret_2(EREMOVE, secinfo, epc_page_addr(epc_page));
}
Quoted 5 lines

instead of all these completely pointless meta functions and build macro maze around it.

Why? Because then every function which is actually used in code has a proper prototype instead of nongrepable magic and a gazillion of wrappers.

I do agree with you as I would NAK this kind of code from TPM because this is basically stuff that needs to be written only once (maybe some minor fixes later on but anyway) and after that the unrolled form is easier to maintain and debug.

I will do as you adviced.

Quoted 3 lines

Thanks,

tglx

Thank you!

/Jarkko

#43 lore raw

PATCH v12 9/13 x86/sgx: EPC page allocation routines

In reply to Jarkko Sakkinen

SGX has a set of data structures to maintain information about the enclaves and their security properties. BIOS reserves a fixed size region of physical memory for these structures by setting Processor Reserved Memory Range Registers (PRMRR). This memory area is called Enclave Page Cache (EPC).

This commit implements the routines to allocate and free pages from different EPC banks. There is also a swapper thread ksgxswapd for EPC pages that gets woken up by sgx_alloc_page() when we run below the low watermark. The swapper thread continues swapping pages up until it reaches the high watermark.

The SGX driver and in future KVM provide a set of callbacks that are used to reclaim, block and write an EPC pages. Kernel takes the responsibility of maintaining LRU cache for them.

---
 arch/x86/include/asm/sgx.h      |  26 ++++
 arch/x86/kernel/cpu/intel_sgx.c | 216 ++++++++++++++++++++++++++++++++
 2 files changed, 242 insertions(+)
arch/x86/include/asm/sgx.h +26 -0
index a42c8ed10f7d..4f5f32b37b5d 100644
@@ -169,8 +169,23 @@ static inline int __emodt(struct sgx_secinfo *secinfo, void *epc)
 #define SGX_EPC_PFN(epc_page) PFN_DOWN((unsigned long)(epc_page->desc))
 #define SGX_EPC_ADDR(epc_page) ((unsigned long)(epc_page->desc) & PAGE_MASK)
 
+struct sgx_epc_page;
+
+struct sgx_epc_page_ops {
+	bool (*get)(struct sgx_epc_page *epc_page);
+	void (*put)(struct sgx_epc_page *epc_page);
+	bool (*reclaim)(struct sgx_epc_page *epc_page);
+	void (*block)(struct sgx_epc_page *epc_page);
+	void (*write)(struct sgx_epc_page *epc_page);
+};
+
+struct sgx_epc_page_impl {
+	const struct sgx_epc_page_ops *ops;
+};
+
 struct sgx_epc_page {
 	unsigned long desc;
+	struct sgx_epc_page_impl *impl;
 	struct list_head list;
 };
 
@@ -186,9 +201,20 @@ struct sgx_epc_bank {
 
 extern bool sgx_enabled;
 extern bool sgx_lc_enabled;
+extern struct list_head sgx_active_page_list;
+extern struct spinlock sgx_active_page_list_lock;
+
+enum sgx_alloc_flags {
+	SGX_ALLOC_ATOMIC	= BIT(0),
+};
 
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags);
+int sgx_free_page(struct sgx_epc_page *page);
 void *sgx_get_page(struct sgx_epc_page *ptr);
 void sgx_put_page(void *epc_page_ptr);
+struct page *sgx_get_backing(struct file *file, pgoff_t index);
+void sgx_put_backing(struct page *backing_page, bool write);
 
 #define SGX_FN(name, params...)		\
 {					\
arch/x86/kernel/cpu/intel_sgx.c +216 -0

254 lines not on this page · raw message

Signature · 1 line
2.17.1
#44 lore raw

Re: [PATCH v12 09/13] x86/sgx: EPC page allocation routines

arch/x86/kernel/cpu/intel_sgx.c line 26

Quoted 13 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/intel_sgx.c +8 -0
+#define SGX_NR_TO_SCAN	16
+#define SGX_NR_LOW_PAGES 32
+#define SGX_NR_HIGH_PAGES 64
+
 bool sgx_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_enabled);
 bool sgx_lc_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_lc_enabled);
+LIST_HEAD(sgx_active_page_list);
+EXPORT_SYMBOL(sgx_active_page_list);
+DEFINE_SPINLOCK(sgx_active_page_list_lock);
+EXPORT_SYMBOL(sgx_active_page_list_lock);

Why is all of this exported. If done right then no call site has to fiddle with the list and the lock at all.

arch/x86/kernel/cpu/intel_sgx.c line 39

Quoted 12 lines

static atomic_t sgx_nr_free_pages = ATOMIC_INIT(0); static struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS]; static int sgx_nr_epc_banks;

arch/x86/kernel/cpu/intel_sgx.c +9 -0
+static struct task_struct *ksgxswapd_tsk;
+static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
+
+static void sgx_swap_cluster(void)
+{
+	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
+	struct sgx_epc_page *epc_page;
+	int i;
+	int j;

int i, j;

Quoted 29 lines
arch/x86/kernel/cpu/intel_sgx.c +29 -0
+	memset(cluster, 0, sizeof(cluster));
+
+	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
+		spin_lock(&sgx_active_page_list_lock);
+		if (list_empty(&sgx_active_page_list)) {
+			spin_unlock(&sgx_active_page_list_lock);
+			break;
+		}
+		epc_page = list_first_entry(&sgx_active_page_list,
+					    struct sgx_epc_page, list);
+		if (!epc_page->impl->ops->get(epc_page)) {
+			list_move_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			continue;
+		}
+		list_del(&epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
+
+		if (epc_page->impl->ops->reclaim(epc_page)) {
+			cluster[j++] = epc_page;
+		} else {
+			spin_lock(&sgx_active_page_list_lock);
+			list_add_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			epc_page->impl->ops->put(epc_page);
+		}
+	}
+
+	for (i = 0; cluster[i]; i++) {

arch/x86/kernel/cpu/intel_sgx.c line 79

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+		epc_page = cluster[i];
+		epc_page->impl->ops->block(epc_page);
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->write(epc_page);
+		epc_page->impl->ops->put(epc_page);
+		sgx_free_page(epc_page);
+	}

Thanks a lot for commenting this piece of art thoughtfully. It's entirely clear how all of this works now.

Quoted 9 lines
arch/x86/kernel/cpu/intel_sgx.c +9 -0
+}
+
+static int ksgxswapd(void *p)
+{
+	set_freezable();
+
+	while (!kthread_should_stop()) {
+		if (try_to_freeze())
+			continue;

arch/x86/kernel/cpu/intel_sgx.c line 98

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+
+		wait_event_freezable(ksgxswapd_waitq, kthread_should_stop() ||
+				     atomic_read(&sgx_nr_free_pages) <
+				     SGX_NR_HIGH_PAGES);
+
+		if (atomic_read(&sgx_nr_free_pages) < SGX_NR_HIGH_PAGES)
+			sgx_swap_cluster();
+	}
+
+	pr_info("%s: done\n", __func__);

Really useful.

Quoted 5 lines
arch/x86/kernel/cpu/intel_sgx.c +5 -0
+	return 0;
+}
+
+static struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
+{

arch/x86/kernel/cpu/intel_sgx.c line 113

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+	struct sgx_epc_bank *bank;
+	struct sgx_epc_page *page = NULL;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
+		bank = &sgx_epc_banks[i];
+
+		down_write(&bank->lock);
+
+		if (atomic_read(&bank->free_cnt))

And these atomics are required becasue bank->lock protection is not sufficient or what am I missing here?

Quoted 51 lines
arch/x86/kernel/cpu/intel_sgx.c +51 -0
+			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
+
+		up_write(&bank->lock);
+
+		if (page)
+			break;
+	}
+
+	if (page) {
+		atomic_dec(&sgx_nr_free_pages);
+		page->impl = impl;
+	}
+
+	return page;
+}
+
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Upon returning the
+ * low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   -ENOMEM if all pages are unreclaimable,
+ *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
+ */
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_try_alloc_page(impl);
+		if (entry)
+			break;
+
+		if (list_empty(&sgx_active_page_list))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {

arch/x86/kernel/cpu/intel_sgx.c line 174

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
+		sgx_swap_cluster();
+		schedule();
+	}
+
+	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
+		wake_up(&ksgxswapd_waitq);

What's the logic of SGX_NR_LOW_PAGES vs. SGX_NR_HIGH_PAGES?

Quoted 19 lines
arch/x86/kernel/cpu/intel_sgx.c +19 -0
+
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);
+
+/**
+ * sgx_free_page - free an EPC page
+ *
+ * @page:	any EPC page
+ *
+ * Remove an EPC page and insert it back to the list of free pages.
+ *
+ * Return: SGX error code
+ */
+int sgx_free_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+	int ret;
+

arch/x86/kernel/cpu/intel_sgx.c line 203

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+	ret = sgx_eremove(page);
+	if (ret) {
+		pr_debug("EREMOVE returned %d\n", ret);
+		return ret;
+	}
+
+	down_read(&bank->lock);
+	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
+	atomic_inc(&sgx_nr_free_pages);
+	up_read(&bank->lock);

I have hard time to see the benefit of this reader/writer semaphore here. Both sides which fiddle with the bank pages are doing a simple de/increment of free_cnt and a store resp. load. So what justifies the overhead of a rwsem?

arch/x86/kernel/cpu/intel_sgx.c static __init void sgx_page_cache_teardown(void) · line 324

Quoted 11 lines

static __init int sgx_init_epc_bank(unsigned long addr, unsigned long size,

arch/x86/kernel/cpu/intel_sgx.c +4 -0
 				    unsigned long index,
 				    struct sgx_epc_bank *bank)
@@ -114,6 +318,11 @@ static __init void sgx_page_cache_teardown(void)
 		kfree(bank->pages);
 		kfree(bank->pages_data);
 	}
+
+	if (ksgxswapd_tsk) {
+		kthread_stop(ksgxswapd_tsk);
+		ksgxswapd_tsk = NULL;

This stops the thread _AFTER_ freeing all the bank memory. Is that actually correct?

Thanks,

tglx

maintainer
#45 lore raw

Re: [PATCH v12 09/13] x86/sgx: EPC page allocation routines

Quoted 17 lines · Thomas Gleixner

arch/x86/kernel/cpu/intel_sgx.c line 26

Quoted 13 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/intel_sgx.c +8 -0
+#define SGX_NR_TO_SCAN	16
+#define SGX_NR_LOW_PAGES 32
+#define SGX_NR_HIGH_PAGES 64
+
 bool sgx_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_enabled);
 bool sgx_lc_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_lc_enabled);
+LIST_HEAD(sgx_active_page_list);
+EXPORT_SYMBOL(sgx_active_page_list);
+DEFINE_SPINLOCK(sgx_active_page_list_lock);
+EXPORT_SYMBOL(sgx_active_page_list_lock);

Why is all of this exported. If done right then no call site has to fiddle with the list and the lock at all.

... and also, why are the other exports not EXPORT_SYMBOL_GPL?

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#46 lore raw

Re: [PATCH v12 09/13] x86/sgx: EPC page allocation routines

Quoted 1 line · Borislav Petkov

... and also, why are the other exports not EXPORT_SYMBOL_GPL?

I think they could just as well be that. No reason to use EXPORT_SYMBOL() even if the source files are dual licensed.

/Jarkko

#47 lore raw

Re: [PATCH v12 09/13] x86/sgx: EPC page allocation routines

In reply to Thomas Gleixner
Quoted 17 lines · Thomas Gleixner

arch/x86/kernel/cpu/intel_sgx.c line 26

Quoted 13 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/intel_sgx.c +8 -0
+#define SGX_NR_TO_SCAN	16
+#define SGX_NR_LOW_PAGES 32
+#define SGX_NR_HIGH_PAGES 64
+
 bool sgx_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_enabled);
 bool sgx_lc_enabled __ro_after_init;
 EXPORT_SYMBOL(sgx_lc_enabled);
+LIST_HEAD(sgx_active_page_list);
+EXPORT_SYMBOL(sgx_active_page_list);
+DEFINE_SPINLOCK(sgx_active_page_list_lock);
+EXPORT_SYMBOL(sgx_active_page_list_lock);

Why is all of this exported. If done right then no call site has to fiddle with the list and the lock at all.

We can fix this in a way that these exports are not needed. Thanks for pointing this out.

Quoted 6 lines
Quoted 6 lines

static atomic_t sgx_nr_free_pages = ATOMIC_INIT(0); static struct sgx_epc_bank sgx_epc_banks[SGX_MAX_EPC_BANKS]; static int sgx_nr_epc_banks;

arch/x86/kernel/cpu/intel_sgx.c +3 -0
+static struct task_struct *ksgxswapd_tsk;
+static DECLARE_WAIT_QUEUE_HEAD(ksgxswapd_waitq);
+
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 39

Quoted 6 lines
arch/x86/kernel/cpu/intel_sgx.c +6 -0
+static void sgx_swap_cluster(void)
+{
+	struct sgx_epc_page *cluster[SGX_NR_TO_SCAN + 1];
+	struct sgx_epc_page *epc_page;
+	int i;
+	int j;

int i, j;

I've always preferred single declaration per line even for index variables but not something that I'm going to argue about too much.

Quoted 34 lines
Quoted 34 lines
arch/x86/kernel/cpu/intel_sgx.c +34 -0
+	memset(cluster, 0, sizeof(cluster));
+
+	for (i = 0, j = 0; i < SGX_NR_TO_SCAN; i++) {
+		spin_lock(&sgx_active_page_list_lock);
+		if (list_empty(&sgx_active_page_list)) {
+			spin_unlock(&sgx_active_page_list_lock);
+			break;
+		}
+		epc_page = list_first_entry(&sgx_active_page_list,
+					    struct sgx_epc_page, list);
+		if (!epc_page->impl->ops->get(epc_page)) {
+			list_move_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			continue;
+		}
+		list_del(&epc_page->list);
+		spin_unlock(&sgx_active_page_list_lock);
+
+		if (epc_page->impl->ops->reclaim(epc_page)) {
+			cluster[j++] = epc_page;
+		} else {
+			spin_lock(&sgx_active_page_list_lock);
+			list_add_tail(&epc_page->list, &sgx_active_page_list);
+			spin_unlock(&sgx_active_page_list_lock);
+			epc_page->impl->ops->put(epc_page);
+		}
+	}
+
+	for (i = 0; cluster[i]; i++) {
+		epc_page = cluster[i];
+		epc_page->impl->ops->block(epc_page);
+	}
+
+	for (i = 0; cluster[i]; i++) {
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 79

Quoted 5 lines
arch/x86/kernel/cpu/intel_sgx.c +5 -0
+		epc_page = cluster[i];
+		epc_page->impl->ops->write(epc_page);
+		epc_page->impl->ops->put(epc_page);
+		sgx_free_page(epc_page);
+	}

Thanks a lot for commenting this piece of art thoughtfully. It's entirely clear how all of this works now.

Got your point.

Quoted 13 lines
Quoted 13 lines
arch/x86/kernel/cpu/intel_sgx.c +13 -0
+}
+
+static int ksgxswapd(void *p)
+{
+	set_freezable();
+
+	while (!kthread_should_stop()) {
+		if (try_to_freeze())
+			continue;
+
+		wait_event_freezable(ksgxswapd_waitq, kthread_should_stop() ||
+				     atomic_read(&sgx_nr_free_pages) <
+				     SGX_NR_HIGH_PAGES);
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 98

Quoted 6 lines
arch/x86/kernel/cpu/intel_sgx.c +6 -0
+
+		if (atomic_read(&sgx_nr_free_pages) < SGX_NR_HIGH_PAGES)
+			sgx_swap_cluster();
+	}
+
+	pr_info("%s: done\n", __func__);

Really useful.

Forgotten cruft, will remove.

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+	return 0;
+}
+
+static struct sgx_epc_page *sgx_try_alloc_page(struct sgx_epc_page_impl *impl)
+{
+	struct sgx_epc_bank *bank;
+	struct sgx_epc_page *page = NULL;
+	int i;
+
+	for (i = 0; i < sgx_nr_epc_banks; i++) {
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 113

Quoted 5 lines
arch/x86/kernel/cpu/intel_sgx.c +5 -0
+		bank = &sgx_epc_banks[i];
+
+		down_write(&bank->lock);
+
+		if (atomic_read(&bank->free_cnt))

And these atomics are required becasue bank->lock protection is not sufficient or what am I missing here?

This is also response to your comment below. It would be better idea to just use a spinlock I guess. Seeing your and Daves point.

Quoted 55 lines
Quoted 55 lines
arch/x86/kernel/cpu/intel_sgx.c +55 -0
+			page = bank->pages[atomic_dec_return(&bank->free_cnt)];
+
+		up_write(&bank->lock);
+
+		if (page)
+			break;
+	}
+
+	if (page) {
+		atomic_dec(&sgx_nr_free_pages);
+		page->impl = impl;
+	}
+
+	return page;
+}
+
+/**
+ * sgx_alloc_page - allocate an EPC page
+ * @flags:	allocation flags
+ * @impl:	implementation for the struct sgx_epc_page
+ *
+ * Try to grab a page from the free EPC page list. If there is a free page
+ * available, it is returned to the caller. If called with SGX_ALLOC_ATOMIC,
+ * the function will return immediately if the list is empty. Otherwise, it
+ * will swap pages up until there is a free page available. Upon returning the
+ * low watermark is checked and ksgxswapd is waken up if we are below it.
+ *
+ * Return:
+ *   a &struct sgx_epc_page instace,
+ *   -ENOMEM if all pages are unreclaimable,
+ *   -EBUSY when called with SGX_ALLOC_ATOMIC and out of free pages
+ */
+struct sgx_epc_page *sgx_alloc_page(struct sgx_epc_page_impl *impl,
+				    unsigned int flags)
+{
+	struct sgx_epc_page *entry;
+
+	for ( ; ; ) {
+		entry = sgx_try_alloc_page(impl);
+		if (entry)
+			break;
+
+		if (list_empty(&sgx_active_page_list))
+			return ERR_PTR(-ENOMEM);
+
+		if (flags & SGX_ALLOC_ATOMIC) {
+			entry = ERR_PTR(-EBUSY);
+			break;
+		}
+
+		if (signal_pending(current)) {
+			entry = ERR_PTR(-ERESTARTSYS);
+			break;
+		}
+
Quoted 8 lines

arch/x86/kernel/cpu/intel_sgx.c line 174

Quoted 6 lines
arch/x86/kernel/cpu/intel_sgx.c +6 -0
+		sgx_swap_cluster();
+		schedule();
+	}
+
+	if (atomic_read(&sgx_nr_free_pages) < SGX_NR_LOW_PAGES)
+		wake_up(&ksgxswapd_waitq);

What's the logic of SGX_NR_LOW_PAGES vs. SGX_NR_HIGH_PAGES?

If the number of pages goes below SGX_NR_LOW_PAGES ksgxswapd swaps pages up until SGX_NR_HIGH_PAGES is reached.

Quoted 39 lines
Quoted 20 lines
arch/x86/kernel/cpu/intel_sgx.c +19 -0
+
+	return entry;
+}
+EXPORT_SYMBOL(sgx_alloc_page);
+
+/**
+ * sgx_free_page - free an EPC page
+ *
+ * @page:	any EPC page
+ *
+ * Remove an EPC page and insert it back to the list of free pages.
+ *
+ * Return: SGX error code
+ */
+int sgx_free_page(struct sgx_epc_page *page)
+{
+	struct sgx_epc_bank *bank = SGX_EPC_BANK(page);
+	int ret;
+

arch/x86/kernel/cpu/intel_sgx.c line 203

Quoted 10 lines
arch/x86/kernel/cpu/intel_sgx.c +10 -0
+	ret = sgx_eremove(page);
+	if (ret) {
+		pr_debug("EREMOVE returned %d\n", ret);
+		return ret;
+	}
+
+	down_read(&bank->lock);
+	bank->pages[atomic_inc_return(&bank->free_cnt) - 1] = page;
+	atomic_inc(&sgx_nr_free_pages);
+	up_read(&bank->lock);

I have hard time to see the benefit of this reader/writer semaphore here. Both sides which fiddle with the bank pages are doing a simple de/increment of free_cnt and a store resp. load. So what justifies the overhead of a rwsem?

Quoted 14 lines

arch/x86/kernel/cpu/intel_sgx.c static __init void sgx_page_cache_teardown(void) · line 324

Quoted 11 lines

static __init int sgx_init_epc_bank(unsigned long addr, unsigned long size,

arch/x86/kernel/cpu/intel_sgx.c +4 -0
 				    unsigned long index,
 				    struct sgx_epc_bank *bank)
@@ -114,6 +318,11 @@ static __init void sgx_page_cache_teardown(void)
 		kfree(bank->pages);
 		kfree(bank->pages_data);
 	}
+
+	if (ksgxswapd_tsk) {
+		kthread_stop(ksgxswapd_tsk);
+		ksgxswapd_tsk = NULL;

This stops the thread _AFTER_ freeing all the bank memory. Is that actually correct?

Should not cause any actual regressions but is a flakky order anyway so I will change it.

/Jarkko

#48 lore raw

PATCH v12 10/13 x86/sgx: sgx_einit()

In reply to Jarkko Sakkinen

From: Sean Christopherson <[email protected]>

Implements function to perform ENCLS(EINIT) leaf function that initializes an enclave, which can be used by a driver for running enclaves and VMMs.

Writing the LE hash MSRs is extraordinarily expensive, e.g. 3-4x slower than normal MSRs, so we use a per-cpu cache to track the last known value of the MSRs to avoid unnecessarily writing the MSRs with the current value.

---
 arch/x86/include/asm/sgx.h      |  2 ++
 arch/x86/kernel/cpu/intel_sgx.c | 46 +++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+)
arch/x86/include/asm/sgx.h +2 -0
index 4f5f32b37b5d..bc09b4576d68 100644
@@ -215,6 +215,8 @@ void *sgx_get_page(struct sgx_epc_page *ptr);
 void sgx_put_page(void *epc_page_ptr);
 struct page *sgx_get_backing(struct file *file, pgoff_t index);
 void sgx_put_backing(struct page *backing_page, bool write);
+int sgx_einit(struct sgx_sigstruct *sigstruct, struct sgx_einittoken *token,
+	      struct sgx_epc_page *secs_page, u64 le_pubkey_hash[4]);
 
 #define SGX_FN(name, params...)		\
 {					\
arch/x86/kernel/cpu/intel_sgx.c +46 -0

58 lines not on this page · raw message

Signature · 1 line
2.17.1
#49 lore raw

PATCH v12 11/13 platform/x86: Intel SGX driver

In reply to Jarkko Sakkinen

Intel Software Guard eXtensions (SGX) is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the enclave is disallowed to access the memory inside the enclave by the CPU access control.

SGX driver provides a ioctl API for loading and initializing enclaves. Address range for enclaves is reserved with mmap() and they are destroyed with munmap(). Enclave construction, measurement and initialization is done with the provided the ioctl API.

---
 arch/x86/include/asm/sgx_pr.h                 |   16 +
 arch/x86/include/uapi/asm/sgx.h               |  138 +++
 drivers/platform/x86/Kconfig                  |    2 +
 drivers/platform/x86/Makefile                 |    1 +
 drivers/platform/x86/intel_sgx/Kconfig        |   22 +
 drivers/platform/x86/intel_sgx/Makefile       |   13 +
 drivers/platform/x86/intel_sgx/sgx.h          |  157 +++
 drivers/platform/x86/intel_sgx/sgx_encl.c     | 1011 +++++++++++++++++
 .../platform/x86/intel_sgx/sgx_encl_page.c    |  286 +++++
 drivers/platform/x86/intel_sgx/sgx_fault.c    |  151 +++
 drivers/platform/x86/intel_sgx/sgx_ioctl.c    |  231 ++++
 drivers/platform/x86/intel_sgx/sgx_main.c     |  298 +++++
 drivers/platform/x86/intel_sgx/sgx_vma.c      |   58 +
 13 files changed, 2384 insertions(+)
 create mode 100644 arch/x86/include/uapi/asm/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/Kconfig
 create mode 100644 drivers/platform/x86/intel_sgx/Makefile
 create mode 100644 drivers/platform/x86/intel_sgx/sgx.h
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_encl_page.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_fault.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_ioctl.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_main.c
 create mode 100644 drivers/platform/x86/intel_sgx/sgx_vma.c
arch/x86/include/asm/sgx_pr.h +16 -0
index 42a4185d4bc9..b2baf12430e2 100644
@@ -10,4 +10,20 @@
 #undef pr_fmt
 #define pr_fmt(fmt) "intel_sgx: " fmt
 
+#define sgx_pr_ratelimited(level, encl, fmt, ...)			\
+	pr_ ## level ## _ratelimited("[%d:0x%p] " fmt,			\
+				     pid_nr((encl)->tgid),		\
+				     (void *)(encl)->base, ##__VA_ARGS__)
+
+#define sgx_dbg(encl, fmt, ...) \
+	sgx_pr_ratelimited(debug, encl, fmt, ##__VA_ARGS__)
+#define sgx_info(encl, fmt, ...) \
+	sgx_pr_ratelimited(info, encl, fmt, ##__VA_ARGS__)
+#define sgx_warn(encl, fmt, ...) \
+	sgx_pr_ratelimited(warn, encl, fmt, ##__VA_ARGS__)
+#define sgx_err(encl, fmt, ...) \
+	sgx_pr_ratelimited(err, encl, fmt, ##__VA_ARGS__)
+#define sgx_crit(encl, fmt, ...) \
+	sgx_pr_ratelimited(crit, encl, fmt, ##__VA_ARGS__)
+
 #endif /* _ASM_X86_SGX_PR_H */
arch/x86/include/uapi/asm/sgx.h new file +138 -0

138 lines not on this page · raw message

drivers/platform/x86/Kconfig +2 -0
index 566644bb496a..645888c2049d 100644
@@ -1218,6 +1218,8 @@ config INTEL_CHTDC_TI_PWRBTN
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_chtdc_ti_pwrbtn.
 
+source "drivers/platform/x86/intel_sgx/Kconfig"
+
 endif # X86_PLATFORM_DEVICES
 
 config PMC_ATOM
drivers/platform/x86/Makefile +1 -0
index 2ba6cb795338..729208a9c958 100644
@@ -91,3 +91,4 @@ obj-$(CONFIG_PMC_ATOM)		+= pmc_atom.o
 obj-$(CONFIG_MLX_PLATFORM)	+= mlx-platform.o
 obj-$(CONFIG_INTEL_TURBO_MAX_3) += intel_turbo_max_3.o
 obj-$(CONFIG_INTEL_CHTDC_TI_PWRBTN)	+= intel_chtdc_ti_pwrbtn.o
+obj-$(CONFIG_INTEL_SGX) += intel_sgx/
drivers/platform/x86/intel_sgx/Kconfig new file +22 -0
new file mode 100644
index 000000000000..8e6c57c28443
@@ -0,0 +1,22 @@
+#
+# Intel SGX
+#
+
+config INTEL_SGX
+	tristate "Intel(R) SGX Driver"
+	default n
+	depends on X86_64 && CPU_SUP_INTEL
+	select INTEL_SGX_CORE
+	select MMU_NOTIFIER
+	select CRYPTO
+	select CRYPTO_SHA256
+	help
+	Intel(R) SGX is a set of CPU instructions that can be used by
+	applications to set aside private regions of code and data.  The code
+	outside the enclave is disallowed to access the memory inside the
+	enclave by the CPU access control.
+
+	The firmware uses PRMRR registers to reserve an area of physical memory
+	called Enclave Page Cache (EPC). There is a hardware unit in the
+	processor called Memory Encryption Engine. The MEE encrypts and decrypts
+	the EPC pages as they enter and leave the processor package.
drivers/platform/x86/intel_sgx/Makefile new file +13 -0
new file mode 100644
index 000000000000..95f254e30a8b
@@ -0,0 +1,13 @@
+#
+# Intel SGX
+#
+
+obj-$(CONFIG_INTEL_SGX) += intel_sgx.o
+
+intel_sgx-$(CONFIG_INTEL_SGX) += \
+	sgx_ioctl.o \
+	sgx_encl.o \
+	sgx_encl_page.o \
+	sgx_main.o \
+	sgx_fault.o \
+	sgx_vma.o \
drivers/platform/x86/intel_sgx/sgx.h new file +157 -0

157 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_encl.c new file +1011 -0

1,011 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_encl_page.c new file +286 -0

286 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_fault.c new file +151 -0

151 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_ioctl.c new file +231 -0

231 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_main.c new file +298 -0

298 lines not on this page · raw message

drivers/platform/x86/intel_sgx/sgx_vma.c new file +58 -0

58 lines not on this page · raw message

Signature · 1 line
2.17.1
#50 lore raw

Re: [PATCH v12 11/13] platform/x86: Intel SGX driver

arch/x86/include/uapi/asm/sgx.h line 100

Quoted 3 lines · Jarkko Sakkinen
arch/x86/include/uapi/asm/sgx.h new file +3 -0
+/* IOCTL return values */
+#define SGX_POWER_LOST_ENCLAVE		0x40000000
+#define SGX_LE_ROLLBACK			0x40000001

I don't think SGX_LE_ROLLBACK is used anymore.

Jethro Beekman | Fortanix

#51 lore raw

Re: [PATCH v12 11/13] platform/x86: Intel SGX driver

Quoted 8 lines · Jethro Beekman

arch/x86/include/uapi/asm/sgx.h line 100

Quoted 3 lines · Jarkko Sakkinen
arch/x86/include/uapi/asm/sgx.h new file +3 -0
+/* IOCTL return values */
+#define SGX_POWER_LOST_ENCLAVE		0x40000000
+#define SGX_LE_ROLLBACK			0x40000001

I don't think SGX_LE_ROLLBACK is used anymore.

Jethro Beekman | Fortanix

Correct. Thaks for spotting that out.

I've on a leave for soon past three weeks and one week is remaining but have been slowly preparing v13 like one day a week so after next week it shouldn't take much time to get it out.

/Jarkko

#52 lore raw

PATCH v12 12/13 platform/x86: ptrace() support for the SGX driver

In reply to Jarkko Sakkinen

Implemented VMA callbacks in order to ptrace() debug enclaves. With debug enclaves data can be read and write the memory word at a time by using ENCLS(EDBGRD) and ENCLS(EDBGWR) leaf instructions.

---
 drivers/platform/x86/intel_sgx/sgx_encl.c |   2 +-
 drivers/platform/x86/intel_sgx/sgx_vma.c  | 116 ++++++++++++++++++++++
 2 files changed, 117 insertions(+), 1 deletion(-)
drivers/platform/x86/intel_sgx/sgx_encl.c +1 -1
index e3d7fa885489..20cd9ff7dd75 100644
@@ -954,7 +954,7 @@ int sgx_encl_load_page(struct sgx_encl_page *encl_page,
 	ret = __eldu(&pginfo, epc_ptr, va_ptr + va_offset);
 	if (ret) {
 		sgx_err(encl, "ELDU returned %d\n", ret);
-		ret = -EFAULT;
+		ret = ENCLS_TO_ERR(ret);
 	}
 
 	kunmap_atomic((void *)(unsigned long)(pginfo.pcmd - pcmd_offset));
drivers/platform/x86/intel_sgx/sgx_vma.c +116 -0

124 lines not on this page · raw message

Signature · 1 line
2.17.1
#53 lore raw

PATCH v12 13/13 x86/sgx: driver documentation

In reply to Jarkko Sakkinen

Documentation of the features of the Software Guard eXtensions used by the Linux kernel and basic design choices for the core and driver and functionality.

---
 Documentation/index.rst         |   1 +
 Documentation/x86/intel_sgx.rst | 185 ++++++++++++++++++++++++++++++++
 2 files changed, 186 insertions(+)
 create mode 100644 Documentation/x86/intel_sgx.rst
Documentation/index.rst +1 -0
index 3b99ab931d41..b9fb92928e8c 100644
@@ -100,6 +100,7 @@ implementation.
    :maxdepth: 2
 
    sh/index
+   x86/index
 
 Korean translations
 -------------------
Documentation/x86/intel_sgx.rst new file +185 -0

185 lines not on this page · raw message

Signature · 1 line
2.17.1