sgx.7: New page with overview of Software Guard eXtensions (SGX)

Signed-off-by Jarkko Sakkinen <[email protected]>

---
v4:
* Did a heavy edit trying to streamline the story a bit and focus on stuff important to the user (e.g. lighten up x86 details).
v3:
* Overhaul based on Michael's comments. Most likely needs to be refined in various places but this is at least a small step forward for sure.
v2:
* Fixed the semantic newlines convention and various style errors etc. that were reported by Alenjandro and Michael.
* SGX was merged to v5.

 man7/sgx.7 | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 196 insertions(+)
 create mode 100644 man7/sgx.7

man7/sgx.7 new file +196 -0

new file mode 100644
index 000000000..c0b67020e
@@ -0,0 +1,196 @@
+.\" Copyright (C) 2021 Intel Corporation
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" Permission is granted to make and distribute verbatim copies of this
+.\" manual provided the copyright notice and this permission notice are
+.\" preserved on all copies.
+.\"
+.\" Permission is granted to copy and distribute modified versions of this
+.\" manual under the conditions for verbatim copying, provided that the
+.\" entire resulting derived work is distributed under the terms of a
+.\" permission notice identical to this one.
+.\"
+.\" Since the Linux kernel and libraries are constantly changing, this
+.\" manual page may be incorrect or out-of-date.  The author(s) assume no
+.\" responsibility for errors or omissions, or for damages resulting from
+.\" the use of the information contained herein.  The author(s) may not
+.\" have taken the same level of care in the production of this manual,
+.\" which is licensed free of charge, as they might when working
+.\" professionally.
+.\"
+.\" Formatted or processed versions of this manual, if unaccompanied by
+.\" the source, must acknowledge the copyright and authors of this work.
+.\" %%%LICENSE_END
+.\"
+.TH SGX 7 2021\-02\-02 "Linux" "Linux Programmer's Manual"
+.PP
+sgx - overview of Software Guard eXtensions
+.SH DESCRIPTION
+.PP
+Intel Software Guard eXtensions (SGX) allow applications to set
+aside private memory regions of code and data.
+These memory regions are called enclaves.
+.PP
+SGX must be enabled in BIOS.
+If SGX appears to be unsupported on a system having hardware support,
+ensure that SGX is enabled in the BIOS.
+If a BIOS presents a choice between \(dqEnabled\(dq and \(dqSoftware
+Enabled\(dq modes for SGX,
+choose \(dqEnabled\(dq.
+.PP
+SGX is available only if the kernel was configured and built with the
+.B CONFIG_X86_SGX
+option.
+You can determine whether the hardware supports SGX by checking
+whether "sgx" appears in the
+.I flags
+field in
+.IR /proc/cpuinfo .
+.SS Overview
+.PP
+An enclave is a region of address space,
+mapping pages from Enclave Page Cache (EPC),
+which consists of sections of non-addressable system memory.
+They are constructed with sub-functions of the privileged (ring-0) ENCLS x86
+instruction,
+and interacted with sub-functions of the unprivileged (ring-3) ENCLU x86
+instruction.
+Any other type of memory access gets asserted by the CPU.
+Enclaves are represented to the user space as memory-mapped files,
+shareable by multiple processes.
+.PP
+An EPC page can be initialized with ENCLS sub-functions to any of the
+following types:
+.TP
+SECS
+SGX Enclave Control Structure (SECS) contains the enclave global properties
+such as the base address,
+size and SHA256 checksum of its contents.
+.TP
+REG
+Regular (REG) pages are code and data pages of the enclave.
+They are mapped to the enclave address space.
+.TP
+TCS
+Thread Control Structure (TCS) pages describe the entry points to an enclave with
+an offset from the base address.
+They are mapped to the enclave address space.
+ENCLU provides EENTER and ERESUME sub-functions,
+which take the address of a TCS page,
+and jump executing inside an enclave from the given offset.
+.PP
+An enclave can be entered only at a fixed set of entry points,
+each defined by a TCS,
+by invoking EENTER and ERESUME.
+Any other type of memory access is strictly prohibited by the CPU.
+.PP
+A thread inside an enclave can read and write memory inside and outside the
+enclave,
+but any action causing execution outside the enclave is asserted by the CPU
+with an exception,
+and ultimately exit from the enclave.
+A thread can cleanly exit from the enclave with EEXIT sub-function of the ENCLU,
+to a given address outside the enclave.
+.PP
+Although carved out of normal DRAM,
+enclave memory is marked in the system memory map as reserved and is not
+managed by the Linux memory manager.
+There may be several enclave regions spread across the system.
+Each contiguous region is called an Enclave Page Cache (EPC) section.
+The pages belonging to the EPC sections are encrypted when they leave the
+Last Level Cache (LLC).
+.SS Construction
+.PP
+An enclave's life-cycle starts by opening
+.I /dev/sgx_enclave,
+and ends when all the file descriptors have been closed.
+After opening the enclave,
+its contents must be populated with the
+.BR ioctl (2)
+interface provided by
+.IR <asm/sgx.h> .
+.PP
+The are rudimentarily the steps to construct an enclave:
+.IP 1.
+Invoke
+.B SGX_IOC_ENCLAVE_CREATE,
+which takes in data for the SECS,
+and initializes an EPC page for it.
+SECS is used by various ENCLS sub-functions to modify the enclave state.
+It is kept internally by the kernel,
+and is never made available to the user space.
+.IP 2.
+Populate regular and TCS pages to the enclave,
+by invoking
+.B SGX_IOC_ENCLAVE_ADD_PAGES.
+.IP 3.
+Invoke
+.B SGX_IOC_ENCLAVE_INIT,
+which makes the enclave executable.
+After this new pages can no longer be added.
+.SS Access rights
+The state of each EPC page is stored to a structure called the
+.I Enclave Page Cache Map (EPCM),
+which takes a portion of the EPC.
+The state consists of page type and access rights,
+among the other things.
+When a page is accessed by a CPU,
+the EPCM permissions are enforced,
+in addition to the
+.BR mmap(2)
+permissions.
+EPCM permissions are defined when invoking
+.B SGX_IOC_ENCLAVE_ADD_PAGES,
+by setting them to the
+.B flags
+field of
+.B struct enclave_add_pages.
+.PP
+A memory access to an unitialized EPC page causes EPCM fault,
+with a new SGX bit set in the error code.
+A new power cycle invalidates the whole EPCM,
+making all EPC pages unitialized.
+E.g. if the system goes to sleep,
+and then wakes up,
+all the data is gone.
+Thus, a user space run-time must be prepared to handle this exception,
+at any point of time.
+.PP
+When the pages are mapped to memory via
+.BR mmap (2)
+or
+.BR mprotect (2),
+the EPCM permissions are compared against the declared permissions.
+If the declared permissions have bits set that are not part of the EPCM
+permissions,
+the operation fails with the error
+.B EACCES.
+.SS Exception handling
+These are the exceptions triggered by an enclave:
+.IP
+Undefined instruction (#UD) exception is triggered when executing inside an enclave,
+for any instruction that may cause VMEXIT,
+I/O instruction, 
+or a change in CPU privilege levels.
+.IP
+Page fault (#PF) with a new SGX bit set,
+also known as EPCM fault,
+is triggered,
+when accessing unitialized EPC page,
+or when the memory access surpasses the EPCM permissions.
+.PP
+In order to assist the run-time,
+the kernel provides a vDSO entry point,
+.BR vsgx_enter_enclave,
+which wraps the code required the enter the enclave.
+When an exception occurs,
+the vDSO populates
+.B struct sgx_enclave_run 
+with the exception data,
+and returns to the caller.
+.SH VERSIONS
+The SGX feature was added in Linux 5.11.
+.SH SEE ALSO
+.BR ioctl (2),
+.BR mmap() (2),
+.BR mprotect (2)

Signature · 1 line

2.30.1

Signed-off-by Jarkko Sakkinen <[email protected]>

---
v4:
* Did a heavy edit trying to streamline the story a bit and focus on stuff important to the user (e.g. lighten up x86 details).
v3:
* Overhaul based on Michael's comments. Most likely needs to be refined in various places but this is at least a small step forward for sure.
v2:
* Fixed the semantic newlines convention and various style errors etc. that were reported by Alenjandro and Michael.
* SGX was merged to v5.

 man7/sgx.7 | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 196 insertions(+)
 create mode 100644 man7/sgx.7

man7/sgx.7 new file +42 -0

new file mode 100644
index 000000000..c0b67020e
@@ -0,0 +1,196 @@
+.\" Copyright (C) 2021 Intel Corporation
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" Permission is granted to make and distribute verbatim copies of this
+.\" manual provided the copyright notice and this permission notice are
+.\" preserved on all copies.
+.\"
+.\" Permission is granted to copy and distribute modified versions of this
+.\" manual under the conditions for verbatim copying, provided that the
+.\" entire resulting derived work is distributed under the terms of a
+.\" permission notice identical to this one.
+.\"
+.\" Since the Linux kernel and libraries are constantly changing, this
+.\" manual page may be incorrect or out-of-date.  The author(s) assume no
+.\" responsibility for errors or omissions, or for damages resulting from
+.\" the use of the information contained herein.  The author(s) may not
+.\" have taken the same level of care in the production of this manual,
+.\" which is licensed free of charge, as they might when working
+.\" professionally.
+.\"
+.\" Formatted or processed versions of this manual, if unaccompanied by
+.\" the source, must acknowledge the copyright and authors of this work.
+.\" %%%LICENSE_END
+.\"
+.TH SGX 7 2021\-02\-02 "Linux" "Linux Programmer's Manual"
+.PP
+sgx - overview of Software Guard eXtensions
+.SH DESCRIPTION
+.PP
+Intel Software Guard eXtensions (SGX) allow applications to set
+aside private memory regions of code and data.
+These memory regions are called enclaves.
+.PP
+SGX must be enabled in BIOS.
+If SGX appears to be unsupported on a system having hardware support,
+ensure that SGX is enabled in the BIOS.
+If a BIOS presents a choice between \(dqEnabled\(dq and \(dqSoftware
+Enabled\(dq modes for SGX,
+choose \(dqEnabled\(dq.
+.PP
+SGX is available only if the kernel was configured and built with the
+.B CONFIG_X86_SGX

man7/sgx.7 new file +4 -0

+such as the base address,
+size and SHA256 checksum of its contents.
+.TP
+REG

Hello Jarkko,

Sorry for the delayed response... Thank you for the revisions. More comments below :-).

Quoted 82 lines · Jarkko Sakkinen

Signed-off-by Jarkko Sakkinen <[email protected]>

---
v4:
* Did a heavy edit trying to streamline the story a bit and focus on stuff important to the user (e.g. lighten up x86 details). v3:
* Overhaul based on Michael's comments. Most likely needs to be refined
  in various places but this is at least a small step forward for sure.
v2:
* Fixed the semantic newlines convention and various style errors etc.
  that were reported by Alenjandro and Michael.
* SGX was merged to v5.
 man7/sgx.7 | 196
+++++++++++++++++++++++++++++++++++++++++++++++++++++

 1 file changed, 196 insertions(+)

 create mode 100644 man7/sgx.7

man7/sgx.7 new file +4 -0

new file mode 100644
index 000000000..c0b67020e
@@ -0,0 +1,196 @@
+.\" Copyright (C) 2021 Intel Corporation
+.\"
+.\" %%%LICENSE_START(VERBATIM)
+.\" Permission is granted to make and distribute verbatim copies of

this
+.\" manual provided the copyright notice and this permission notice are
+.\" preserved on all copies.
+.\"
+.\" Permission is granted to copy and distribute modified versions of this
+.\" manual under the conditions for verbatim copying, provided that the
+.\" entire resulting derived work is distributed under the terms of a
+.\" permission notice identical to this one.
+.\"
+.\" Since the Linux kernel and libraries are constantly changing, this
+.\" manual page may be incorrect or out-of-date.  The author(s) assume no
+.\" responsibility for errors or omissions, or for damages resulting from
+.\" the use of the information contained herein.  The author(s) may not
+.\" have taken the same level of care in the production of this manual,
+.\" which is licensed free of charge, as they might when working +.\" professionally.
+.\"
+.\" Formatted or processed versions of this manual, if unaccompanied by
+.\" the source, must acknowledge the copyright and authors of this work.
+.\" %%%LICENSE_END
+.\"
+.TH SGX 7 2021\-02\-02 "Linux" "Linux Programmer's Manual" +.PP

man7/sgx.7 new file +9 -0

+sgx - overview of Software Guard eXtensions
+.SH DESCRIPTION
+.PP
+Intel Software Guard eXtensions (SGX) allow applications to set
+aside private memory regions of code and data.
+These memory regions are called enclaves.
+.PP
+SGX must be enabled in BIOS.
+If SGX appears to be unsupported on a system having hardware

support,

man7/sgx.7 new file +6 -0

+ensure that SGX is enabled in the BIOS.
+If a BIOS presents a choice between \(dqEnabled\(dq and \(dqSoftware
+Enabled\(dq modes for SGX,
+choose \(dqEnabled\(dq.
+.PP
+SGX is available only if the kernel was configured and built with

the
+.B CONFIG_X86_SGX

man7/sgx.7 line 52

Quoted 10 lines

man7/sgx.7 new file +10 -0

+option.
+You can determine whether the hardware supports SGX by checking
+whether "sgx" appears in the
+.I flags
+field in
+.IR /proc/cpuinfo .
+.SS Overview
+.PP
+An enclave is a region of address space,
+mapping pages from Enclave Page Cache (EPC),

s/from/from the/

Quoted 3 lines

man7/sgx.7 new file +2 -0

+which consists of sections of non-addressable system memory.
+They are constructed with sub-functions of the privileged (ring-0)

ENCLS x86

The reference of "They" is unclear. Should it be

"The EPC is constructed using subfunctions..."?

Quoted 3 lines

man7/sgx.7 new file +2 -0

+instruction,
+and interacted with sub-functions of the unprivileged (ring-3) ENCLU

x86

Wording problem...
s/with/with using/?

man7/sgx.7 line 58

Quoted 2 lines

man7/sgx.7 new file +2 -0

+instruction.
+Any other type of memory access gets asserted by the CPU.

I think it would be good to add a few words to explain what is meant by "asserted by the CPU". (I realize that "asserted" is a technical term that may be quite normal for you, but not necessarily for all of the readers...)

But, there's also another problem: how does this sentence fit into the surrounding paragraph? It seems somehow out of place.

Quoted 1 line

+Enclaves are represented to the user space as memory-mapped files,

s/to the user space/to user-space/

Quoted 9 lines

man7/sgx.7 new file +3 -0

+shareable by multiple processes.
+.PP
+An EPC page can be initialized with ENCLS sub-functions to any of

the

man7/sgx.7 new file +4 -0

+following types:
+.TP
+SECS
+SGX Enclave Control Structure (SECS) contains the enclave global

properties

s/SGX/The SGX/

Quoted 10 lines

man7/sgx.7 new file +9 -0

+such as the base address,
+size and SHA256 checksum of its contents.
+.TP
+REG
+Regular (REG) pages are code and data pages of the enclave.
+They are mapped to the enclave address space.
+.TP
+TCS
+Thread Control Structure (TCS) pages describe the entry points to an

enclave with

man7/sgx.7 line 80

Quoted 5 lines

man7/sgx.7 new file +5 -0

+an offset from the base address.
+They are mapped to the enclave address space.
+ENCLU provides EENTER and ERESUME sub-functions,
+which take the address of a TCS page,
+and jump executing inside an enclave from the given offset.

The wording here seems a bit off. What do you mean by "jump executing"? Can you reword/explain in a bit more detail?

Quoted 2 lines

+.PP
+An enclave can be entered only at a fixed set of entry points,

You do not explain what you mean by "enter". I presume it means that you can only jump into code at specific addresses (defined by a TCS) in the enclave. If that's correct, I think it would be helpful to have add some words to make that point explicit.

man7/sgx.7 line 85

Quoted 3 lines

man7/sgx.7 new file +3 -0

+each defined by a TCS,
+by invoking EENTER and ERESUME.
+Any other type of memory access is strictly prohibited by the CPU.

Maybe it would be helpful to add (inside parentheses) one or two examples of types of memory access that are prohibited?

Quoted 3 lines

+.PP
+A thread inside an enclave can read and write memory inside and outside the

s/A thread inside an enclave can...
  /While executing inside an enclave, a thread can.../ ?

man7/sgx.7 line 91

Quoted 5 lines

man7/sgx.7 new file +2 -0

+enclave,
+but any action causing execution outside the enclave is asserted by

the CPU

man7/sgx.7 new file +2 -0

+with an exception,
+and ultimately exit from the enclave.

What does "ultimateley exit from the enclave" mean? It's not quite clear.

Quoted 2 lines

+A thread can cleanly exit from the enclave with EEXIT sub-function of the ENCLU,

s/EEXIT/the EEXIT/

Quoted 8 lines

man7/sgx.7 new file +4 -0

+to a given address outside the enclave.
+.PP
+Although carved out of normal DRAM,
+enclave memory is marked in the system memory map as reserved and is

not

man7/sgx.7 new file +3 -0

+managed by the Linux memory manager.
+There may be several enclave regions spread across the system.
+Each contiguous region is called an Enclave Page Cache (EPC)

Quoted 4 lines

section.
+The pages belonging to the EPC sections are encrypted when they leave the
+Last Level Cache (LLC).

Quoted 2 lines

+.SS Construction
+.PP

Remove the preceding ".PP" (it's not needed, and produces linter errors).

Quoted 2 lines

+An enclave's life-cycle starts by opening
+.I /dev/sgx_enclave,

.IR /dev/sgx_enclave ,

man7/sgx.7 line 113

Quoted 8 lines

man7/sgx.7 new file +8 -0

+and ends when all the file descriptors have been closed.
+After opening the enclave,
+its contents must be populated with the
+.BR ioctl (2)
+interface provided by
+.IR <asm/sgx.h> .
+.PP
+The are rudimentarily the steps to construct an enclave:

s/The/These/
s/rudimentarily the/key/

Quoted 3 lines

+.IP 1.
+Invoke
+.B SGX_IOC_ENCLAVE_CREATE,

.BR SGX_IOC_ENCLAVE_CREATE ,

man7/sgx.7 line 121

Quoted 6 lines

man7/sgx.7 new file +3 -0

+which takes in data for the SECS,
+and initializes an EPC page for it.
+SECS is used by various ENCLS sub-functions to modify the enclave

state.

man7/sgx.7 new file +2 -0

+It is kept internally by the kernel,
+and is never made available to the user space.

s/the user space/user space/

man7/sgx.7 line 125

Quoted 4 lines

+.IP 2.

man7/sgx.7 new file +3 -0

+Populate regular and TCS pages to the enclave,
+by invoking
+.B SGX_IOC_ENCLAVE_ADD_PAGES.

.BR SGX_IOC_ENCLAVE_ADD_PAGES .

Quoted 3 lines

+.IP 3.
+Invoke
+.B SGX_IOC_ENCLAVE_INIT,

.BR SGX_IOC_ENCLAVE_INIT ,

man7/sgx.7 line 130

Quoted 2 lines

man7/sgx.7 new file +2 -0

+which makes the enclave executable.
+After this new pages can no longer be added.

s/After this/After this step,/
s/added/added to the enclave/

Quoted 2 lines

+.SS Access rights
+The state of each EPC page is stored to a structure called the

s/to/in/

Quoted 2 lines

+.I Enclave Page Cache Map (EPCM),
+which takes a portion of the EPC.

Better: s/takes/occupies/ ?

Quoted 1 line

+The state consists of page type and access rights,

s/state/state information of an EPC page/
s/page type/the page type/

man7/sgx.7 line 140

Quoted 5 lines

man7/sgx.7 new file +5 -0

+among the other things.
+When a page is accessed by a CPU,
+the EPCM permissions are enforced,
+in addition to the
+.BR mmap(2)

.BR mmap (2)

man7/sgx.7 line 143

Quoted 3 lines

man7/sgx.7 new file +3 -0

+permissions.
+EPCM permissions are defined when invoking
+.B SGX_IOC_ENCLAVE_ADD_PAGES,

.BR SGX_IOC_ENCLAVE_ADD_PAGES ,

Quoted 2 lines

+by setting them to the
+.B flags

s/.B/.I/

Quoted 2 lines

+field of
+.B struct enclave_add_pages.

.IR struct enclave_add_pages .

Quoted 2 lines

+.PP
+A memory access to an unitialized EPC page causes EPCM fault,

s/unitialized/uninitialized/
s/EPCM/an EPCM/

man7/sgx.7 line 152

Quoted 3 lines

man7/sgx.7 new file +3 -0

+with a new SGX bit set in the error code.
+A new power cycle invalidates the whole EPCM,
+making all EPC pages unitialized.

s/unitialized/uninitialized/

Quoted 1 line

+E.g. if the system goes to sleep,

s/E.g./For example,/

Quoted 5 lines

man7/sgx.7 new file +3 -0

+and then wakes up,
+all the data is gone.
+Thus, a user space run-time must be prepared to handle this

exception,
+at any point of time.

s/of/in/

man7/sgx.7 line 162

Quoted 5 lines

+.PP

man7/sgx.7 new file +4 -0

+When the pages are mapped to memory via
+.BR mmap (2)
+or
+.BR mprotect (2),

This last does not seem right. mprotect(2) sets memory protections, whereas you start the sentence by talking about mapping pages. I'm not
sure what the fix should be. Maybe:

    When the pages are mapped by mmap(2) or page protections are changed vi mprotect(2)...

Quoted 2 lines

+the EPCM permissions are compared against the declared permissions.

By "declared permissions", do you mean the 'prot' argument of mmap()/mprotect? If so, I think it would be good to say this more explicitly.

man7/sgx.7 line 167

Quoted 5 lines

+If the declared permissions have bits set that are not part of the EPCM

man7/sgx.7 new file +3 -0

+permissions,
+the operation fails with the error
+.B EACCES.

.BR EACCES .

Quoted 5 lines

+.SS Exception handling

man7/sgx.7 new file +3 -0

+These are the exceptions triggered by an enclave:
+.IP
+Undefined instruction (#UD) exception is triggered when executing

inside an enclave,

s/Undefined/An undefined/

man7/sgx.7 line 174

Quoted 3 lines

man7/sgx.7 new file +3 -0

+for any instruction that may cause VMEXIT,
+I/O instruction, 
+or a change in CPU privilege levels.

s/CPU privilege levels/the CPU privilege level/ ?

Quoted 2 lines

+.IP
+Page fault (#PF) with a new SGX bit set,

s/Page/A page/

Quoted 1 line

+also known as EPCM fault,

s/EPCM/an EPCM/

man7/sgx.7 line 179

Quoted 2 lines

man7/sgx.7 new file +2 -0

+is triggered,
+when accessing unitialized EPC page,

s/unitialized/an uninitialized/

Quoted 1 line

+or when the memory access surpasses the EPCM permissions.

s/surpasses/violates/?

man7/sgx.7 line 184

Quoted 4 lines

+.PP

man7/sgx.7 new file +3 -0

+In order to assist the run-time,
+the kernel provides a vDSO entry point,
+.BR vsgx_enter_enclave,

.BR vsgx_enter_enclave ,

man7/sgx.7 line 187

Quoted 3 lines

man7/sgx.7 new file +3 -0

+which wraps the code required the enter the enclave.
+When an exception occurs,
+the vDSO populates

s/the vDSO/vsgx_enter_enclave/ ?

Quoted 1 line

+.B struct sgx_enclave_run

This is the first time you mention "struct sgx_enclave_run". What is it? Where is it? (Is it in the vDSO?) Some explanation is needed here, I thi9nk.

man7/sgx.7 line 196

Quoted 8 lines

man7/sgx.7 new file +8 -0

+with the exception data,
+and returns to the caller.
+.SH VERSIONS
+The SGX feature was added in Linux 5.11.
+.SH SEE ALSO
+.BR ioctl (2),
+.BR mmap() (2),
+.BR mprotect (2)

Thanks,

Michael