Skip to content

Add Cgroup support for SGX EPC memory

Kristen Carlson Accardi cgroups linux-doc linux-sgx lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest18 patches 46 repliesIn review
  2. v126 patches 16 repliesSuperseded

Changes in v2 from the cover letter

* rename struct sgx_epc_lru to sgx_epc_lru_lists to be more clear that this struct contains 2 lists.
* use inline functions rather than macros for sgx_epc_page_list* wrappers.
* Remove flags macros and open code all flags.
* Improve the commit message for RECLAIM_IN_PROGRESS patch to make it more clear what the patch does.
* remove notifier_block from misc cgroup changes and use a set of ops for callbacks instead.
* rename root_misc to misc_cg_root and parent_misc to misc_cg_parent
* consolidate misc cgroup changes to 2 patches and remove most of the previous helper functions.

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/18x86/sgx: Call cond_resched() at the end of sgx_reclaim_pages()arch/x86/kernel/cpu/sgx/main.c+11 -65 replies
2/18x86/sgx: Store struct sgx_encl when allocating new VA pagesarch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/encl.h arch/x86/kernel/cpu/sgx/ioctl.c +1 more+6 -46 replies
3/18x86/sgx: Add 'struct sgx_epc_lru_lists' to encapsulate lru list(s)arch/x86/kernel/cpu/sgx/sgx.h+65 -03 replies
4/18x86/sgx: Use sgx_epc_lru_lists for existing active page listarch/x86/kernel/cpu/sgx/main.c+19 -203 replies
5/18x86/sgx: Track epc pages on reclaimable or unreclaimable listsarch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/ioctl.c arch/x86/kernel/cpu/sgx/main.c +2 more+51 -282 replies
6/18x86/sgx: Introduce RECLAIM_IN_PROGRESS flag for EPC pagesarch/x86/kernel/cpu/sgx/main.c arch/x86/kernel/cpu/sgx/sgx.h+12 -53 replies
7/18x86/sgx: Use a list to track to-be-reclaimed pages during reclaimarch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/ioctl.c arch/x86/kernel/cpu/sgx/main.c +1 more+85 -384 replies
8/18x86/sgx: Allow reclaiming up to 32 pages, but scan 16 by defaultarch/x86/kernel/cpu/sgx/main.c+15 -102 replies
9/18x86/sgx: Return the number of EPC pages that were successfully reclaimedarch/x86/kernel/cpu/sgx/main.c+8 -41 reply
10/18x86/sgx: Add option to ignore age of page during EPC reclaimarch/x86/kernel/cpu/sgx/main.c+29 -171 reply
11/18x86/sgx: Prepare for multiple LRUsarch/x86/kernel/cpu/sgx/main.c+7 -21 reply
12/18x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgrouparch/x86/kernel/cpu/sgx/main.c arch/x86/kernel/cpu/sgx/sgx.h+5 -31 reply
13/18x86/sgx: Add helper to grab pages from an arbitrary EPC LRUarch/x86/kernel/cpu/sgx/main.c arch/x86/kernel/cpu/sgx/sgx.h+44 -261 reply
14/18x86/sgx: Add EPC OOM path to forcefully reclaim EPCarch/x86/kernel/cpu/sgx/encl.c arch/x86/kernel/cpu/sgx/encl.h arch/x86/kernel/cpu/sgx/main.c +1 more+201 -115 replies
15/18cgroup/misc: Add per resource callbacks for css eventsinclude/linux/misc_cgroup.h kernel/cgroup/misc.c+54 -32 replies
16/18cgroup/misc: Prepare for SGX usageinclude/linux/misc_cgroup.h kernel/cgroup/misc.c+41 -131 reply
17/18x86/sgx: Add support for misc cgroup controllerarch/x86/Kconfig arch/x86/kernel/cpu/sgx/Makefile arch/x86/kernel/cpu/sgx/epc_cgroup.c +3 more+688 -161 reply
18/18Docs/x86/sgx: Add description for cgroup supportDocumentation/x86/sgx.rst+77 -00 replies

Reviews trailers per patch

1 of 18 patches has a Reviewed-by or Acked-by.

PatchBagas Sanjaya
1/18
2/18
3/18
4/18
5/18
6/18
7/18
8/18
9/18
10/18
11/18
12/18
13/18
14/18
15/18
16/18
17/18
18/18R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 00/18] Add Cgroup support for SGX EPC memory

Utilize the Miscellaneous cgroup controller to regulate the distribution of SGX EPC memory, which is a subset of system RAM that is used to provide SGX-enabled applications with protected memory, and is otherwise inaccessible.

SGX EPC memory allocations are separate from normal RAM allocations, and is managed solely by the SGX subsystem. The existing cgroup memory controller cannot be used to limit or account for SGX EPC memory.

This patchset implements the support for sgx_epc memory within the misc cgroup controller, and then utilizes the misc cgroup controller to provide support for setting the total system capacity, max limit per cgroup, and events.

This work was originally authored by Sean Christopherson a few years ago, and was modified to work with more recent kernels, and to utilize the misc cgroup controller rather than a custom controller. It is currently based on top of the MCA patches.

Here's the MCA patchset for reference.
https://lore.kernel.org/linux-sgx/2d52c8c4-8ed0-6df2-2911-da5b9fcc9ae4-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org/T/#t

The patchset adds support for multiple LRUs to track both reclaimable EPC pages (i.e. pages the reclaimer knows about), as well as unreclaimable EPC pages (i.e. pages which the reclaimer isn't aware of, such as va pages). These pages are assigned to an LRU, as well as an enclave, so that an enclave's full EPC usage can be tracked, and limited to a max value. During OOM events, an enclave can be have its memory zapped, and all the EPC pages not tracked by the reclaimer can be freed.

I appreciate your comments and feedback.

Changelog:

v2:
* rename struct sgx_epc_lru to sgx_epc_lru_lists to be more clear that this struct contains 2 lists.
* use inline functions rather than macros for sgx_epc_page_list* wrappers.
* Remove flags macros and open code all flags.
* Improve the commit message for RECLAIM_IN_PROGRESS patch to make it more clear what the patch does.
* remove notifier_block from misc cgroup changes and use a set of ops for callbacks instead.
* rename root_misc to misc_cg_root and parent_misc to misc_cg_parent
* consolidate misc cgroup changes to 2 patches and remove most of the previous helper functions.

Kristen Carlson Accardi (7):
x86/sgx: Add 'struct sgx_epc_lru_lists' to encapsulate lru list(s) x86/sgx: Use sgx_epc_lru_lists for existing active page list x86/sgx: Track epc pages on reclaimable or unreclaimable lists cgroup/misc: Add per resource callbacks for css events cgroup/misc: Prepare for SGX usage
x86/sgx: Add support for misc cgroup controller Docs/x86/sgx: Add description for cgroup support

Sean Christopherson (11):
x86/sgx: Call cond_resched() at the end of sgx_reclaim_pages() x86/sgx: Store struct sgx_encl when allocating new VA pages x86/sgx: Introduce RECLAIM_IN_PROGRESS flag for EPC pages x86/sgx: Use a list to track to-be-reclaimed pages during reclaim x86/sgx: Allow reclaiming up to 32 pages, but scan 16 by default x86/sgx: Return the number of EPC pages that were successfully
reclaimed
x86/sgx: Add option to ignore age of page during EPC reclaim x86/sgx: Prepare for multiple LRUs
x86/sgx: Expose sgx_reclaim_pages() for use by EPC cgroup x86/sgx: Add helper to grab pages from an arbitrary EPC LRU x86/sgx: Add EPC OOM path to forcefully reclaim EPC

 Documentation/x86/sgx.rst            |  77 ++++
 arch/x86/Kconfig                     |  13 +
 arch/x86/kernel/cpu/sgx/Makefile     |   1 +
 arch/x86/kernel/cpu/sgx/encl.c       |  90 ++++-
 arch/x86/kernel/cpu/sgx/encl.h       |   4 +-
 arch/x86/kernel/cpu/sgx/epc_cgroup.c | 539 +++++++++++++++++++++++++++
 arch/x86/kernel/cpu/sgx/epc_cgroup.h |  59 +++
 arch/x86/kernel/cpu/sgx/ioctl.c      |  14 +-
 arch/x86/kernel/cpu/sgx/main.c       | 412 ++++++++++++++++----
 arch/x86/kernel/cpu/sgx/sgx.h        | 122 +++++-
 arch/x86/kernel/cpu/sgx/virt.c       |  28 +-
 include/linux/misc_cgroup.h          |  35 ++
 kernel/cgroup/misc.c                 |  76 +++-
 13 files changed, 1341 insertions(+), 129 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/sgx/epc_cgroup.c
 create mode 100644 arch/x86/kernel/cpu/sgx/epc_cgroup.h
Signature · 1 line
2.38.1