nitro_enclaves: Add support for Nitro Sandboxes

Nitro Sandboxes [1] are a new EC2 feature that build on top of the technical foundations of Nitro Enclaves - an EC2 feature that allows you to spawn a virtio-vsock-only VM sibling constructed out of resources donated by the VM you execute in.

Nitro Sandboxes build on that concept and the same hypervisor interface, but add the ability to implement virtio devices in the parent, run vCPUs oversubscribed, add device hotplug support and allow to spawn a sandbox without attestation, but shared memory. They also boot a full EC2-like VM that runs with a UEFI stack and executes the EIF-wrapped kernel as UEFI executable.

In combination, Nitro Sandboxes provide a way to create a confidential compute VM for almost all workloads, but without the biggest gaps that other confidential compute offerings have: The parent has no timing control and the parent also has no control over the base device model, such as timers, PCIe, UART, etc. All base devices are implemented in Nitro.

This series adds parent-side support for Nitro Sandboxes to the nitro_enclaves driver.

This supports adds a few non-obvious technical foundations:

guest_memfd like CMA allocator - With Nitro Enclaves, memory came from hugetlbfs and user space maintained visibility into it. That led to a few problems. With this new enablement, we add a guest_memfd like allocator that seeds from CMA, allows the parent to seed the EIF before hand-over and then loses any mechanism to access donated memory. Thanks to CMA, it also provides much larger contiguous chunks than 2M huge pages.

Virtio notifications - To implement virtio in the parent, we need to handle notifications in both directions. Parent to sandbox goes through a BAR write; sandbox to parent through multiplexed IRQs. We provide eventfd support in both directions, which also enables vhost support.

NUMA - With Nitro Sandboxes, NUMA became a first class citizen and all APIs now support NUMA targeting.

Memory Donations - Every sandbox requires hypervisor memory. To allow for a larger scale of sandboxes you can run, we add a way to donate memory from the parent to the hypervisor. The user visible effect of that is that there is a slot limit sysfs file which you can increase. On write, we allocate physically contiguous CMA memory and donate it.

This series depends on "nitro_enclaves: Fix enclave release, command retries and the CPU pool", which I posted earlier today.

I used Kiro, an AI coding assistant running Claude Opus and GPT models, to draft the initial code and patch descritptions.

I validated all new functionality on x86_64 and arm64 as well as ensured that Nitro Enclaves are still fully functional after this patch series.

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-ec2-nitro-sandboxes.html

Alex

Alexander Graf (40):
nitro_enclaves: Allow the CPU pool to span NUMA nodes
nitro_enclaves: Add NE_ADD_ANY_VCPU
nitro_enclaves: Expose CPU pool state under sysfs
nitro_enclaves: Document multi-NUMA CPU pools
dma-contiguous: add dma_get_numa_cma_area() accessor
nitro_enclaves: Allocate physically contiguous enclave memory
nitro_enclaves: Add NE_MEMORY_SHARED
nitro_enclaves: Add NE_ADD_MEMORY_REGION
nitro_enclaves: Document the memory pool
nitro_enclaves: Add NE_CREATE_VM2 with a launch flag word
nitro_enclaves: Add NE_ENCLAVE_PCIE_MODE launch flag
nitro_enclaves: Name the device's error codes in the UAPI header
nitro_enclaves: Log a reached device limit at info level
nitro_enclaves: Add NE_SET_INFO_PAGE
nitro_enclaves: Document the info page in the memory pool
nitro_enclaves: Turn the command reply into a per-command union
nitro_enclaves: Add the slot pool donation commands
nitro_enclaves: Donate memory for the slot pool
nitro_enclaves: Add the slot pool growth commands
nitro_enclaves: Add a helper to grow the slot pool
nitro_enclaves: Document the slot pool
nitro_enclaves: Expose slot pool state under sysfs
nitro_enclaves: Add the device add command
nitro_enclaves: Add NE_ADD_DEVICE
nitro_enclaves: Read the interrupt multiplexer geometry at probe
nitro_enclaves: Donate a bitmap page per multiplexed vector
nitro_enclaves: Dispatch multiplexed interrupts to per-queue bindings
nitro_enclaves: Add NE_SET_NOTIFY_EVENTFD
nitro_enclaves: Report the device's info notification
nitro_enclaves: Count multiplexed interrupts under debugfs
nitro_enclaves: Map the doorbell window at probe
nitro_enclaves: Add NE_SET_VRING_CALL
nitro_enclaves: Assert the layout of the ioctl argument structs
nitro_enclaves: Pass on the Nitro Sandbox launch flags
nitro_enclaves: Add NE_ENCLAVE_DIRECT_MODE
nitro_enclaves: Add NE_ENCLAVE_PCIE_HOTPLUG_MODE
nitro_enclaves: Add NE_ENCLAVE_CPU_OVERCOMMIT_MODE
nitro_enclaves: Add NE_GET_CAPS
nitro_enclaves: Add NE_UPDATE_DEVICE_CONFIG
nitro_enclaves: Document the driver side of Nitro Sandboxes

 .../sysfs-devices-virtual-misc-nitro_enclaves |  232 +
 Documentation/virt/ne_overview.rst            |  256 +-
 MAINTAINERS                                   |    1 +
 drivers/virt/nitro_enclaves/Kconfig           |   19 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c     | 4832 +++++++++++++++--
 drivers/virt/nitro_enclaves/ne_misc_dev.h     |  206 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.c      | 1568 +++++-
 drivers/virt/nitro_enclaves/ne_pci_dev.h      |  599 +-
 include/linux/dma-map-ops.h                   |    7 +
 include/uapi/linux/nitro_enclaves.h           | 1182 +++-
 kernel/dma/contiguous.c                       |   36 +
 samples/nitro_enclaves/ne_ioctl_sample.c      |   10 +-
 12 files changed, 8282 insertions(+), 666 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
prerequisite-patch-id: 542b3d78fda912095b75a49becd1b93109befdbc
prerequisite-patch-id: 318368e966c1925652b7976948f435d42b87c72f
prerequisite-patch-id: eb35b5d713e7f50f6d1d02310c9afe10b1628fc6
prerequisite-patch-id: 223bf6e60620ca8fe72f3ff6f3bd3f5839cdd6d5
prerequisite-patch-id: e5305ce0515a02f7f88e72adf02c6a2bee2fd2e0
prerequisite-patch-id: 759b5920f842d70e675770608d9088639c10559e
prerequisite-patch-id: e1837df37663da3d3ce53afa4594828b2d602152
prerequisite-patch-id: f0d4302bd36f94de95951a0db1ae9eb9747076aa
prerequisite-patch-id: 41159b92e250fef705be52f714b6e2761a2cd1f7
prerequisite-patch-id: a0c3a56907acac2297c800209faa2849a8df0e48
prerequisite-patch-id: a80a95bd682b540be5760e8fdf7044379fe32d61
prerequisite-patch-id: 9c01e0fecd6810f3448582715c87cd3dd7a6cdf1
prerequisite-patch-id: b94dfc8bb1039740d88a30535445bb7b3dd1fe74
prerequisite-patch-id: bd76f15728dc0ee2ca3417cd95a6b23a7d63e91f
prerequisite-patch-id: 0c60f1574b590f68885a25760017e9ff87752f66
prerequisite-patch-id: 63dff43da52934a6e6f5e65c3e235a7114b6b4dc
prerequisite-patch-id: af52cf938525874cadd6d879aa76495e563876d2
prerequisite-patch-id: f53ee22764b5f13708cacd3e48a59a82d71a270c
prerequisite-patch-id: 258b8c18168db49aacbda5156e33978b539666ab

2.47.1