Skip to content

nitro_enclaves: Support multi-NUMA CPU pools and per-node allocation

Alexander Graf linux-doc lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest8 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8nitro_enclaves: Let 32-bit processes use the ioctl interfacedrivers/virt/nitro_enclaves/ne_misc_dev.c+2 -00 replies
2/8nitro_enclaves: Initialise the CPU pool mutex staticallydrivers/virt/nitro_enclaves/ne_misc_dev.c+3 -30 replies
3/8nitro_enclaves: Slot pool cores by sibling maskdrivers/virt/nitro_enclaves/ne_misc_dev.c drivers/virt/nitro_enclaves/ne_misc_dev.h+125 -650 replies
4/8nitro_enclaves: Allow the CPU pool to span NUMA nodesdrivers/virt/nitro_enclaves/ne_misc_dev.c drivers/virt/nitro_enclaves/ne_misc_dev.h include/uapi/linux/nitro_enclaves.h+110 -510 replies
5/8nitro_enclaves: Add NE_SET_ALLOC_NUMA_NODEdrivers/virt/nitro_enclaves/ne_misc_dev.c include/uapi/linux/nitro_enclaves.h+80 -01 reply
6/8nitro_enclaves: Expose CPU pool state under sysfsdrivers/virt/nitro_enclaves/ne_misc_dev.c+122 -121 reply
7/8Documentation: ABI: Describe nitro_enclaves cpu_pool sysfsDocumentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves MAINTAINERS+45 -00 replies
8/8Documentation: virt: Describe multi-NUMA Nitro Enclaves CPU poolsDocumentation/virt/ne_overview.rst+24 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/8] nitro_enclaves: Support multi-NUMA CPU pools and per-node allocation

An enclave draws its vCPU cores from the driver's single CPU pool, and by design every CPU in that pool sits on one NUMA node: ne_cpus= refuses a wider CPU list with -EINVAL and "CPUs with different NUMA nodes" in dmesg. An operator who wants two enclaves on two nodes has nowhere to put the second one's cores, and an enclave wider than one node has to spare cannot be built at all.

Lift it, and let a caller name the node. A pool spanning nodes records itself as node-agnostic. NE_SET_ALLOC_NUMA_NODE sets a sticky allocation target on the enclave fd and NE_ALLOC_NUMA_NODE_ANY drops it; NE_ADD_VCPU with vcpu_id 0 draws from the target and returns the id it picked, so placing an enclave across two nodes is setting the target once per node and draining the count. A named node does not spill, yielding a core on that node or NE_ERR_NO_CPUS_AVAIL_IN_POOL. A cpu_pool attribute group on the misc device reports mode, total, used and avail as CPU-list strings; pool threads are offline, so the per-node view comes from intersecting avail with the node<N>/cpu<M> symlinks under /sys/devices/system/node.

I picked a per-fd target over a second NE_ADD_VCPU carrying the node on every call. NE_ADD_VCPU already reports the id it chose, so a VMM spreading an enclave over several nodes needs no new call, only the target and the count it already drains; the variant would make that same VMM learn a new ioctl for behaviour it already has.

Patches 1 to 3 are not the feature: a missing .compat_ioctl, a pool mutex taken before mutex_init(), and a thread-mask index that collides cores from different packages, the last preparation with no functional change. Both fixes carry Fixes: and neither Cc: stable, the mutex splat needing CONFIG_DEBUG_MUTEXES to appear.

Nothing consumes this yet; the kernel API goes first, and its shape is what nitro-cli, or a VMM like QEMU, gets built against. A dynamic pool mode and CMA-sourced enclave memory, both reusing this target, follow in separate series.

I wrote this series with an AI coding assistant, which drafted the code and the changelogs, this cover letter included; I reviewed and reworked all of it before sending, and each commit carries an Assisted-by:
trailer. All eight commits were built detached in three configurations (x86_64 defconfig with CONFIG_NITRO_ENCLAVES=m, the same with CONFIG_NUMA=n, and allmodconfig) for 24 warning-free driver builds, and scripts/checkpatch.pl --strict is clean on all eight.

Alexander Graf (8):
nitro_enclaves: Let 32-bit processes use the ioctl interface
nitro_enclaves: Initialise the CPU pool mutex statically
nitro_enclaves: Slot pool cores by sibling mask
nitro_enclaves: Allow the CPU pool to span NUMA nodes
nitro_enclaves: Add NE_SET_ALLOC_NUMA_NODE
nitro_enclaves: Expose CPU pool state under sysfs
Documentation: ABI: Describe nitro_enclaves cpu_pool sysfs
Documentation: virt: Describe multi-NUMA Nitro Enclaves CPU pools

 .../sysfs-devices-virtual-misc-nitro_enclaves |  44 ++
 Documentation/virt/ne_overview.rst            |  25 +-
 MAINTAINERS                                   |   1 +
 drivers/virt/nitro_enclaves/ne_misc_dev.c     | 489 +++++++++++++-----
 drivers/virt/nitro_enclaves/ne_misc_dev.h     |  16 +-
 include/uapi/linux/nitro_enclaves.h           |  62 ++-
 6 files changed, 508 insertions(+), 129 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-virtual-misc-nitro_enclaves

base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff

Signature · 1 line
2.47.1