Skip to content

Introduce nvgrace-egm driver for Extended GPU Memory

Ankit Agrawal lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 12 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM regionsMAINTAINERS drivers/platform/Kconfig drivers/platform/Makefile +3 more+479 -04 replies
2/4platform/nvidia: Implement mmap and memory scrubbing for EGM chardevdrivers/platform/nvidia/egm.c+121 -42 replies
3/4platform/nvidia: Handle retired ECC pages and expose via ioctlDocumentation/userspace-api/ioctl/ioctl-number.rst MAINTAINERS drivers/platform/nvidia/egm.c +1 more+231 -102 replies
4/4platform/nvidia: Register EGM PFNMAP range with memory_failuredrivers/platform/nvidia/egm.c+115 -80 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/4] Introduce nvgrace-egm driver for Extended GPU Memory

Background
==========
NVIDIA Grace-based systems (Grace Hopper, Grace Blackwell Superchip) have a special bios-configurable mode called EGM (Extended GPU Memory) that allows a carveout to be created from the system memory. This carveout is a contiguous region of special range of system DRAM reserved by the UEFI boot firmware.

This range does not appear in the EFI memory map and is therefore invisible to the host kernel. i.e. there are no struct pages for it and it cannot be managed by the normal page allocator. This range is described in ACPI as vendor DSD properties on the GPU PCI device node (see ACPI Design below). It is intended to be used by a VMM to back an entire virtual machine's physical memory. The VMM maps the range into its address space via mmap() on a char device and passes it to KVM as the backing store for guest RAM similar to memfd or hugetlbfs. KVM builds Stage-2 page tables mapping the guest's GPA range linearly to the EGM SPAs i.e. a guest offset X maps to SPA (egm_base + X).

The range is special because the GPU on these systems has direct high-speed access to it that bypasses the host system IOMMU. Normal PCI DMA is gated by the IOMMU and requires an explicit IOMMU mapping. The GPU's path to this memory goes via NVLink/NVSwitch directly to the memory controller and does not involve the host IOMMU. Since KVM maps guest GPA to EGM SPA linearly, the GPU can reach guest memory at those same SPAs without needing any additional host IOMMU mapping. Normal system memory that the kernel sees is protected by the IOMMU in the usual way and this range is deliberately kept separate.

To make this work a driver is needed to parse the ACPI, capture the physical memory range, expose it as a char device and allow it to be mmap'd for KVM. A VMM like qemu can open this char dev and then use it much like a hugetlbfs or memfd to back the VM. Due to the security difference the memory has to be strictly separated from the rest of the hypervisor memory. Exposing the range as a DMABUF for iommufd is planned as future work and is not part of this series.

This hardware feature allows the system memory to be access across nodes on a multi-node system such as Grace Blackwell NVL72. It can also provide throughput improvements because the GPU can access VM system memory via its high-bandwidth fabric rather than going through the IOMMU-gated PCIe path. Moreover skipping IOMMU translations could also improve performance for any use cases targeted towards high performance.

Due to the security difference, the EGM range is strictly isolated from the rest of the hypervisor memory (which is covered through IOMMU).

A driver is needed to:
- Parse ACPI to locate the physical range (base SPA, size, proximity domain)
- Expose the range as a char device (/dev/egmX, X = NUMA proximity domain) that a VMM such as QEMU can open and mmap to back guest RAM
- Zero the region on first open to isolate successive VM uses
- Track and expose retired ECC pages to the VMM via ioctl
- Register the PFN range with the memory_failure infrastructure for runtime uncorrectable error handling

ACPI Design
===========
The EGM region properties are exposed as NVIDIA vendor DSD properties on the GPU's PCI ACPI device node:

nvidia,egm-pxm NUMA proximity domain of the region
nvidia,egm-base-pa Physical base address (SPA) of the
region
nvidia,egm-size Length of the region in bytes
nvidia,egm-retired-pages-data-base Physical address of the firmware-
provided retired ECC pages table

This ACPI construct helps with describing and discovering this physically present memory range that is intentionally absent from the EFI memory map and is tied to a specific device for its access properties.

On a multi-socket system there is one EGM region per socket. Multiple GPUs on the same socket advertise identical region properties (same proximity domain, base address and size). The driver deduplicates by proximity
domain: the first GPU probed for a given domain creates the char device and subsequent GPUs on the same socket are linked to it via sysfs.

Implementation
==============
Patch 1 introduces the driver skeleton: Kconfig/Makefile/MAINTAINERS, the egm device class, the char device minor number range (MAX_EGM_NODES=4 matching the largest 4-socket Grace configuration), ACPI property walking (nvidia,egm-pxm / nvidia,egm-base-pa / nvidia,egm-size), per-region state tracking with PXM-based deduplication, char device creation (/dev/egmX), GPU-to-EGM sysfs symlinks and the egm_size sysfs attribute.

Patch 2 implements the file operations: memory scrubbing on the first open() using memremap()/memset() in 1 GiB chunks with cond_resched() to avoid RCU stalls and mmap() via remap_pfn_range().

Patch 3 adds retired ECC page handling: reads the firmware-provided retired-pages table from the ACPI DSD property nvidia,egm-retired-pages-data-base at module init, stores the page offsets in a per-device xarray and exposes them to userspace via the EGM_RETIRED_PAGES_LIST ioctl (UAPI header include/uapi/linux/egm.h). Each firmware entry covers a 64 KiB retired region. The driver inserts one xarray entry per kernel page so a lookup at the running page size hits the right retired range on both 4K and 64K kernels.

Patch 4 registers the EGM PFN range with the memory_failure infrastructure on the first open() and unregisters it on the last close(). The pfn_to_vma_pgoff callback validates that a reported PFN falls within the EGM region, converts it to an in-region file offset, and inserts it into the retired-pages xarray so runtime uncorrectable errors are surfaced alongside ACPI-reported retired pages via the same ioctl.

Enablement
==========
EGM mode is enabled through a flag in the platform firmware (SBIOS). The size of the host-visible partition is separately configurable; all remaining system DRAM is reserved as the EGM region invisible to the hypervisor.

RFC Posting
===========

Verification
============
Tested on a 2-socket (4 GPU) Grace Blackwell platform by booting a VM with QEMU [1] and kernel branch [2] and confirming EGM capability is visible inside the guest and basic cuda workloads. Note that this has dependency on the generic dmabuf importer/exporter work in progress [3]. Until then, a bypass PFNMAP workaround is used [4] (also part of the kernel branch [2].

Ankit Agrawal (4):
platform/nvidia: Introduce nvgrace-egm driver and enumerate EGM
regions
platform/nvidia: Implement mmap and memory scrubbing for EGM chardev platform/nvidia: Handle retired ECC pages and expose via ioctl platform/nvidia: Register EGM PFNMAP range with memory_failure

 .../userspace-api/ioctl/ioctl-number.rst      |   1 +
 MAINTAINERS                                   |   6 +
 drivers/platform/Kconfig                      |   1 +
 drivers/platform/Makefile                     |   1 +
 drivers/platform/nvidia/Kconfig               |  10 +
 drivers/platform/nvidia/Makefile              |   3 +
 drivers/platform/nvidia/egm.c                 | 874 ++++++++++++++++++
 include/uapi/linux/egm.h                      |  28 +
 8 files changed, 924 insertions(+)
 create mode 100644 drivers/platform/nvidia/Kconfig
 create mode 100644 drivers/platform/nvidia/Makefile
 create mode 100644 drivers/platform/nvidia/egm.c
 create mode 100644 include/uapi/linux/egm.h
Signature · 1 line
2.34.1