Add UALink infrastructure series 2
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest posting 3/33 patches 7 repliesIn review
- v2 posting 2/395 patches 0 repliesSuperseded
- v2 posting 1/33 patches 0 repliesSuperseded
Earlier versions · 1
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v1 | 3 patches | 4 replies | Superseded |
Changes in v2 from the cover letter
- Update the Documentation based on review feedback
Patches in v2 posting 3/3 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/3 | drm/gem: Add callback for when handle count goes to 0drivers/ drivers/ include/ | +24 | 7 replies | |
| 2/3 | drm/amdgpu: Add ioctl infra for exporting/importing UALink handlesdrivers/ drivers/ drivers/ +1 more | +78 | 0 replies | |
| 3/3 | Documentation: add initial UALink DocumentationDocumentation/ Documentation/ | +76 | 0 replies |
Reviews trailers per patch
1 of 3 patches has a Reviewed-by or Acked-by.
| Patch | Felix Kuehling |
|---|---|
| 1/3 | R-b |
| 2/3 | |
| 3/3 |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.
Cover letter [PATCH V2 0/3] Add UALink infrastructure series 2
This adds the core infrastructure for supporting UALink (Ultra Accelerator Link) connected scale up pods. I've split this into two series, one to add the core infrastructure, and one to expose the new IOCTL interface and add the documentation to avoid spamming the larger audience with the implemenation defails. This is the second series.
The first series can be found here:
V2: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151914.html
V1: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151200.html
Both series can be accessed via this git branch:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads
This implements UALoE (UALink over Ethernet). An overview of the complete solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html
Overview
Connected GPUs in a pod can directly access the remove memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Nework Physical Address) space. This address space is partitioned between the GPUs so that each GPU has it's own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM.
Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned for P2P access. Remote TLB shootdowns from the exporter are used when the exported memory is freed in order to remove access by remote GPUs
To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining refrences that are lost when that happens.
User interface
Export Memory
To export memory, a handle must be created for an allocation that can be shared with another node in the pod. To do this the exporter calls the GEM UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL returns a unique handle which can be shared with the remote host.
Import Memory
To import the memory, the handle from the remote node must be converted from a unique handle to a local GEM object which represents the local reference to the NPA space on the importer. If the memory has already been imported, it just returns a new reference to the existing object. If not, the importer queries the exporter to get the NPA address. Once it has that the importer can create a dma-buf to represent the NPA space used by the allocation and that is returned to the application.
Proposed Userspace
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036
v2:
- Update the Documentation based on review feedback
Alex Deucher (1):
Documentation: add initial UALink Documentation
Mukul Joshi (2):
drm/gem: Add callback for when handle count goes to 0 drm/amdgpu: Add ioctl infra for exporting/importing UALink handles
Documentation/gpu/amdgpu/index.rst | 1 + Documentation/gpu/amdgpu/ualink.rst | 75 ++++++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 2 + drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 9 +++ drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h | 2 + drivers/gpu/drm/drm_gem.c | 5 +- include/drm/drm_gem.h | 11 ++++ include/uapi/drm/amdgpu_drm.h | 30 +++++++++ 9 files changed, 178 insertions(+), 1 deletion(-) create mode 100644 Documentation/gpu/amdgpu/ualink.rst
Signature · 1 line
2.55.0