Skip to content

Add UALink infrastructure series 2

Alex Deucher amd-gfx dri-devel linux-next rust-for-linux 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest posting 3/33 patches 7 repliesIn review
  2. v2 posting 2/395 patches 0 repliesSuperseded
  3. v2 posting 1/33 patches 0 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v13 patches4 repliesSuperseded

Changes in v2 from the cover letter

- Update the Documentation based on review feedback

Patches in v2 posting 3/3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3drm/gem: Add callback for when handle count goes to 0drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c drivers/gpu/drm/drm_gem.c include/drm/drm_gem.h+24 -17 replies
2/3drm/amdgpu: Add ioctl infra for exporting/importing UALink handlesdrivers/gpu/drm/amd/amdgpu/amdgpu_drv.c drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h +1 more+78 -00 replies
3/3Documentation: add initial UALink DocumentationDocumentation/gpu/amdgpu/index.rst Documentation/gpu/amdgpu/ualink.rst+76 -00 replies

Reviews trailers per patch

1 of 3 patches has a Reviewed-by or Acked-by.

PatchFelix Kuehling
1/3R-b
2/3
3/3

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH V2 0/3] Add UALink infrastructure series 2

This adds the core infrastructure for supporting UALink (Ultra Accelerator Link) connected scale up pods. I've split this into two series, one to add the core infrastructure, and one to expose the new IOCTL interface and add the documentation to avoid spamming the larger audience with the implemenation defails. This is the second series.

The first series can be found here:
V2: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151914.html
V1: https://lists.freedesktop.org/archives/amd-gfx/2026-August/151200.html

Both series can be accessed via this git branch:
https://gitlab.freedesktop.org/agd5f/linux/-/commits/ualink?ref_type=heads

This implements UALoE (UALink over Ethernet). An overview of the complete solution can be found here:
https://www.amd.com/en/products/rackscale-solutions/helios.html

Overview

Connected GPUs in a pod can directly access the remove memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Nework Physical Address) space. This address space is partitioned between the GPUs so that each GPU has it's own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM.

Exported memory is pinned. This is similar to how dma-bufs in VRAM are pinned for P2P access. Remote TLB shootdowns from the exporter are used when the exported memory is freed in order to remove access by remote GPUs

To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining refrences that are lost when that happens.

User interface

Export Memory

To export memory, a handle must be created for an allocation that can be shared with another node in the pod. To do this the exporter calls the GEM UALink IOCTL with the GEM handle to the buffer it wants to export. The IOCTL returns a unique handle which can be shared with the remote host.

Import Memory

To import the memory, the handle from the remote node must be converted from a unique handle to a local GEM object which represents the local reference to the NPA space on the importer. If the memory has already been imported, it just returns a new reference to the existing object. If not, the importer queries the exporter to get the NPA address. Once it has that the importer can create a dma-buf to represent the NPA space used by the allocation and that is returned to the application.

Proposed Userspace
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L971
https://github.com/ROCm/rocm-systems/blob/35959f8e1260c7cee3e51a740e320be3856ee4ff/projects/rocr-runtime/libhsakmt/src/memory.c#L1036

v2:
- Update the Documentation based on review feedback

Alex Deucher (1):
Documentation: add initial UALink Documentation

Mukul Joshi (2):
drm/gem: Add callback for when handle count goes to 0 drm/amdgpu: Add ioctl infra for exporting/importing UALink handles

 Documentation/gpu/amdgpu/index.rst         |  1 +
 Documentation/gpu/amdgpu/ualink.rst        | 75 ++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    |  2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c    |  9 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.c | 44 +++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ualink.h |  2 +
 drivers/gpu/drm/drm_gem.c                  |  5 +-
 include/drm/drm_gem.h                      | 11 ++++
 include/uapi/drm/amdgpu_drm.h              | 30 +++++++++
 9 files changed, 178 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/gpu/amdgpu/ualink.rst
Signature · 1 line
2.55.0