Skip to content

RFC Enable render node VA mapping API for KFD BOs

Felix Kuehling amd-gfx dri-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4drm/amdkfd: Improve amdgpu_vm_handle_moveddrivers/gpu/drm/amd/amdgpu/amdgpu_cs.c drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 more+21 -84 replies
2/4drm/amdgpu: Attach eviction fence on allocdrivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+31 -310 replies
3/4drm/amdgpu: update mappings not managed by KFDdrivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+15 -00 replies
4/4drm/amdgpu: Do bo_va ref counting for KFD BOsdrivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c+21 -50 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/4] Enable render node VA mapping API for KFD BOs

The KFD API is quite inflexible in that it allows only mapping entire BOs at the same virtual address on all GPUs. This is incompatible with newer CUDA memory management APIs.
(see https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html)

Instead of inventing more KFD APIs to fix this, the goal of this patch series is to enable ROCr to use the DRM render node API for more flexible VA mappings. It builds on the DMABuf export API that is being proposed for RDMA. DMABuf handles exported by KFD can be imported by libdrm-amdgpu and then used with amdgpu_bo_va_op.

This is a first proof of concept and request for comment.

A complete solution will likely need minor tweaks to the KFD API to allow memory allocation and import without a pre-determined virtual address.

Other options that were considered but rejected:
- Using GEM API to create BO in KFD VMs. This would require significant plumbing to get those BOs registered with the KFD eviction fence mechanism and "no overcommitment" memory limits
- Variation of the above using AMDGPU_GEM_CREATE_VM_ALWAYS_VALID to simplify validation. Doesn't work because it doesn't allow DMABuf exports
- Creating KFD BOs with GEM handles. Doesn't help because there is no way to import GEM handles into libdrm-amdgpu

Felix Kuehling (4):
drm/amdkfd: Improve amdgpu_vm_handle_moved
drm/amdgpu: Attach eviction fence on alloc
drm/amdgpu: update mappings not managed by KFD
drm/amdgpu: Do bo_va ref counting for KFD BOs

 .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c  | 99 ++++++++++++-------
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c        |  6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c   |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 18 +++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h        |  3 +-
 5 files changed, 86 insertions(+), 42 deletions(-)
Signature · 1 line
2.32.0