Skip to content

RFC Support DEVICE_GENERIC memory in migrate_vma_*

Felix Kuehling amd-gfx dri-devel linux-mm 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest5 patches 9 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/5drm/amdkfd: add SPM support for SVMdrivers/gpu/drm/amd/amdkfd/kfd_migrate.c drivers/gpu/drm/amd/amdkfd/kfd_svm.h kernel/resource.c+10 -52 replies
2/5drm/amdkfd: generic type as sys mem on migration to ramdrivers/gpu/drm/amd/amdkfd/kfd_migrate.c+2 -10 replies
3/5include/linux/mm.h: helper to check zone device generic typeinclude/linux/mm.h+7 -00 replies
4/5mm: add generic type support for device zone page migrationmm/migrate.c+8 -51 reply
5/5mm: changes to unref pages with Generic typeinclude/linux/mm.h mm/memremap.c+5 -12 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/5] Support DEVICE_GENERIC memory in migrate_vma_*

AMD is building a system architecture for the Frontier supercomputer with a coherent interconnect between CPUs and GPUs. This hardware architecture allows the CPUs to coherently access GPU device memory. We have hardware in our labs and we are working with our partner HPE on the BIOS, firmware and software for delivery to the DOE.

The system BIOS advertises the GPU device memory (aka VRAM) as SPM (special purpose memory) in the UEFI system address map. The amdgpu driver looks it up with lookup_resource and registers it with devmap as MEMORY_DEVICE_GENERIC using devm_memremap_pages.

Now we're trying to migrate data to and from that memory using the migrate_vma_* helpers so we can support page-based migration in our unified memory allocations, while also supporting CPU access to those pages.

This patch series makes a few changes to make MEMORY_DEVICE_GENERIC pages behave correctly in the migrate_vma_* helpers. We are looking for feedback about this approach. If we're close, what's needed to make our patches acceptable upstream? If we're not close, any suggestions how else to achieve what we are trying to do (i.e. page migration and coherent CPU access to VRAM)?

This work is based on HMM and our SVM memory manager that was recently upstreamed to Dave Airlie's drm-next branch
[https://cgit.freedesktop.org/drm/drm/log/?h=drm-next]. On top of that we did some rework of our VRAM management for migrations to remove some incorrect assumptions, allow partially successful migrations and GPU memory mappings that mix pages in VRAM and system memory.
[https://patchwork.kernel.org/project/dri-devel/list/?series=489811]

In this RFC, patches 1 and 2 are for context to show how we are looking up the SPM memory and registering it with devmap.

Patches 3-5 are the changes we are trying to upstream or rework to make them acceptable upstream.

Alex Sierra (5):
drm/amdkfd: add SPM support for SVM
drm/amdkfd: generic type as sys mem on migration to ram include/linux/mm.h: helper to check zone device generic type
mm: add generic type support for device zone page migration
mm: changes to unref pages with Generic type

 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 15 +++++++++++----
 drivers/gpu/drm/amd/amdkfd/kfd_svm.h     |  1 -
 include/linux/mm.h                       |  8 ++++++++
 kernel/resource.c                        |  2 +-
 mm/memremap.c                            |  5 ++++-
 mm/migrate.c                             | 13 ++++++++-----
 6 files changed, 32 insertions(+), 12 deletions(-)
Signature · 1 line
2.31.1