Skip to contentSkip to conversation

*** GPU Direct RDMA (P2P DMA) for Device Private Pages ***

PATCH In review Yonatan Maman 8 participants 37 messages Jul 18, 2025 – Aug 5, 2025

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v2 cover *** GPU Direct RDMA (P2P DMA) for Device Private Pages ***

From: Yonatan Maman <[email protected]>

This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement reduces data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.

To fully support Peer-to-Peer for device private pages, the following changes are proposed:

Memory Management (MM)
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)

IB Drivers
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA requests.

Nouveau driver
Add support for the Nouveau p2p_page callback function: This update integrates P2P DMA support into the Nouveau driver, allowing it to handle P2P page operations seamlessly.

MLX5 Driver
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize DMA P2P for private device pages. Also, when P2P DMA mapping fails due to inaccessible bridges, the system falls back to standard DMA, which uses host memory, for the affected PFNs

Previous version:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/

Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages nouveau/dmem: HMM P2P DMA for private dev pages IB/core: P2P DMA for device private pages RDMA/mlx5: Enable P2P DMA with fallback mechanism RDMA/mlx5: Enabling ATS for ODP memory

 drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
 drivers/infiniband/core/umem_odp.c     |   4 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h   |   6 +-
 drivers/infiniband/hw/mlx5/odp.c       |  24 +++++-
 include/linux/hmm.h                    |   3 +-
 include/linux/memremap.h               |   8 ++
 mm/hmm.c                               |  57 ++++++++++---
 7 files changed, 195 insertions(+), 17 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/5 mm/hmm: HMM API to enable P2P DMA for device private pages

From: Yonatan Maman <[email protected]>

hmm_range_fault() by default triggered a page fault on device private when HMM_PFN_REQ_FAULT flag was set. pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead.

Introduced an API to support P2P DMA for device private pages,includes:
- Leveraging the struct pagemap_ops for P2P Page Callbacks. This callback involves mapping the page for P2P DMA and returning the corresponding PCI_P2P page.

- Utilizing hmm_range_fault for initializing P2P DMA. The API also adds the HMM_PFN_REQ_TRY_P2P flag option for the hmm_range_fault caller to initialize P2P. If set, hmm_range_fault attempts initializing the P2P connection first, if the owner device supports P2P, using p2p_page. In case of failure or lack of support, hmm_range_fault will continue with the regular flow of migrating the page to RAM.

This change does not affect previous use-cases of hmm_range_fault, because both the caller and the page owner must explicitly request and support it to initialize P2P connection.

---
 include/linux/hmm.h      |  2 ++
 include/linux/memremap.h |  8 ++++++
 mm/hmm.c                 | 57 +++++++++++++++++++++++++++++++---------
 3 files changed, 55 insertions(+), 12 deletions(-)
include/linux/hmm.h +2 -0
index db75ffc949a7..988c98c0edcc 100644
@@ -27,6 +27,7 @@ struct mmu_interval_notifier;
  * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer
  * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation
  *                      to mark that page is already DMA mapped
+ * HMM_PFN_ALLOW_P2P - Allow returning PCI P2PDMA page
  *
  * On input:
  * 0                 - Return the current state of the page, do not fault it.
@@ -47,6 +48,7 @@ enum hmm_pfn_flags {
 	HMM_PFN_DMA_MAPPED = 1UL << (BITS_PER_LONG - 4),
 	HMM_PFN_P2PDMA     = 1UL << (BITS_PER_LONG - 5),
 	HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6),
+	HMM_PFN_ALLOW_P2P = 1UL << (BITS_PER_LONG - 7),
 
 	HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 11),
 
include/linux/memremap.h +8 -0
index 4aa151914eab..79becc37df00 100644
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);
+
 	/*
 	 * Handle the memory failure happens on a range of pfns.  Notify the
 	 * processes who are using these pfns, and try to recover the data on
mm/hmm.c +45 -12

70 lines not on this page · raw message

Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

include/linux/memremap.h struct dev_pagemap_ops { · line 98

Quoted 12 lines · Yonatan Maman

+++ b/include/linux/memremap.h

include/linux/memremap.h +7 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);

This makes no sense. If a page is addressable then it has a PFN. If a page is not addressable then it doesn't have a PFN.

#3 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Yonatan Maman

+++ b/include/linux/memremap.h

include/linux/memremap.h +0 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
Quoted 10 lines

include/linux/memremap.h struct dev_pagemap_ops { · line 98

Quoted 7 lines
include/linux/memremap.h +7 -0
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);

This makes no sense. If a page is addressable then it has a PFN. If a page is not addressable then it doesn't have a PFN.

The DEVICE_PRIVATE pages have a PFN, but it is not usable for anything.

This is effectively converting from a DEVICE_PRIVATE page to an actual DMA'able address of some kind. The DEVICE_PRIVATE is just a non-usable proxy, like a swap entry, for where the real data is sitting.

Jason

#4 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 18 lines · Jason Gunthorpe
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Yonatan Maman

+++ b/include/linux/memremap.h

include/linux/memremap.h +0 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
Quoted 10 lines

include/linux/memremap.h struct dev_pagemap_ops { · line 98

Quoted 7 lines
include/linux/memremap.h +7 -0
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);

This makes no sense. If a page is addressable then it has a PFN. If a page is not addressable then it doesn't have a PFN.

Quoted 6 lines

The DEVICE_PRIVATE pages have a PFN, but it is not usable for anything.

This is effectively converting from a DEVICE_PRIVATE page to an actual DMA'able address of some kind. The DEVICE_PRIVATE is just a non-usable proxy, like a swap entry, for where the real data is sitting.

Yes, it's on my backlog to start looking at using something other than a real PFN for this proxy. Because having it as an actual PFN has caused us all sorts of random issues as it still needs to reserve a real physical address range which may or may not be available on a given machine.

Quoted 3 lines

Jason

#5 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

In reply to Jason Gunthorpe
Quoted 18 lines · Jason Gunthorpe
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Yonatan Maman

+++ b/include/linux/memremap.h

include/linux/memremap.h +0 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
Quoted 10 lines

include/linux/memremap.h struct dev_pagemap_ops { · line 98

Quoted 7 lines
include/linux/memremap.h +7 -0
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);

This makes no sense. If a page is addressable then it has a PFN. If a page is not addressable then it doesn't have a PFN.

Quoted 2 lines

The DEVICE_PRIVATE pages have a PFN, but it is not usable for anything.

OK, then I don't understand what DEVICE PRIVATE means.

I thought it was for memory on a PCIe device that isn't even visible through a BAR and so the CPU has no way of addressing it directly. But now you say that it has a PFN, which means it has a physical address, which means it's accessible to the CPU.

So what is it?

Quoted 6 lines

This is effectively converting from a DEVICE_PRIVATE page to an actual DMA'able address of some kind. The DEVICE_PRIVATE is just a non-usable proxy, like a swap entry, for where the real data is sitting.

Jason

#6 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 19 lines · Matthew Wilcox
Quoted 18 lines · Jason Gunthorpe
Quoted 16 lines · Matthew Wilcox

include/linux/memremap.h struct dev_pagemap_ops { · line 98

Quoted 12 lines · Yonatan Maman

+++ b/include/linux/memremap.h

include/linux/memremap.h +7 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {
 	 */
 	vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);
 
+	/*
+	 * Used for private (un-addressable) device memory only. Return a
+	 * corresponding PFN for a page that can be mapped to device
+	 * (e.g using dma_map_page)
+	 */
+	int (*get_dma_pfn_for_device)(struct page *private_page,
+				      unsigned long *dma_pfn);

This makes no sense. If a page is addressable then it has a PFN. If a page is not addressable then it doesn't have a PFN.

Quoted 7 lines
Quoted 2 lines

The DEVICE_PRIVATE pages have a PFN, but it is not usable for anything.

OK, then I don't understand what DEVICE PRIVATE means.

I thought it was for memory on a PCIe device that isn't even visible through a BAR and so the CPU has no way of addressing it directly.

Correct.

Quoted 2 lines

But now you say that it has a PFN, which means it has a physical address, which means it's accessible to the CPU.

Having a PFN doesn't mean it's actually accessible to the CPU. It is a real physical address in the CPU address space, but it is a completely bogus/invalid address - if the CPU actually tries to access it will cause a machine check or whatever other exception gets generated when accessing an invalid physical address.

Obviously we're careful to avoid that. The PFN is used solely to get to/from a struct page (via pfn_to_page() or page_to_pfn()).

Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems (eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

We discussed this briefly at LSFMM, I think your suggestion for a data structure was to use a maple tree. I'm yet to look at this more deeply but I'd like to figure out where memdescs fit in this picture too.

- Alistair

Quoted 6 lines
Quoted 6 lines

This is effectively converting from a DEVICE_PRIVATE page to an actual DMA'able address of some kind. The DEVICE_PRIVATE is just a non-usable proxy, like a swap entry, for where the real data is sitting.

Jason

#7 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 5 lines · Alistair Popple
Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems

IMHO what DEVICE PRIVATE really boils down to is a way to have swap entries that point to some kind of opaque driver managed memory.

We have alot of assumptions all over about pfn/phys to page relationships so anything that has a struct page also has to come with a fake PFN today..

Quoted 5 lines

(eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

It could work, but any of the pfn conversions would have to be tracked down.. Could be troublesome.

Jason

#8 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Alistair Popple
Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems

Quoted 6 lines

IMHO what DEVICE PRIVATE really boils down to is a way to have swap entries that point to some kind of opaque driver managed memory.

We have alot of assumptions all over about pfn/phys to page relationships so anything that has a struct page also has to come with a fake PFN today..

Hmm ... maybe. To get that PFN though we have to come from either a special swap entry which we already have special cases for, or a struct page (which is a device private page) which we mostly have to handle specially anyway. I'm not sure there's too many places that can sensibly handle a fake PFN without somehow already knowing it is device-private PFN.

Quoted 8 lines
Quoted 5 lines

(eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

It could work, but any of the pfn conversions would have to be tracked down.. Could be troublesome.

I looked at this a while back and I'm reasonably optimistic that this is doable because we already have to treat these specially everywhere anyway. The proof will be writing the patches of course.

- Alistair

Quoted 1 line

Jason

#9 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 27 lines · Alistair Popple
Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Alistair Popple
Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems

Quoted 6 lines

IMHO what DEVICE PRIVATE really boils down to is a way to have swap entries that point to some kind of opaque driver managed memory.

We have alot of assumptions all over about pfn/phys to page relationships so anything that has a struct page also has to come with a fake PFN today..

Hmm ... maybe. To get that PFN though we have to come from either a special swap entry which we already have special cases for, or a struct page (which is a device private page) which we mostly have to handle specially anyway. I'm not sure there's too many places that can sensibly handle a fake PFN without somehow already knowing it is device-private PFN.

Quoted 6 lines
Quoted 5 lines

(eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

Quoted 5 lines
Quoted 2 lines

It could work, but any of the pfn conversions would have to be tracked down.. Could be troublesome.

I looked at this a while back and I'm reasonably optimistic that this is doable because we already have to treat these specially everywhere anyway.

How would that look like?

E.g., we have code like

if (is_device_private_entry(entry)) {
	page = pfn_swap_entry_to_page(entry);
	folio = page_folio(page);

	...
	folio_get(folio);
	...
}

We could easily stop allowing pfn_swap_entry_to_page(), turning these into non-pfn swap entries.

Would it then be something like

if (is_device_private_entry(entry)) {
	page = device_private_entry_to_page(entry);
	
	...
}

Whereby device_private_entry_to_page() obtains the "struct page" not via the PFN but some other magical (index) value?

Signature · 3 lines
Cheers,

David / dhildenb
#10 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 51 lines · David Hildenbrand
Quoted 27 lines · Alistair Popple
Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Alistair Popple
Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems

Quoted 6 lines

IMHO what DEVICE PRIVATE really boils down to is a way to have swap entries that point to some kind of opaque driver managed memory.

We have alot of assumptions all over about pfn/phys to page relationships so anything that has a struct page also has to come with a fake PFN today..

Hmm ... maybe. To get that PFN though we have to come from either a special swap entry which we already have special cases for, or a struct page (which is a device private page) which we mostly have to handle specially anyway. I'm not sure there's too many places that can sensibly handle a fake PFN without somehow already knowing it is device-private PFN.

Quoted 6 lines
Quoted 5 lines

(eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

Quoted 5 lines
Quoted 2 lines

It could work, but any of the pfn conversions would have to be tracked down.. Could be troublesome.

I looked at this a while back and I'm reasonably optimistic that this is doable because we already have to treat these specially everywhere anyway.

How would that look like?

E.g., we have code like

if (is_device_private_entry(entry)) {
	page = pfn_swap_entry_to_page(entry);
	folio = page_folio(page);

	...
	folio_get(folio);
	...
}

We could easily stop allowing pfn_swap_entry_to_page(), turning these into non-pfn swap entries.

Would it then be something like

Quoted 8 lines
if (is_device_private_entry(entry)) {
	page = device_private_entry_to_page(entry);
	
	...
}

Whereby device_private_entry_to_page() obtains the "struct page" not via the PFN but some other magical (index) value?

Exactly. The observation being that when you convert a PTE from a swap entry to a page we already know it is a device private entry, so can go look up the struct page with special magic (eg. an index into some other array or data structure).

And if you have a struct page you already know it's a device private page so if you need to create the swap entry you can look up the magic index using some alternate function.

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

- Alistair

Quoted 6 lines
Signature · 4 lines
-- 
Cheers,

David / dhildenb
#11 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 66 lines · Alistair Popple
Quoted 51 lines · David Hildenbrand
Quoted 27 lines · Alistair Popple
Quoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Alistair Popple
Quoted 1 line

So what is it?

IMHO a hack, because obviously we shouldn't require real physical addresses for something the CPU can't actually address anyway and this causes real problems

Quoted 6 lines

IMHO what DEVICE PRIVATE really boils down to is a way to have swap entries that point to some kind of opaque driver managed memory.

We have alot of assumptions all over about pfn/phys to page relationships so anything that has a struct page also has to come with a fake PFN today..

Hmm ... maybe. To get that PFN though we have to come from either a special swap entry which we already have special cases for, or a struct page (which is a device private page) which we mostly have to handle specially anyway. I'm not sure there's too many places that can sensibly handle a fake PFN without somehow already knowing it is device-private PFN.

Quoted 6 lines
Quoted 5 lines

(eg. it doesn't actually work on anything other than x86_64). There's no reason the "PFN" we store in device-private entries couldn't instead just be an index into some data structure holding pointers to the struct pages. So instead of using pfn_to_page()/page_to_pfn() we would use device_private_index_to_page() and page_to_device_private_index().

Quoted 5 lines
Quoted 2 lines

It could work, but any of the pfn conversions would have to be tracked down.. Could be troublesome.

I looked at this a while back and I'm reasonably optimistic that this is doable because we already have to treat these specially everywhere anyway.

How would that look like?

E.g., we have code like

if (is_device_private_entry(entry)) {
	page = pfn_swap_entry_to_page(entry);
	folio = page_folio(page);

	...
	folio_get(folio);
	...
}

We could easily stop allowing pfn_swap_entry_to_page(), turning these into non-pfn swap entries.

Would it then be something like

Quoted 8 lines
if (is_device_private_entry(entry)) {
	page = device_private_entry_to_page(entry);
	
	...
}

Whereby device_private_entry_to_page() obtains the "struct page" not via the PFN but some other magical (index) value?

Exactly. The observation being that when you convert a PTE from a swap entry to a page we already know it is a device private entry, so can go look up the struct page with special magic (eg. an index into some other array or data structure).

Quoted 7 lines

And if you have a struct page you already know it's a device private page so if you need to create the swap entry you can look up the magic index using some alternate function.

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

I guess memory compaction and friends don't apply to ZONE_DEVICE, and even memory_failure() handling goes a separate path.

Signature · 3 lines
Cheers,

David / dhildenb
#12 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

In reply to Alistair Popple
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

This isn't set in stone and could be changed..

But broadly, you'd want to entirely eliminate the ability to go from pfn to device private or from device private to pfn.

Instead you'd want to work on some (space #, space index) tuple, maybe encoded in a pfn_t, but absolutely and typesafely distinct. Each driver gets its own 0 based space for device private information, the space is effectively the pgmap.

And if you do this, maybe we don't need struct page (I mean the type!) backing device memory at all.... Which would be a very worthwhile project.

Do we ever even use anything in the device private struct page? Do we refcount it?

Jason

#13 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 19 lines · Jason Gunthorpe
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

This isn't set in stone and could be changed..

But broadly, you'd want to entirely eliminate the ability to go from pfn to device private or from device private to pfn.

Instead you'd want to work on some (space #, space index) tuple, maybe encoded in a pfn_t, but absolutely and typesafely distinct. Each driver gets its own 0 based space for device private information, the space is effectively the pgmap.

Quoted 6 lines

And if you do this, maybe we don't need struct page (I mean the type!) backing device memory at all.... Which would be a very worthwhile project.

Do we ever even use anything in the device private struct page? Do we refcount it?

ref-counted and map-counted ...

Signature · 3 lines
Cheers,

David / dhildenb
#14 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 20 lines · David Hildenbrand
Quoted 19 lines · Jason Gunthorpe
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

This isn't set in stone and could be changed..

But broadly, you'd want to entirely eliminate the ability to go from pfn to device private or from device private to pfn.

Instead you'd want to work on some (space #, space index) tuple, maybe encoded in a pfn_t, but absolutely and typesafely distinct. Each driver gets its own 0 based space for device private information, the space is effectively the pgmap.

Quoted 8 lines
Quoted 6 lines

And if you do this, maybe we don't need struct page (I mean the type!) backing device memory at all.... Which would be a very worthwhile project.

Do we ever even use anything in the device private struct page? Do we refcount it?

ref-counted and map-counted ...

Hm, so it would turn into another struct page split up where we get ourselves a struct device_private and change all the places touching its refcount and mapcount to use the new type.

If we could use some index scheme we could then divorce from struct page and strink the struct size sooner.

Jason

#15 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 10 lines · Jason Gunthorpe
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Jason Gunthorpe
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

What callers need to do pfn to page when finding a device private pfn via hmm_range_fault()? GPU drivers don't, they tend just to use the pfn as an offset from the start of the pgmap to find whatever data structure they are using to track device memory allocations.

The migrate_vma_*() calls do, but they could easily be changed to whatever index scheme we use so long as we can encode that this is a device entry in the MIGRATE_PFN flags.

So other than adding a HMM_PFN flag to say this is really a device index I don't see too many issues here.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

This isn't set in stone and could be changed..

But broadly, you'd want to entirely eliminate the ability to go from pfn to device private or from device private to pfn.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Instead you'd want to work on some (space #, space index) tuple, maybe encoded in a pfn_t, but absolutely and typesafely distinct. Each driver gets its own 0 based space for device private information, the space is effectively the pgmap.

And if you do this, maybe we don't need struct page (I mean the type!) backing device memory at all.... Which would be a very worthwhile project.

Exactly! Although we still need enough of a struct page or something else to still be able to map them in normal anonymous VMAs. Short term the motivation for this project is that the current scheme of "stealing" pfns for the device doesn't actually work in a lot of cases.

Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Do we ever even use anything in the device private struct page? Do we refcount it?

ref-counted and map-counted ...

Hm, so it would turn into another struct page split up where we get ourselves a struct device_private and change all the places touching its refcount and mapcount to use the new type.

If we could use some index scheme we could then divorce from struct page and strink the struct size sooner.

Right, that is roughly along the lines of what I was thinking.

Quoted 1 line

Jason

#16 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 9 lines · Alistair Popple
Quoted 8 lines · Jason Gunthorpe
Quoted 7 lines · David Hildenbrand
Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

What callers need to do pfn to page when finding a device private pfn via hmm_range_fault()? GPU drivers don't, they tend just to use the pfn as an offset from the start of the pgmap to find whatever data structure they are using to track device memory allocations.

All drivers today must. You have no idea if the PFN returned is a private or CPU page. The only way to know is to check the struct page type, by looking inside the struct page.

Quoted 2 lines

So other than adding a HMM_PFN flag to say this is really a device index I don't see too many issues here.

Christoph suggested exactly this, and it would solve the issue. Seems quite easy too. Let's do it.

Jason

#17 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

In reply to Jason Gunthorpe
Quoted 25 lines · Jason Gunthorpe
Quoted 24 lines · David Hildenbrand
Quoted 23 lines · Jason Gunthorpe
Quoted 3 lines · Alistair Popple

The only issue would be if there were generic code paths that somehow have a raw pfn obtained from neither a page-table walk or struct page. My assumption (yet to be proven/tested) is that these paths don't exist.

hmm does it, it encodes the device private into a pfn and expects the caller to do pfn to page.

This isn't set in stone and could be changed..

But broadly, you'd want to entirely eliminate the ability to go from pfn to device private or from device private to pfn.

Instead you'd want to work on some (space #, space index) tuple, maybe encoded in a pfn_t, but absolutely and typesafely distinct. Each driver gets its own 0 based space for device private information, the space is effectively the pgmap.

And if you do this, maybe we don't need struct page (I mean the type!) backing device memory at all.... Which would be a very worthwhile project.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Do we ever even use anything in the device private struct page? Do we refcount it?

ref-counted and map-counted ...

Hm, so it would turn into another struct page split up where we get ourselves a struct device_private and change all the places touching its refcount and mapcount to use the new type.

We're already working with folios in all cases where we modify either refcount or mapcount IIUC.

The rmap handling (try to migrate, soon folio splitting) currently depends on the mapcount.

Not sure how that will all look like without a ... struct folio / struct page.

Signature · 3 lines
Cheers,

David / dhildenb
#18 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

In reply to Yonatan Maman
Quoted 8 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

hmm_range_fault() by default triggered a page fault on device private when HMM_PFN_REQ_FAULT flag was set. pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead.

You don't enable DMA for device private pages. You allow discovering a DMAable alias for device private pages.

Also absolutely nothing GPU specific here.

mm/hmm.c static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, · line 253

Quoted 7 lines
mm/hmm.c +7 -0
+	/*
+	 * Don't fault in device private pages owned by the caller,
+	 * just report the PFN.
+	 */
+	if (pgmap->owner == range->dev_private_owner) {
+		*hmm_pfn = swp_offset_pfn(entry);
+		goto found;

This is dangerous because it mixes actual DMAable alias PFNs with the device private fake PFNs. Maybe your hardware / driver can handle it, but just leaking this out is not a good idea.

mm/hmm.c static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, · line 302

Quoted 1 line
mm/hmm.c +1 -0
+		    hmm_handle_device_private(range, pfn_req_flags, entry, hmm_pfn))

Overly long line here.

#19 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

hmm_range_fault() by default triggered a page fault on device private when HMM_PFN_REQ_FAULT flag was set. pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead.

Quoted 5 lines

You don't enable DMA for device private pages. You allow discovering a DMAable alias for device private pages.

Also absolutely nothing GPU specific here.

Ok, understood, I will change it (v3).

Quoted 12 lines

mm/hmm.c static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, · line 253

Quoted 7 lines
mm/hmm.c +7 -0
+	/*
+	 * Don't fault in device private pages owned by the caller,
+	 * just report the PFN.
+	 */
+	if (pgmap->owner == range->dev_private_owner) {
+		*hmm_pfn = swp_offset_pfn(entry);
+		goto found;

This is dangerous because it mixes actual DMAable alias PFNs with the device private fake PFNs. Maybe your hardware / driver can handle it, but just leaking this out is not a good idea.

In the current implementation, regular pci_p2p pages are returned as-is from hmm_range_fault() - for virtual address backed by pci_p2p page, it will return the corresponding PFN. That said, we can mark these via the hmm_pfn output flags so the caller can handle them appropriately.

Quoted 4 lines

mm/hmm.c static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, · line 302

Quoted 1 line
mm/hmm.c +1 -0
+		    hmm_handle_device_private(range, pfn_req_flags, entry, hmm_pfn))

Overly long line here.

will be fixed (v3)

maintainer
#20 lore raw

Re: [PATCH v2 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages

In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig

mm/hmm.c static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, · line 253

Quoted 7 lines
mm/hmm.c +7 -0
+	/*
+	 * Don't fault in device private pages owned by the caller,
+	 * just report the PFN.
+	 */
+	if (pgmap->owner == range->dev_private_owner) {
+		*hmm_pfn = swp_offset_pfn(entry);
+		goto found;

This is dangerous because it mixes actual DMAable alias PFNs with the device private fake PFNs. Maybe your hardware / driver can handle it, but just leaking this out is not a good idea.

For better or worse that is how the hmm API works today.

Recall the result is an array of unsigned long with a pfn and flags:

enum hmm_pfn_flags {
	/* Output fields and flags */
	HMM_PFN_VALID = 1UL << (BITS_PER_LONG - 1),
	HMM_PFN_WRITE = 1UL << (BITS_PER_LONG - 2),
	HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3),

The only promise is that every pfn has a struct page behind it.

If the caller specifies dev_private_owner then it must also look into the struct page of every returned pfn to see if it is device private or not.

hmm_dma_map_pfn() already unconditionally calls pci_p2pdma_state() which checks for P2P struct pages.

It does sound like a good improvement to return the type of the pfn (normal, p2p, private) in the flags bits as well to optimize away these extra struct page lookups.

But this is a different project..

Jason

#21 lore raw

PATCH v2 2/5 nouveau/dmem: HMM P2P DMA for private dev pages

In reply to Yonatan Maman

From: Yonatan Maman <[email protected]>

Enabling Peer-to-Peer DMA (P2P DMA) access in GPU-centric applications is crucial for minimizing data transfer overhead (e.g., for RDMA use- case).

This change aims to enable that capability for Nouveau over HMM device private pages. P2P DMA for private device pages allows the GPU to directly exchange data with other devices (e.g., NICs) without needing to traverse system RAM.

To fully support Peer-to-Peer for device private pages, the following changes are made:

- Introduce struct nouveau_dmem_hmm_p2p within struct nouveau_dmem to manage BAR1 PCI P2P memory. p2p_start_addr holds the virtual address allocated with pci_alloc_p2pmem(), and p2p_size represents the allocated size of the PCI P2P memory.

- nouveau_dmem_init - Ensure BAR1 accessibility and assign struct pages (PCI_P2P_PAGE) for all BAR1 pages. Introduce nouveau_alloc_bar1_pci_p2p_mem in nouveau_dmem to expose BAR1 for use as P2P memory via pci_p2pdma_add_resource and implement static allocation and assignment of struct pages using pci_alloc_p2pmem. This function will be called from nouveau_dmem_init, and failure triggers a warning message instead of driver failure.

- nouveau_dmem_fini - Ensure BAR1 PCI P2P memory is properly destroyed during driver cleanup. Introduce nouveau_destroy_bar1_pci_p2p_mem to handle freeing of PCI P2P memory associated with Nouveau BAR1. Modify nouveau_dmem_fini to call nouveau_destroy_bar1_pci_p2p_mem.

- Implement Nouveau p2p_page callback function - Implement BAR1 mapping for the chunk using io_mem_reserve if no mapping exists. Retrieve the pre-allocated P2P virtual address and size from hmm_p2p. Calculate the page offset within BAR1 and return the corresponding P2P page.

---
 drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
 1 file changed, 110 insertions(+)
drivers/gpu/drm/nouveau/nouveau_dmem.c +110 -0
index ca4932a150e3..acac1449d8cb 100644
@@ -40,6 +40,9 @@
 #include <linux/hmm.h>
 #include <linux/memremap.h>
 #include <linux/migrate.h>
+#include <linux/pci-p2pdma.h>
+#include <nvkm/core/pci.h>
+
 
 /*
  * FIXME: this is ugly right now we are using TTM to allocate vram and we pin
@@ -77,9 +80,15 @@ struct nouveau_dmem_migrate {
 	struct nouveau_channel *chan;
 };
 
+struct nouveau_dmem_hmm_p2p {
+	size_t p2p_size;
+	void *p2p_start_addr;
+};
+
 struct nouveau_dmem {
 	struct nouveau_drm *drm;
 	struct nouveau_dmem_migrate migrate;
+	struct nouveau_dmem_hmm_p2p hmm_p2p;
 	struct list_head chunks;
 	struct mutex mutex;
 	struct page *free_pages;
@@ -159,6 +168,60 @@ static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage,
 	return 0;
 }
 
+static int nouveau_dmem_bar1_mapping(struct nouveau_bo *nvbo,
+				     unsigned long long *bus_addr)
+{
+	int ret;
+	struct ttm_resource *mem = nvbo->bo.resource;
+
+	if (mem->bus.offset) {
+		*bus_addr = mem->bus.offset;
+		return 0;
+	}
+
+	if (PFN_UP(nvbo->bo.base.size) > PFN_UP(nvbo->bo.resource->size))
+		return -EINVAL;
+
+	ret = ttm_bo_reserve(&nvbo->bo, false, false, NULL);
+	if (ret)
+		return ret;
+
+	ret = nvbo->bo.bdev->funcs->io_mem_reserve(nvbo->bo.bdev, mem);
+	*bus_addr = mem->bus.offset;
+
+	ttm_bo_unreserve(&nvbo->bo);
+	return ret;
+}
+
+static int nouveau_dmem_get_dma_pfn(struct page *private_page,
+				    unsigned long *dma_pfn)
+{
+	int ret;
+	unsigned long long offset_in_chunk;
+	unsigned long long chunk_bus_addr;
+	unsigned long long bar1_base_addr;
+	struct nouveau_drm *drm = page_to_drm(private_page);
+	struct nouveau_bo *nvbo = nouveau_page_to_chunk(private_page)->bo;
+	struct nvkm_device *nv_device = nvxx_device(drm);
+	size_t p2p_size = drm->dmem->hmm_p2p.p2p_size;
+
+	bar1_base_addr = nv_device->func->resource_addr(nv_device, 1);
+	offset_in_chunk =
+		(page_to_pfn(private_page) << PAGE_SHIFT) -
+		nouveau_page_to_chunk(private_page)->pagemap.range.start;
+
+	ret = nouveau_dmem_bar1_mapping(nvbo, &chunk_bus_addr);
+	if (ret)
+		return ret;
+
+	*dma_pfn = chunk_bus_addr + offset_in_chunk;
+	if (!p2p_size || *dma_pfn > bar1_base_addr + p2p_size ||
+	    *dma_pfn < bar1_base_addr)
+		return -ENOMEM;
+
+	return 0;
+}
+
 static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
 {
 	struct nouveau_drm *drm = page_to_drm(vmf->page);
@@ -222,6 +285,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)
 static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = {
 	.page_free		= nouveau_dmem_page_free,
 	.migrate_to_ram		= nouveau_dmem_migrate_to_ram,
+	.get_dma_pfn_for_device = nouveau_dmem_get_dma_pfn,
 };
 
 static int
@@ -407,14 +471,31 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)
 	kvfree(dma_addrs);
 }
 
+static void nouveau_destroy_bar1_pci_p2p_mem(struct nouveau_drm *drm,
+					     struct pci_dev *pdev,
+					     void *p2p_start_addr,
+					     size_t p2p_size)
+{
+	if (p2p_size)
+		pci_free_p2pmem(pdev, p2p_start_addr, p2p_size);
+
+	NV_INFO(drm, "PCI P2P memory freed(%p)\n", p2p_start_addr);
+}
+
 void
 nouveau_dmem_fini(struct nouveau_drm *drm)
 {
 	struct nouveau_dmem_chunk *chunk, *tmp;
+	struct nvkm_device *nv_device = nvxx_device(drm);
 
 	if (drm->dmem == NULL)
 		return;
 
+	nouveau_destroy_bar1_pci_p2p_mem(drm,
+					 nv_device->func->pci(nv_device)->pdev,
+					 drm->dmem->hmm_p2p.p2p_start_addr,
+					 drm->dmem->hmm_p2p.p2p_size);
+
 	mutex_lock(&drm->dmem->mutex);
 
 	list_for_each_entry_safe(chunk, tmp, &drm->dmem->chunks, list) {
@@ -579,10 +660,28 @@ nouveau_dmem_migrate_init(struct nouveau_drm *drm)
 	return -ENODEV;
 }
 
+static int nouveau_alloc_bar1_pci_p2p_mem(struct nouveau_drm *drm,
+					  struct pci_dev *pdev, size_t size,
+					  void **pp2p_start_addr)
+{
+	int ret;
+
+	ret = pci_p2pdma_add_resource(pdev, 1, size, 0);
+	if (ret)
+		return ret;
+
+	*pp2p_start_addr = pci_alloc_p2pmem(pdev, size);
+
+	NV_INFO(drm, "PCI P2P memory allocated(%p)\n", *pp2p_start_addr);
+	return 0;
+}
+
 void
 nouveau_dmem_init(struct nouveau_drm *drm)
 {
 	int ret;
+	struct nvkm_device *nv_device = nvxx_device(drm);
+	size_t bar1_size;
 
 	/* This only make sense on PASCAL or newer */
 	if (drm->client.device.info.family < NV_DEVICE_INFO_V0_PASCAL)
@@ -603,6 +702,17 @@ nouveau_dmem_init(struct nouveau_drm *drm)
 		kfree(drm->dmem);
 		drm->dmem = NULL;
 	}
+
+	/* Expose BAR1 for HMM P2P Memory */
+	bar1_size = nv_device->func->resource_size(nv_device, 1);
+	ret = nouveau_alloc_bar1_pci_p2p_mem(drm,
+					     nv_device->func->pci(nv_device)->pdev,
+					     bar1_size,
+					     &drm->dmem->hmm_p2p.p2p_start_addr);
+	drm->dmem->hmm_p2p.p2p_size = (ret) ? 0 : bar1_size;
+	if (ret)
+		NV_WARN(drm,
+			"PCI P2P memory allocation failed, HMM P2P won't be supported\n");
 }
 
 static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
Signature · 1 line
2.34.1
#22 lore raw

Re: [PATCH v2 2/5] nouveau/dmem: HMM P2P DMA for private dev pages

drivers/gpu/drm/nouveau/nouveau_dmem.c static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) · line 288

Quoted 1 line · Yonatan Maman
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
+	.get_dma_pfn_for_device = nouveau_dmem_get_dma_pfn,

Please don't shorten the method name prefix in the implementation symbol name, as that makes reading / refactoring the code a pain.

This might also be a hint that your method name is too long.

#23 lore raw

Re: [PATCH v2 2/5] nouveau/dmem: HMM P2P DMA for private dev pages

Quoted 8 lines · Christoph Hellwig

drivers/gpu/drm/nouveau/nouveau_dmem.c static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) · line 288

Quoted 1 line · Yonatan Maman
drivers/gpu/drm/nouveau/nouveau_dmem.c +1 -0
+	.get_dma_pfn_for_device = nouveau_dmem_get_dma_pfn,

Please don't shorten the method name prefix in the implementation symbol name, as that makes reading / refactoring the code a pain.

This might also be a hint that your method name is too long.

got it, will be fixed in V3, thanks.

#24 lore raw

PATCH v2 3/5 IB/core: P2P DMA for device private pages

In reply to Yonatan Maman

From: Yonatan Maman <[email protected]>

Add Peer-to-Peer (P2P) DMA request for hmm_range_fault calling, utilizing capabilities introduced in mm/hmm. By setting range.default_flags to HMM_PFN_REQ_FAULT | HMM_PFN_REQ_TRY_P2P, HMM attempts to initiate P2P DMA connections for device private pages (instead of page fault handling).

This enhancement utilizes P2P DMA to reduce performance overhead during data migration between devices (e.g., GPU) and system memory, providing performance benefits for GPU-centric applications that utilize RDMA and device private pages.

---
 drivers/infiniband/core/umem_odp.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/infiniband/core/umem_odp.c +4 -0
index b1c44ec1a3f3..7ba80ed4977c 100644
@@ -362,6 +362,10 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
 			range.default_flags |= HMM_PFN_REQ_WRITE;
 	}
 
+	if (access_mask & HMM_PFN_ALLOW_P2P)
+		range.default_flags |= HMM_PFN_ALLOW_P2P;
+
+	range.pfn_flags_mask = HMM_PFN_ALLOW_P2P;
 	range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]);
 	timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
 
Signature · 1 line
2.34.1
#25 lore raw

PATCH v2 4/5 RDMA/mlx5: Enable P2P DMA with fallback mechanism

In reply to Yonatan Maman

From: Yonatan Maman <[email protected]>

Add support for P2P for MLX5 NIC devices with automatic fallback to standard DMA when P2P mapping fails.

The change introduces P2P DMA requests by default using the HMM_PFN_ALLOW_P2P flag. If P2P mapping fails with -EFAULT error, the operation is retried without the P2P flag, ensuring a fallback to standard DMA flow (using host memory).

---
 drivers/infiniband/hw/mlx5/odp.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/infiniband/hw/mlx5/odp.c +16 -0
index f6abd64f07f7..6a0171117f48 100644
@@ -715,6 +715,10 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp,
 	if (odp->umem.writable && !downgrade)
 		access_mask |= HMM_PFN_WRITE;
 
+	/*
+	 * try fault with HMM_PFN_ALLOW_P2P flag
+	 */
+	access_mask |= HMM_PFN_ALLOW_P2P;
 	np = ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, fault);
 	if (np < 0)
 		return np;
@@ -724,6 +728,18 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp,
 	 * ib_umem_odp_map_dma_and_lock already checks this.
 	 */
 	ret = mlx5r_umr_update_xlt(mr, start_idx, np, page_shift, xlt_flags);
+	if (ret == -EFAULT) {
+		/*
+		 * Indicate P2P Mapping Error, retry with no HMM_PFN_ALLOW_P2P
+		 */
+		mutex_unlock(&odp->umem_mutex);
+		access_mask &= ~(HMM_PFN_ALLOW_P2P);
+		np = ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, fault);
+		if (np < 0)
+			return np;
+		ret = mlx5r_umr_update_xlt(mr, start_idx, np, page_shift, xlt_flags);
+	}
+
 	mutex_unlock(&odp->umem_mutex);
 
 	if (ret < 0) {
Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH v2 4/5] RDMA/mlx5: Enable P2P DMA with fallback mechanism

Quoted 4 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

Add support for P2P for MLX5 NIC devices with automatic fallback to standard DMA when P2P mapping fails.

That's now how the P2P API works. You need to check the P2P availability higher up.

#27 lore raw

Re: [PATCH v2 4/5] RDMA/mlx5: Enable P2P DMA with fallback mechanism

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

Add support for P2P for MLX5 NIC devices with automatic fallback to standard DMA when P2P mapping fails.

That's now how the P2P API works. You need to check the P2P availability higher up.

How do you mean?

This looks OKish to me, for ODP and HMM it has to check the P2P availability on a page by page basis because every single page can be a different origin device.

There isn't really a higher up here...

Jason

#28 lore raw

Re: [PATCH v2 4/5] RDMA/mlx5: Enable P2P DMA with fallback mechanism

Quoted 10 lines · Jason Gunthorpe
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

Add support for P2P for MLX5 NIC devices with automatic fallback to standard DMA when P2P mapping fails.

That's now how the P2P API works. You need to check the P2P availability higher up.

Quoted 7 lines

How do you mean?

This looks OKish to me, for ODP and HMM it has to check the P2P availability on a page by page basis because every single page can be a different origin device.

There isn't really a higher up here...

The DMA API expects the caller to already check for connectability, why can't HMM do that like everyone else?

#29 lore raw

Re: [PATCH v2 4/5] RDMA/mlx5: Enable P2P DMA with fallback mechanism

Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Jason Gunthorpe
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

Add support for P2P for MLX5 NIC devices with automatic fallback to standard DMA when P2P mapping fails.

That's now how the P2P API works. You need to check the P2P availability higher up.

How do you mean?

Quoted 8 lines
Quoted 5 lines

This looks OKish to me, for ODP and HMM it has to check the P2P availability on a page by page basis because every single page can be a different origin device.

There isn't really a higher up here...

The DMA API expects the caller to already check for connectability, why can't HMM do that like everyone else?

It does, this doesn't change anything about how the DMA API works.

All this series does, and you stated it perfectly, is to allow HMM to return the single PCI P2P alias of the device private page.

HMM already blindly returns normal P2P pages in a VMA, it should also blindly return the P2P alias pages too.

Once the P2P is returned the xisting code in hmm_dma_map_pfn() calls pci_p2pdma_state() to find out if it is compatible or not.

Lifting the pci_p2pdma_state() from hmm_dma_map_pfn() and into hmm_range_fault() is perhaps possible and may be reasonable, but not really related to this series.

Jason

#30 lore raw

PATCH v2 5/5 RDMA/mlx5: Enabling ATS for ODP memory

In reply to Yonatan Maman

From: Yonatan Maman <[email protected]>

ATS (Address Translation Services) mainly utilized to optimize PCI Peer-to-Peer transfers and prevent bus failures. This change employed ATS usage for ODP memory, to optimize DMA P2P for ODP memory. (e.g DMA P2P for private device pages - ODP memory).

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -3
index fde859d207ae..a7b7a565b7e8 100644
@@ -1734,9 +1734,9 @@ static inline bool rt_supported(int ts_cap)
 static inline bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev,
 				       struct ib_umem *umem, int access_flags)
 {
-	if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf)
-		return false;
-	return access_flags & IB_ACCESS_RELAXED_ORDERING;
+	if (MLX5_CAP_GEN(dev->mdev, ats) && (umem->is_dmabuf || umem->is_odp))
+		return access_flags & IB_ACCESS_RELAXED_ORDERING;
+	return false;
 }
 
 int set_roce_addr(struct mlx5_ib_dev *dev, u32 port_num,
Signature · 1 line
2.34.1
#31 lore raw
In reply to Yonatan Maman
Quoted 29 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement reduces data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.

To fully support Peer-to-Peer for device private pages, the following changes are proposed:

Memory Management (MM)
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)

IB Drivers
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA requests.

Nouveau driver
Add support for the Nouveau p2p_page callback function: This update integrates P2P DMA support into the Nouveau driver, allowing it to handle P2P page operations seamlessly.

Quoted 5 lines

MLX5 Driver
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize DMA P2P for private device pages. Also, when P2P DMA mapping fails due to inaccessible bridges, the system falls back to standard DMA, which uses host memory, for the affected PFNs

I'm probably missing something very important, but why can't you always perform p2p if two devices support it? It is strange that IB and not HMM has a fallback mode.

Thanks

Quoted 24 lines

Previous version:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/

Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages nouveau/dmem: HMM P2P DMA for private dev pages IB/core: P2P DMA for device private pages RDMA/mlx5: Enable P2P DMA with fallback mechanism RDMA/mlx5: Enabling ATS for ODP memory

 drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
 drivers/infiniband/core/umem_odp.c     |   4 +
 drivers/infiniband/hw/mlx5/mlx5_ib.h   |   6 +-
 drivers/infiniband/hw/mlx5/odp.c       |  24 +++++-
 include/linux/hmm.h                    |   3 +-
 include/linux/memremap.h               |   8 ++
 mm/hmm.c                               |  57 ++++++++++---
 7 files changed, 195 insertions(+), 17 deletions(-)
Signature · 1 line
2.34.1
#32 lore raw
Quoted 39 lines · Leon Romanovsky

External email: Use caution opening links or attachments

Quoted 29 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement reduces data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.

To fully support Peer-to-Peer for device private pages, the following changes are proposed:

Memory Management (MM)
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)

IB Drivers
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA requests.

Nouveau driver
Add support for the Nouveau p2p_page callback function: This update integrates P2P DMA support into the Nouveau driver, allowing it to handle P2P page operations seamlessly.

Quoted 5 lines

MLX5 Driver
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize DMA P2P for private device pages. Also, when P2P DMA mapping fails due to inaccessible bridges, the system falls back to standard DMA, which uses host memory, for the affected PFNs

Quoted 6 lines

I'm probably missing something very important, but why can't you always perform p2p if two devices support it? It is strange that IB and not HMM has a fallback mode.

Thanks

P2P mapping can fail even when both devices support it, due to PCIe bridge limitations or IOMMU restrictions that block direct P2P access. The fallback is in IB rather than HMM because HMM only manages memory pages - it doesn't do DMA mapping. The IB driver does the actual DMA operations, so it knows when P2P mapping fails and can fall back to copying through system memory. In fact, hmm_range_fault doesn't have information about the destination device that will perform the DMA mapping.

Quoted 24 lines
Quoted 24 lines

Previous version:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/

Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages nouveau/dmem: HMM P2P DMA for private dev pages IB/core: P2P DMA for device private pages RDMA/mlx5: Enable P2P DMA with fallback mechanism RDMA/mlx5: Enabling ATS for ODP memory

  drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
  drivers/infiniband/core/umem_odp.c     |   4 +
  drivers/infiniband/hw/mlx5/mlx5_ib.h   |   6 +-
  drivers/infiniband/hw/mlx5/odp.c       |  24 +++++-
  include/linux/hmm.h                    |   3 +-
  include/linux/memremap.h               |   8 ++
  mm/hmm.c                               |  57 ++++++++++---
  7 files changed, 195 insertions(+), 17 deletions(-)

--
2.34.1

#33 lore raw
Quoted 46 lines · Yonatan Maman
Quoted 43 lines · Leon Romanovsky

External email: Use caution opening links or attachments

Quoted 29 lines · Yonatan Maman

From: Yonatan Maman <[email protected]>

This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement reduces data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.

To fully support Peer-to-Peer for device private pages, the following changes are proposed:

Memory Management (MM)
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)

IB Drivers
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA requests.

Nouveau driver
Add support for the Nouveau p2p_page callback function: This update integrates P2P DMA support into the Nouveau driver, allowing it to handle P2P page operations seamlessly.

Quoted 5 lines

MLX5 Driver
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize DMA P2P for private device pages. Also, when P2P DMA mapping fails due to inaccessible bridges, the system falls back to standard DMA, which uses host memory, for the affected PFNs

I'm probably missing something very important, but why can't you always perform p2p if two devices support it? It is strange that IB and not HMM has a fallback mode.

Quoted 5 lines
Quoted 2 lines

Thanks

P2P mapping can fail even when both devices support it, due to PCIe bridge limitations or IOMMU restrictions that block direct P2P access.

Yes, it is how p2p works. The decision "if p2p is supported or not" is calculated by pci_p2pdma_map_type(). That function needs to get which two devices will be connected.

In proposed HMM_PFN_ALLOW_P2P flag, you don't provide device information and for the system with more than 2 p2p devices, you will get completely random result.

Quoted 3 lines

The fallback is in IB rather than HMM because HMM only manages memory pages - it doesn't do DMA mapping. The IB driver does the actual DMA operations, so it knows when P2P mapping fails and can fall back to copying through system memory.

The thing is that in proposed patch, IB doesn't check that p2p is established with right device.
https://lore.kernel.org/all/[email protected]/

Quoted 2 lines

In fact, hmm_range_fault doesn't have information about the destination device that will perform the DMA mapping.

So probably you need to teach HMM to perform page_faults on specific device.

Thansk

Quoted 27 lines
Quoted 24 lines
Quoted 24 lines

Previous version:
https://lore.kernel.org/linux-mm/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/

Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages nouveau/dmem: HMM P2P DMA for private dev pages IB/core: P2P DMA for device private pages RDMA/mlx5: Enable P2P DMA with fallback mechanism RDMA/mlx5: Enabling ATS for ODP memory

  drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++
  drivers/infiniband/core/umem_odp.c     |   4 +
  drivers/infiniband/hw/mlx5/mlx5_ib.h   |   6 +-
  drivers/infiniband/hw/mlx5/odp.c       |  24 +++++-
  include/linux/hmm.h                    |   3 +-
  include/linux/memremap.h               |   8 ++
  mm/hmm.c                               |  57 ++++++++++---
  7 files changed, 195 insertions(+), 17 deletions(-)

--
2.34.1

#34 lore raw
Quoted 4 lines · Leon Romanovsky
Quoted 2 lines

In fact, hmm_range_fault doesn't have information about the destination device that will perform the DMA mapping.

So probably you need to teach HMM to perform page_faults on specific device.

That isn't how the HMM side is supposed to work, this API is just giving the one and only P2P page that is backing the device private.

The providing driver shouldn't be doing any p2pdma operations to check feasibility.

Otherwise we are doing p2p operations twice on every page, doesn't make sense.

We've consistently been saying the P2P is done during the DMA mapping side only, I think we should stick with that. Failing P2P is an exception case, and the fix is to trigger page migration which the general hmm code knows how to do. So calling hmm range fault again makes sense to me. I wouldn't want drivers open coding the migration logic in the new callback.

Jason

#35 lore raw
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Leon Romanovsky
Quoted 2 lines

In fact, hmm_range_fault doesn't have information about the destination device that will perform the DMA mapping.

So probably you need to teach HMM to perform page_faults on specific device.

That isn't how the HMM side is supposed to work, this API is just giving the one and only P2P page that is backing the device private.

I know, but somehow you need to say: "please give me p2p pages for specific device and not random device in the system as it is now". This is what is missing from my PoV.

Thanks

#36 lore raw
In reply to Yonatan Maman

Please use a more suitable name for your series. There's absolutle nothing GPU-specific here, and reusing the name from a complete trainwreck that your company pushed over the last few years doesn't help either.