GPU Direct RDMA (P2P DMA) for Device Private Pages
Subsystems (9)
cover GPU Direct RDMA (P2P DMA) for Device Private Pages
From: Yonatan Maman <[email protected]>
Based on: Provide a new two step DMA mapping API patchset
https://lore.kernel.org/kvm/[email protected]/T/#t
This patch series aims to enable Peer-to-Peer (P2P) DMA access in GPU-centric applications that utilize RDMA and private device pages. This enhancement reduces data transfer overhead by allowing the GPU to directly expose device private page data to devices such as NICs, eliminating the need to traverse system RAM, which is the native method for exposing device private page data.
To fully support Peer-to-Peer for device private pages, the following changes are proposed:
Memory Management (MM)
* Leverage struct pagemap_ops to support P2P page operations: This
modification ensures that the GPU can directly map device private pages for P2P DMA.
* Utilize hmm_range_fault to support P2P connections for device private
pages (instead of Page fault)
IB Drivers
Add TRY_P2P_REQ flag for the hmm_range_fault call: This flag indicates the need for P2P mapping, enabling IB drivers to efficiently handle P2P DMA requests.
Nouveau driver
Add support for the Nouveau p2p_page callback function: This update integrates P2P DMA support into the Nouveau driver, allowing it to handle P2P page operations seamlessly.
MLX5 Driver
Utilize NIC Address Translation Service (ATS) for ODP memory, to optimize DMA P2P for private device pages. Also, when P2P DMA mapping fails due to inaccessible bridges, the system falls back to standard DMA, which uses host memory, for the affected PFNs
Yonatan Maman (5):
mm/hmm: HMM API to enable P2P DMA for device private pages nouveau/dmem: HMM P2P DMA for private dev pages IB/core: P2P DMA for device private pages RDMA/mlx5: Add fallback for P2P DMA errors RDMA/mlx5: Enabling ATS for ODP memory
drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++ drivers/infiniband/core/umem_odp.c | 4 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +- drivers/infiniband/hw/mlx5/odp.c | 24 +++++- include/linux/hmm.h | 3 +- include/linux/memremap.h | 8 ++ mm/hmm.c | 57 ++++++++++--- 7 files changed, 195 insertions(+), 17 deletions(-)
Signature · 1 line
2.34.1
RFC 1/5 mm/hmm: HMM API to enable P2P DMA for device private pages
From: Yonatan Maman <[email protected]>
hmm_range_fault() by default triggered a page fault on device private when HMM_PFN_REQ_FAULT flag was set. pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead.
Introduced an API to support P2P DMA for device private pages,includes:
- Leveraging the struct pagemap_ops for P2P Page Callbacks. This callback involves mapping the page for P2P DMA and returning the corresponding PCI_P2P page.
- Utilizing hmm_range_fault for initializing P2P DMA. The API also adds the HMM_PFN_REQ_TRY_P2P flag option for the hmm_range_fault caller to initialize P2P. If set, hmm_range_fault attempts initializing the P2P connection first, if the owner device supports P2P, using p2p_page. In case of failure or lack of support, hmm_range_fault will continue with the regular flow of migrating the page to RAM.
This change does not affect previous use-cases of hmm_range_fault, because both the caller and the page owner must explicitly request and support it to initialize P2P connection.
--- include/linux/hmm.h | 3 ++- include/linux/memremap.h | 8 ++++++ mm/hmm.c | 57 +++++++++++++++++++++++++++++++++------- 3 files changed, 57 insertions(+), 11 deletions(-)
include/linux/hmm.h +2 -0
@@ -26,6 +26,7 @@ struct mmu_interval_notifier;* HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped +* HMM_PFN_ALLOW_P2P - Allow returning PCI P2PDMA page * * On input: * 0 - Return the current state of the page, do not fault it.
@@ -41,7 +42,7 @@ enum hmm_pfn_flags {HMM_PFN_ERROR =1UL << (BITS_PER_LONG -3 ),/* Sticky flag, carried from Input to Output */ +HMM_PFN_ALLOW_P2P =1UL << (BITS_PER_LONG -6 ),HMM_PFN_DMA_MAPPED =1UL << (BITS_PER_LONG -7 ), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8),
include/linux/memremap.h +8 -0
@@ -89,6 +89,14 @@ struct dev_pagemap_ops {*/ vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf); +/* +* Used for private (un-addressable) device memory only. Return a +* corresponding PFN for a page that can be mapped to device +* (e.g using dma_map_page) +*/ +int (*get_dma_pfn_for_device)(struct page *private_page, +unsigned long *dma_pfn); +/* * Handle the memory failure happens on a range of pfns. Notify the * processes who are using these pfns, and try to recover the data on
mm/hmm.c +43 -0
49 lines not on this page · raw message
unsigned long *hmm_pfn)
@@ -249,17 +294,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
if (!pte_present(pte)) {
swp_entry_t entry = pte_to_swp_entry(pte);
- /*
- * Don't fault in device private pages owned by the caller,
- * just report the PFN.
- */
if (is_device_private_entry(entry) &&
- pfn_swap_entry_to_page(entry)->pgmap->owner ==
- range->dev_private_owner) {
- cpu_flags = HMM_PFN_VALID;
- if (is_writable_device_private_entry(entry))
- cpu_flags |= HMM_PFN_WRITE;
- *hmm_pfn = (*hmm_pfn & HMM_PFN_DMA_MAPPED) | swp_offset_pfn(entry) | cpu_flags;
+ hmm_handle_device_private(range, pfn_req_flags, entry, hmm_pfn)) {
+ *hmm_pfn = *hmm_pfn & HMM_PFN_DMA_MAPPED;
return 0;
}
Signature · 1 line
2.34.1
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Hi, Jonatan
Quoted 35 lines · Yonatan Maman
From: Yonatan Maman <[email protected]>
hmm_range_fault()by default triggered a page fault on device private whenHMM_PFN_REQ_FAULTflag was set. pages, migrating them to RAM. In some
cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades
performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead.Introduced an API to support P2P DMA for device private pages,includes:
- Leveraging the structpagemap_opsfor P2P Page Callbacks. This callback
involves mapping the page for P2P DMA and returning the corresponding
PCI_P2Ppage.- Utilizing
hmm_range_faultfor initializing P2P DMA. The API also adds theHMM_PFN_REQ_TRY_P2Pflag option for thehmm_range_faultcaller to initialize P2P. If set,hmm_range_faultattempts initializing the P2P connection first, if the owner device
supports P2P, usingp2p_page. In case of failure or lack of support,
hmm_range_faultwill continue with the regular flow of migrating the
page to RAM.This change does not affect previous use-cases of
hmm_range_fault, because both the caller and the page owner must explicitly request and
support it to initialize P2P connection.
Quoted 8 lines
Signed-off-by Yonatan Maman <[email protected]>Signed-off-by Gal Shalom <[email protected]>---
include/linux/hmm.h | 3 ++-include/linux/memremap.h| 8 ++++++
mm/hmm.c | 57 +++++++++++++++++++++++++++++++++----- --3 files changed, 57 insertions(+), 11 deletions(-)
It appears we're working on a very similar thing, (In fact the original proposals were sent out the same day. I have a couple of questions).
Quoted 84 lines
include/linux/hmm.h +0
-0@@ -26,6 +26,7 @@ struct mmu_interval_notifier; * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-outputtransformation
* to mark that page is already DMA mapped
+ *HMM_PFN_ALLOW_P2P- Allow returning PCI P2PDMA page
*
* On input:
* 0 - Return the current state of the page, do not fault it.include/linux/hmm.h +0
-0@@ -41,7 +42,7 @@ enum hmm_pfn_flags { HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3),
/* Sticky flag, carried from Input to Output */include/linux/hmm.h +1
-0+ HMM_PFN_ALLOW_P2P = 1UL << (BITS_PER_LONG - 6),
HMM_PFN_DMA_MAPPED= 1UL << (BITS_PER_LONG- 7),
HMM_PFN_ORDER_SHIFT= (BITS_PER_LONG- 8),include/linux/memremap.h +0
-0@@ -89,6 +89,14 @@ struct dev_pagemap_ops {*/ vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf);include/linux/memremap.h +2
-0+/* +* Used for private (un-addressable) device memory only. Return a
include/linux/memremap.h +6
-0+* corresponding PFN for a page that can be mapped to device +* (e.g using dma_map_page) +*/ +int (*get_dma_pfn_for_device)(struct page *private_page, + unsigned long *dma_pfn); +/*
* Handle the memory failure happens on a range of pfns. Notify the
* processes who are using these pfns, and try to recover the data onmm/hmm.c +0
-0@@ -226,6 +226,51 @@ static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range,return
pte_write(pte)? (HMM_PFN_VALID|HMM_PFN_WRITE) :HMM_PFN_VALID;
}mm/hmm.c +23
-0+static bool hmm_handle_device_private (struct hmm_range *range, + unsigned long pfn_req_flags, + swp_entry_t entry, + unsigned long *hmm_pfn) +{ +struct page *page =pfn_swap_entry_to_page (entry); +struct dev_pagemap *pgmap = page->pgmap; +int ret; + pfn_req_flags &= range->pfn_flags_mask; + pfn_req_flags |= range->default_flags; + +/* +* Don't fault in device private pages owned by the caller, +* just report the PFN. +*/ +if (pgmap->owner == range->dev_private_owner) { + *hmm_pfn =swp_offset_pfn (entry); + goto found; + } + +/* +* P2P for supported pages, and according to caller request +* translate the private page to the match P2P page if it fails
mm/hmm.c +5
-0+* continue with the regular flow +*/ +if (pfn_req_flags &HMM_PFN_ALLOW_P2P && + pgmap->ops->get_dma_pfn_for_device) { + ret = pgmap->ops->get_dma_pfn_for_device(page,
Quoted 1 line
hmm_pfn);
How would the pgmap device know whether P2P is actually possible without knowing the client device, (like calling pci_p2pdma_distance) and also if looking into access control, whether it is allowed?
I wonder whether you could consider using something that is a little more generic that would fit also our use-case. Here the caller provides a callback as to whether devmem access is allowed, but leaves any dma- mapping or pfn mangling to be done after the call to hmm_range_fault(), since hmm_range_fault() really only needs to know whether it has to migrate to system or not. One benefit of using this alternative approach is that struct hmm_range can be subclassed by the caller and for example cache device pairs for which p2p is allowed.
Current version (after the feedback from Jason looks like this). It looks like your use-case could easily be made to fit this one, but, as I understand it, not vice versa: (Could send this as a separate patch if needed).
Thanks,
Thomas
include/linux/hmm.h +0 -0
@@ -76,6 +76,21 @@ static inline unsigned int
hmm_pfn_to_map_order(unsigned long hmm_pfn)
return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F;
}
+struct hmm_range;
+
+/**
+ * struct hmm_range_ops - Functions for detailed cross-device access.
+ */
+struct hmm_range_ops {
+ /**
+ * @devmem_allow: Whether to allow cross-device access todevice_private pages.
+ * @hrange: Pointer to a struct hmm_range. Typically
subclassed by the caller
+ * to provide needed information.
+ * @page: The page being queried.
+ */
+ bool (*devmem_allow)(struct hmm_range *hrange, struct page
*page);
+};
+
/*
* struct hmm_range - track invalidation lock on virtual address range *
@@ -87,6 +102,7 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn)
* @default_flags: default flags for the range (write, read, ... see
hmm doc)
* @pfn_flags_mask: allows to mask pfn flags so that onlydefault_flags matter
* @dev_private_owner: owner of device private pages
+ * @ops: Pointer to a struct hmm_range_ops or NULL if no ops provided. */
struct hmm_range {
struct mmu_interval_notifier *notifier;
@@ -97,6 +113,7 @@ struct hmm_range {
unsigned long default_flags;
unsigned long pfn_flags_mask;
void *dev_private_owner;
+ const struct hmm_range_ops *ops;
};
/*
mm/hmm.c +0 -0
@@ -220,6 +220,15 @@ static inline unsigned long
pte_to_hmm_pfn_flags(struct hmm_range *range,
return pte_write(pte) ? (HMM_PFN_VALID | HMM_PFN_WRITE) :HMM_PFN_VALID;
}
+static bool hmm_devmem_allow(struct hmm_range *range, struct page *page)
+{
+ if (likely(page->pgmap->owner == range->dev_private_owner))
+ return true;
+ if (likely(!range->ops))
+ return false;
+ return range->ops->devmem_allow(range, page);
+}
+
static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long
addr,
unsigned long end, pmd_t *pmdp, pte_t
*ptep,
unsigned long *hmm_pfn)
@@ -245,11 +254,10 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
/*
* Don't fault in device private pages owned by the
caller,
- * just report the PFN.
+ * or that are accessible to the caller. Just report
the PFN.
*/
if (is_device_private_entry(entry) &&
- pfn_swap_entry_to_page(entry)->pgmap->owner ==
- range->dev_private_owner) {
+ hmm_devmem_allow(range,pfn_swap_entry_to_page(entry))) {
cpu_flags = HMM_PFN_VALID;
if (is_writable_device_private_entry(entry))
cpu_flags |= HMM_PFN_WRITE;
Signature · 1 line
2.48.1
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 3 lines · Thomas Hellström
How would the pgmap device know whether P2P is actually possible without knowing the client device, (like calling
pci_p2pdma_distance) and also if looking into access control, whether it is allowed?
The DMA API will do this, this happens after this patch is put on top of Leon's DMA API patches. The mapping operation will fail and it will likely be fatal to whatever is going on.
get_dma_pfn_for_device() returns a new PFN, but that is not a DMA mapped address, it is just a PFN that has another struct page under it.
There is an implicit assumption here that P2P will work and we don't need a 3rd case to handle non-working P2P..
Quoted 3 lines
but leaves any dma- mapping or pfn mangling to be done after the call to
hmm_range_fault(), sincehmm_range_fault()really only needs to know whether it has to migrate to system or not.
See above, this is already the case..
Quoted 3 lines
One benefit of using this alternative
approach is that structhmm_rangecan be subclassed by the caller and for example cache device pairs for which p2p is allowed.
If you want to directly address P2P non-uniformity I'd rather do it directly in the core code than using a per-driver callback. Every driver needs exactly the same logic for such a case.
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 16 lines · Jason Gunthorpe
Quoted 4 lines · Thomas Hellström
How would the pgmap device know whether P2P is actually possible without knowing the client device, (like calling
pci_p2pdma_distance)
and also if looking into access control, whether it is allowed?The DMA API will do this, this happens after this patch is put on top of Leon's DMA API patches. The mapping operation will fail and it will
likely be fatal to whatever is going on.get_dma_pfn_for_device()returns a new PFN, but that is not a DMA mapped address, it is just a PFN that has another struct page under it.
Quoted 2 lines
There is an implicit assumption here that P2P will work and we don't need a 3rd case to handle non-working P2P..
OK. We will have the case where we want pfnmaps with driver-private fast interconnects to return "interconnect possible, don't migrate" whereas possibly other gpus and other devices would return "interconnect unsuitable, do migrate", so (as I understand it) something requiring a more flexible interface than this.
Quoted 7 lines
Quoted 5 lines
but leaves any dma- mapping or pfn mangling to be done after the call to
hmm_range_fault(), sincehmm_range_fault()really only needs
to know whether it has to migrate to system or not.See above, this is already the case..
Well what I meant was at hmm_range_fault() time only consider whether to migrate or not. Afterwards at dma-mapping time you'd expose the alternative pfns that could be used for dma-mapping.
We were actually looking at a solution where the pagemap implements something along
bool devmem_allowed(pagemap, client); //for hmm_range_fault
plus dma_map() and dma_unmap() methods.
In this way you'd don't need to expose special p2p dma pages and the interface could also handle driver-private interconnects, where dma_maps and dma_unmap() methods become trivial.
Quoted 9 lines
Quoted 5 lines
One benefit of using this alternative
approach is that structhmm_rangecan be subclassed by the caller and
for example cache device pairs for which p2p is allowed.If you want to directly address P2P non-uniformity I'd rather do it directly in the core code than using a per-driver callback. Every driver needs exactly the same logic for such a case.
Yeah, and that would look something like the above, although initially we intended to keep these methods in drm allocator around its pagemaps, but could of course look into doing this directly in dev_pagemap ops. But still would probably need some guidance into what's considered acceptable, and I don't think the solution proposed in this patch meets our needs.
Thanks,
Thomas
Quoted 2 lines
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 17 lines · Thomas Hellström
Quoted 16 lines · Jason Gunthorpe
Quoted 4 lines · Thomas Hellström
How would the pgmap device know whether P2P is actually possible without knowing the client device, (like calling
pci_p2pdma_distance)
and also if looking into access control, whether it is allowed?The DMA API will do this, this happens after this patch is put on top of Leon's DMA API patches. The mapping operation will fail and it will
likely be fatal to whatever is going on.get_dma_pfn_for_device()returns a new PFN, but that is not a DMA mapped address, it is just a PFN that has another struct page under it.
Quoted 8 lines
Quoted 2 lines
There is an implicit assumption here that P2P will work and we don't need a 3rd case to handle non-working P2P..
OK. We will have the case where we want pfnmaps with driver-private fast interconnects to return "interconnect possible, don't migrate" whereas possibly other gpus and other devices would return "interconnect unsuitable, do migrate", so (as I understand it) something requiring a more flexible interface than this.
I'm not sure this doesn't handle that case?
Here we are talking about having DEVICE_PRIVATE struct page mappings. On a GPU this should represent GPU local memory that is non-coherent with the CPU, and not mapped into the CPU.
This series supports three case:
1) pgmap->owner == range->dev_private_owner
This is "driver private fast interconnect" in this case HMM should immediately return the page. The calling driver understands the private parts of the pgmap and computes the private interconnect address.
This requires organizing your driver so that all private interconnect has the same pgmap->owner.
2) The page is DEVICE_PRIVATE and get_dma_pfn_for_device() exists. The exporting driver has the option to return a P2P struct page that can be used for PCI P2P without any migration. In a PCI GPU context this means the GPU has mapped its local memory to a PCI address. The assumption is that P2P always works and so this address can be DMA'd from.
3) Migrate back to CPU memory - then eveything works.
Is that not enough? Where do you want something different?
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
but leaves any dma- mapping or pfn mangling to be done after the call to
hmm_range_fault(), sincehmm_range_fault()really only needs
to know whether it has to migrate to system or not.See above, this is already the case..
Well what I meant was at
hmm_range_fault()time only consider whether to migrate or not. Afterwards at dma-mapping time you'd expose the alternative pfns that could be used for dma-mapping.
That sounds like you are talking about multipath, we are not really ready to tackle general multipath yet at the DMA API level, IMHO.
If you are just talking about your private multi-path, then that is already handled..
Quoted 6 lines
We were actually looking at a solution where the pagemap implements something along
bool
devmem_allowed(pagemap, client); //forhmm_range_faultplus
dma_map()anddma_unmap()methods.
This sounds like dmabuf philosophy, and I don't think we should go in this direction. The hmm caller should always be responsible for dma mapping and we need to improve the DMA API to make this work better, not build side hacks like this.
You can read my feelings and reasoning on this topic within this huge thread:
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 1 line
In this way you'd don't need to expose special p2p dma pages and the
Removing the "special p2p dma pages" has to be done by improving the DMA API to understand how to map phsyical addresses without struct page. We are working toward this, slowly.
pgmap->ops->dma_map/unmap() ideas just repeat the DMABUF mistake of mis-using the DMA API for P2P cases. Today you cannot correctly DMA map P2P memory without the struct page.
Quoted 2 lines
interface could also handle driver-private interconnects, where
dma_mapsanddma_unmap()methods become trivial.
We already handle private interconnect.
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines
One benefit of using this alternative
approach is that structhmm_rangecan be subclassed by the caller and
for example cache device pairs for which p2p is allowed.If you want to directly address P2P non-uniformity I'd rather do it directly in the core code than using a per-driver callback. Every driver needs exactly the same logic for such a case.
Yeah, and that would look something like the above
No, it would look like the core HMM code calling pci distance on the P2P page returned from get_dma_pfn_for_device() and if P2P was impossible then proceed to option #3 fault to CPU.
Quoted 5 lines
although initially we intended to keep these methods in drm allocator around its pagemaps, but could of course look into doing this directly in
dev_pagemapops. But still would probably need some guidance into what's considered acceptable, and I don't think the solution proposed in this patch meets our needs.
I'm still not sure what you are actually trying to achieve?
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 46 lines · Jason Gunthorpe
Quoted 24 lines · Thomas Hellström
Quoted 19 lines · Jason Gunthorpe
Quoted 5 lines · Thomas Hellström
How would the pgmap device know whether P2P is actually possible
without knowing the client device, (like callingpci_p2pdma_distance)
and also if looking into access control, whether it is allowed?The DMA API will do this, this happens after this patch is put on top
of Leon's DMA API patches. The mapping operation will fail and it will
likely be fatal to whatever is going on.get_dma_pfn_for_device()returns a new PFN, but that is not a DMA mapped address, it is just a PFN that has another struct page under
it.Quoted 3 lines
There is an implicit assumption here that P2P will work and we don't
need a 3rd case to handle non-working P2P..Quoted 5 lines
OK. We will have the case where we want pfnmaps with driver-private fast interconnects to return "interconnect possible, don't migrate" whereas possibly other gpus and other devices would return "interconnect unsuitable, do migrate", so (as I understand it) something requiring a more flexible interface than this.
I'm not sure this doesn't handle that case?
Here we are talking about having
DEVICE_PRIVATEstruct page mappings. On a GPU this should represent GPU local memory that is non-coherent with the CPU, and not mapped into the CPU.This series supports three case:
1)
pgmap->owner==range->dev_private_owner
This is "driver private fast interconnect" in this case HMM should
immediately return the page. The calling driver understands the private parts of the pgmap and computes the private interconnect address.
Quoted 2 lines
This requires organizing your driver so that all private interconnect has the same
pgmap->owner.
Yes, although that makes this map static, since pgmap->owner has to be set at pgmap creation time. and we were during initial discussions looking at something dynamic here. However I think we can probably do with a per-driver owner for now and get back if that's not sufficient.
Quoted 7 lines
2) The page is
DEVICE_PRIVATEandget_dma_pfn_for_device()exists. The exporting driver has the option to return a P2P struct page that can be used for PCI P2P without any migration. In a PCI GPU context this means the GPU has mapped its local memory to a PCI address. The assumption is that P2P always works and so this address can be DMA'd from.
So do I understand it correctly, that the driver then needs to set up one device_private struct page and one pcie_p2p struct page for each page of device memory participating in this way?
Quoted 18 lines
3) Migrate back to CPU memory - then eveything works.
Is that not enough? Where do you want something different?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
but leaves any dma- mapping or pfn mangling to be done after the
call tohmm_range_fault(), sincehmm_range_fault()really only needs
to know whether it has to migrate to system or not.Quoted 6 lines
Quoted 1 line
See above, this is already the case..
Well what I meant was at
hmm_range_fault()time only consider whether
to migrate or not. Afterwards at dma-mapping time you'd expose the alternative pfns that could be used for dma-mapping.
Quoted 5 lines
That sounds like you are talking about multipath, we are not really ready to tackle general multipath yet at the DMA API level, IMHO.
If you are just talking about your private multi-path, then that is already handled..
No, the issue I'm having with this is really why would hmm_range_fault() need the new pfn when it could easily be obtained from the device-private pfn by the hmm_range_fault() caller? The only thing hmm_range_fault() needs to know is, again, whether to migrate or not. But I guess if the plan is to have hmm_range_fault() call pci_p2pdma_distance() on it, and we don't want the exporter to do that, it makes sense.
Quoted 21 lines
Quoted 7 lines
We were actually looking at a solution where the pagemap implements something along
bool
devmem_allowed(pagemap, client); //forhmm_range_faultplus
dma_map()anddma_unmap()methods.This sounds like dmabuf philosophy, and I don't think we should go in this direction. The hmm caller should always be responsible for dma mapping and we need to improve the DMA API to make this work better, not build side hacks like this.
You can read my feelings and reasoning on this topic within this huge thread:
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 2 lines
In this way you'd don't need to expose special p2p dma pages and the
Quoted 8 lines
Removing the "special p2p dma pages" has to be done by improving the DMA API to understand how to map phsyical addresses without struct page. We are working toward this, slowly.
pgmap->ops->dma_map/unmap() ideas just repeat the DMABUF mistake of mis-using the DMA API for P2P cases. Today you cannot correctly DMA
map P2P memory without the struct page.
Yeah, I don't want to drag hmm into that discussion, although admittedly the idea of pgmap->ops->dma_map/unmap mimics the dma-buf behaviour.
So anyway what we'll do is to try to use an interconnect-common owner for now and revisit the problem if that's not sufficient so we can come up with an acceptable solution.
/Thomas
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 9 lines · Thomas Hellström
Quoted 9 lines
This series supports three case:
1)
pgmap->owner==range->dev_private_owner
This is "driver private fast interconnect" in this case HMM should
immediately return the page. The calling driver understands the private parts of the pgmap and computes the private interconnect address.
Quoted 7 lines
Quoted 2 lines
This requires organizing your driver so that all private interconnect has the same
pgmap->owner.Yes, although that makes this map static, since
pgmap->ownerhas to be set at pgmap creation time. and we were during initial discussions looking at something dynamic here. However I think we can probably do with a per-driver owner for now and get back if that's not sufficient.
The pgmap->owner doesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?
Quoted 10 lines
Quoted 6 lines
2) The page is
DEVICE_PRIVATEandget_dma_pfn_for_device()exists. The exporting driver has the option to return a P2P struct page that can be used for PCI P2P without any migration. In a PCI GPU context this means the GPU has mapped its local memory to a PCI address. The assumption is that P2P always works and so this address can be DMA'd from.So do I understand it correctly, that the driver then needs to set up one
device_privatestruct page and onepcie_p2pstruct page for each page of device memory participating in this way?
Yes, for now. I hope to remove the p2p page eventually.
Quoted 6 lines
Quoted 2 lines
If you are just talking about your private multi-path, then that is already handled..
No, the issue I'm having with this is really why would
hmm_range_fault()need the new pfn when it could easily be obtained from the device-private pfn by thehmm_range_fault()caller?
That isn't the API of HMM, the caller uses hmm to get PFNs it can use.
Deliberately returning PFNs the caller cannot use is nonsensical to it's purpose :)
Quoted 3 lines
So anyway what we'll do is to try to use an interconnect-common owner for now and revisit the problem if that's not sufficient so we can come up with an acceptable solution.
That is the intention for sure. The idea was that the drivers under the private pages would somehow generate unique owners for shared private interconnect segments.
I wouldn't say this is the end all of the idea, if there are better ways to handle accepting private pages they can certainly be explored..
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 18 lines · Jason Gunthorpe
Quoted 9 lines · Thomas Hellström
Quoted 9 lines
This series supports three case:
1)
pgmap->owner==range->dev_private_owner
This is "driver private fast interconnect" in this case HMM should
immediately return the page. The calling driver understands the private parts of the pgmap and computes the private interconnect address.Quoted 7 lines
Quoted 2 lines
This requires organizing your driver so that all private interconnect has the same
pgmap->owner.Yes, although that makes this map static, since
pgmap->ownerhas to be set at pgmap creation time. and we were during initial discussions looking at something dynamic here. However I think we can probably do with a per-driver owner for now and get back if that's not sufficient.
Quoted 4 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?
The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on pgmap->owner to figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet.
You can fake that by doing these checks after hmm_range_fault returned, and if you get a bunch of unsuitable pages, toss it back to hmm_range_fault asking for an unconditional migration to system memory for those. But that's kinda not great and I think goes at least against the spirit of how you want to handle pci p2p in step 2 below?
Cheers, Sima
Quoted 38 lines
Quoted 10 lines
Quoted 6 lines
2) The page is
DEVICE_PRIVATEandget_dma_pfn_for_device()exists. The exporting driver has the option to return a P2P struct page that can be used for PCI P2P without any migration. In a PCI GPU context this means the GPU has mapped its local memory to a PCI address. The assumption is that P2P always works and so this address can be DMA'd from.So do I understand it correctly, that the driver then needs to set up one
device_privatestruct page and onepcie_p2pstruct page for each page of device memory participating in this way?Yes, for now. I hope to remove the p2p page eventually.
Quoted 6 lines
Quoted 2 lines
If you are just talking about your private multi-path, then that is already handled..
No, the issue I'm having with this is really why would
hmm_range_fault()need the new pfn when it could easily be obtained from the device-private pfn by thehmm_range_fault()caller?That isn't the API of HMM, the caller uses hmm to get PFNs it can use.
Deliberately returning PFNs the caller cannot use is nonsensical to it's purpose :)
Quoted 3 lines
So anyway what we'll do is to try to use an interconnect-common owner for now and revisit the problem if that's not sufficient so we can come up with an acceptable solution.
That is the intention for sure. The idea was that the drivers under the private pages would somehow generate unique owners for shared private interconnect segments.
I wouldn't say this is the end all of the idea, if there are better ways to handle accepting private pages they can certainly be explored..
Jason
Signature · 3 lines
Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 11 lines · Simona Vetter
Quoted 4 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on
pgmap->ownerto figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet.
Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in hmm_range_fault..
The notion is that owner should represent a contiguous region of connectivity. IMHO you can always create this, but I suppose there could be corner cases where you need to split/merge owners.
But again, this isn't set in stone, if someone has a better way to match the private interconnects without going to driver callbacks then try that too.
I think driver callbacks inside hmm_range_fault should be the last resort..
Quoted 5 lines
You can fake that by doing these checks after
hmm_range_faultreturned, and if you get a bunch of unsuitable pages, toss it back tohmm_range_faultasking for an unconditional migration to system memory for those. But that's kinda not great and I think goes at least against the spirit of how you want to handle pci p2p in step 2 below?
Right, hmm_range_fault should return pages that can be used and you should not call it twice.
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 20 lines · Jason Gunthorpe
Quoted 9 lines · Simona Vetter
Quoted 5 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before
you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?The trouble I'm seeing is device probe and the fundemantal issue that you
never know when you're done. And so if we entirely rely on pgmap-Quoted 8 lines
Quoted 1 line
owner to
figure out the driver private interconnect topology, that's going to be
messy. That's why I'm also leaning towards both comparing owners and
having an additional check whether the interconnect is actually there or
not yet.
Quoted 3 lines
Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in
hmm_range_fault..
Just a comment on the performance concern here. This can be crafted in a way that only if the driver provides a callback, there is a (small) hit. If there is no callback at that point, we're looking at a migration to ram. If there is a callback it's typically followed by an address computation and page-table setup. Compared to those, the callback performance impact is probably unmeasureable.
/Thomas
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
In reply to Jason GunthorpeQuoted 14 lines · Jason Gunthorpe
Quoted 11 lines · Simona Vetter
Quoted 4 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on
pgmap->ownerto figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet.
Quoted 3 lines
Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in
hmm_range_fault..
I'm not sure how you want to make the owner mutable.
The only design that I think is solid is to evict all device private memory, unregister the dev_pagemap and register a new one with the updated owner. I think any other approach boils down to the same issue, except we pretend it's easier and just ignore all the race conditions.
And I've looked at the lifetime fun of unregistering a dev_pagemap for device hotunplug and pretty firmly concluded it's unfixable and that I should run away to do something else :-P
An optional callback is a lot less scary to me here (or redoing hmm_range_fault or whacking the migration helpers a few times) looks a lot less scary than making pgmap->owner mutable in some fashion.
Cheers, Sima
Quoted 20 lines
The notion is that owner should represent a contiguous region of connectivity. IMHO you can always create this, but I suppose there could be corner cases where you need to split/merge owners.
But again, this isn't set in stone, if someone has a better way to match the private interconnects without going to driver callbacks then try that too.
I think driver callbacks inside
hmm_range_faultshould be the last resort..Quoted 5 lines
You can fake that by doing these checks after
hmm_range_faultreturned, and if you get a bunch of unsuitable pages, toss it back tohmm_range_faultasking for an unconditional migration to system memory for those. But that's kinda not great and I think goes at least against the spirit of how you want to handle pci p2p in step 2 below?Right,
hmm_range_faultshould return pages that can be used and you should not call it twice.Jason
Signature · 3 lines
Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 15 lines · Simona Vetter
Quoted 14 lines · Jason Gunthorpe
Quoted 11 lines · Simona Vetter
Quoted 4 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on
pgmap->ownerto figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet.
Quoted 5 lines
Quoted 3 lines
Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in
hmm_range_fault..I'm not sure how you want to make the owner mutable.
You'd probably have to use a system where you never free them until all the page maps are destroyed.
You could also use an integer instead of a pointer to indicate the cluster of interconnect, I think there are many options..
Quoted 3 lines
And I've looked at the lifetime fun of unregistering a
dev_pagemapfor device hotunplug and pretty firmly concluded it's unfixable and that I should run away to do something else :-P
? It is supposed to work, it blocks until all the pages are freed, but AFAIK ther is no fundamental life time issue. The driver is responsible to free all its usage.
Quoted 3 lines
An optional callback is a lot less scary to me here (or redoing
hmm_range_faultor whacking the migration helpers a few times) looks a lot less scary than makingpgmap->ownermutable in some fashion.
It extra for every single 4k page on every user :\
And what are you going to do better inside this callback?
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 20 lines · Jason Gunthorpe
Quoted 19 lines · Simona Vetter
Quoted 17 lines · Jason Gunthorpe
Quoted 11 lines · Simona Vetter
Quoted 4 lines
The
pgmap->ownerdoesn't *have* to fixed, certainly during early boot before you hand out any page references it can be changed. I wouldn't be surprised if this is useful to some requirements to build up the private interconnect topology?The trouble I'm seeing is device probe and the fundemantal issue that you never know when you're done. And so if we entirely rely on
pgmap->ownerto figure out the driver private interconnect topology, that's going to be messy. That's why I'm also leaning towards both comparing owners and having an additional check whether the interconnect is actually there or not yet.Hoenstely, I'd rather invest more effort into being able to update owner for those special corner cases than to slow down the fast path in
hmm_range_fault..
Quoted 7 lines
Quoted 1 line
I'm not sure how you want to make the owner mutable.
You'd probably have to use a system where you never free them until all the page maps are destroyed.
You could also use an integer instead of a pointer to indicate the cluster of interconnect, I think there are many options..
Hm yeah I guess an integer allocater of the atomic_inc kind plus "surely 32bit is enough" could work. But I don't think it's needed, if we can reliable just unregister the entire dev_pagemap and then just set up a new one. Plus that avoids thinking about which barriers we might need where exactly all over mm code that looks at the owner field.
Quoted 7 lines
Quoted 3 lines
And I've looked at the lifetime fun of unregistering a
dev_pagemapfor device hotunplug and pretty firmly concluded it's unfixable and that I should run away to do something else :-P? It is supposed to work, it blocks until all the pages are freed, but AFAIK ther is no fundamental life time issue. The driver is responsible to free all its usage.
Hm I looked at it again, and I guess with the fixes to make migration to system memory work reliable in Matt Brost's latest series it should indeed work reliable. The devm_ version still freaks me out because of how easily people misuse these for things that are memory allocations.
Quoted 7 lines
Quoted 3 lines
An optional callback is a lot less scary to me here (or redoing
hmm_range_faultor whacking the migration helpers a few times) looks a lot less scary than makingpgmap->ownermutable in some fashion.It extra for every single 4k page on every user :\
And what are you going to do better inside this callback?
Having more comfy illusions :-P
Slightly more seriously, I can grab some locks and make life easier, whereas sprinkling locking or even barriers over pgmap->owner in core mm is not going to fly. Plus more flexibility, e.g. when the interconnect doesn't work for atomics or some other funny reason it only works for some of the traffic, where you need to more dynamically decide where memory is ok to sit. Or checking p2pdma connectivity and all that stuff.
But we can also do all that stuff by checking afterwards or migrating memory around as needed. At least for drivers who cooperate and all set the same owner, which I think is Thomas' current plan.
Also note that fundamentally you can't protect against the hotunplug or driver unload case for hardware access. So some access will go to nowhere when that happens, until we've torn down all the mappings and migrated memory out.
-Sima
Signature · 3 lines
Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 8 lines · Simona Vetter
Quoted 2 lines
You could also use an integer instead of a pointer to indicate the cluster of interconnect, I think there are many options..
Hm yeah I guess an integer allocater of the
atomic_inckind plus "surely 32bit is enough" could work. But I don't think it's needed, if we can reliable just unregister the entiredev_pagemapand then just set up a new one. Plus that avoids thinking about which barriers we might need where exactly all over mm code that looks at the owner field.
IMHO that is the best answer if it works for the driver.
Quoted 8 lines
Quoted 3 lines
? It is supposed to work, it blocks until all the pages are freed, but AFAIK ther is no fundamental life time issue. The driver is responsible to free all its usage.
Hm I looked at it again, and I guess with the fixes to make migration to system memory work reliable in Matt Brost's latest series it should indeed work reliable. The
devm_version still freaks me out because of how easily people misuse these for things that are memory allocations.
I also don't like the devm stuff, especially in costly places like this. Oh well.
Quoted 9 lines
Quoted 7 lines
Quoted 3 lines
An optional callback is a lot less scary to me here (or redoing
hmm_range_faultor whacking the migration helpers a few times) looks a lot less scary than makingpgmap->ownermutable in some fashion.It extra for every single 4k page on every user :\
And what are you going to do better inside this callback?
Having more comfy illusions :-P
Exactly!
Quoted 1 line
Slightly more seriously, I can grab some locks and make life easier,
Yes, but then see my concern about performance again. Now you are locking/unlocking every 4k? And then it still races since it can change after hmm_range_fault returns. That's not small, and not really better.
Quoted 5 lines
whereas sprinkling locking or even barriers over
pgmap->ownerin core mm is not going to fly. Plus more flexibility, e.g. when the interconnect doesn't work for atomics or some other funny reason it only works for some of the traffic, where you need to more dynamically decide where memory is ok to sit.
Sure, an asymmetric mess could be problematic, and we might need more later, but lets get to that first..
Quoted 1 line
Or checking p2pdma connectivity and all that stuff.
Should be done in the core code, don't want drivers open coding this stuff.
Quoted 4 lines
Also note that fundamentally you can't protect against the hotunplug or driver unload case for hardware access. So some access will go to nowhere when that happens, until we've torn down all the mappings and migrated memory out.
I think a literal (safe!) hot unplug must always use the page map revoke, and that should be safely locked between hmm_range_fault and the notifier.
If the underlying fabric is loosing operations during an unplanned hot unplug I expect things will need resets to recover..
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Simona Vetter
Quoted 7 lines
Quoted 3 lines
An optional callback is a lot less scary to me here (or redoing
hmm_range_faultor whacking the migration helpers a few times) looks a lot less scary than makingpgmap->ownermutable in some fashion.It extra for every single 4k page on every user :\
And what are you going to do better inside this callback?
Having more comfy illusions :-P
Quoted 8 lines
Exactly!
Quoted 1 line
Slightly more seriously, I can grab some locks and make life easier,
Yes, but then see my concern about performance again. Now you are locking/unlocking every 4k? And then it still races since it can change after
hmm_range_faultreturns. That's not small, and not really better.
Hm yeah, I think that's the death argument for the callback. Consider me convinced on that being a bad idea.
Quoted 6 lines
Quoted 5 lines
whereas sprinkling locking or even barriers over
pgmap->ownerin core mm is not going to fly. Plus more flexibility, e.g. when the interconnect doesn't work for atomics or some other funny reason it only works for some of the traffic, where you need to more dynamically decide where memory is ok to sit.
Quoted 7 lines
Sure, an asymmetric mess could be problematic, and we might need more later, but lets get to that first..
Quoted 1 line
Or checking p2pdma connectivity and all that stuff.
Should be done in the core code, don't want drivers open coding this stuff.
Yeah so after amdkfd and noveau I agree that letting drivers mess this up isn't great. But see below, I'm not sure whether going all the way to core code is the right approach, at least for gpu internal needs.
Quoted 11 lines
Quoted 4 lines
Also note that fundamentally you can't protect against the hotunplug or driver unload case for hardware access. So some access will go to nowhere when that happens, until we've torn down all the mappings and migrated memory out.
I think a literal (safe!) hot unplug must always use the page map revoke, and that should be safely locked between
hmm_range_faultand the notifier.If the underlying fabric is loosing operations during an unplanned hot unplug I expect things will need resets to recover..
So one aspect where I don't like the pgmap->owner approach much is that it's a big thing to get right, and it feels a bit to me that we don't yet know the right questions.
A bit related is that we'll have to do some driver-specific migration after hmm_range_fault anyway for allocation policies. With coherent interconnect that'd be up to numactl, but for driver private it's all up to the driver. And once we have that, we can also migrate memory around that's misplaced for functional and not just performance reasons.
The plan I discussed with Thomas a while back at least for gpus was to have that as a drm_devpagemap library, which would have a common owner (or maybe per driver or so as Thomas suggested). Then it would still not be in drivers, but also a bit easier to mess around with for experiments. And once we have some clear things that hmm_range_fault should do instead for everyone, we can lift them up.
Doing this at a pagemap level should also be much more efficient, since I think we can make the assumption that access limitations are uniform for a given dev_pagemap (and if they're not if e.g. not the entire vram is bus visible, drivers can handle that by splitting things up).
But upfront speccing all this out doesn't seem like a good idea to, because I honestly don't know what we all need.
Cheers, Sima
Signature · 3 lines
Simona Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 3 lines · Simona Vetter
So one aspect where I don't like the
pgmap->ownerapproach much is that it's a big thing to get right, and it feels a bit to me that we don't yet know the right questions.
Well, I would say it isn't really complete yet. No driver has yet attempted to use a private interconnect with these scheme. Probably it needs more work.
Quoted 5 lines
A bit related is that we'll have to do some driver-specific migration after
hmm_range_faultanyway for allocation policies. With coherent interconnect that'd be up to numactl, but for driver private it's all up to the driver. And once we have that, we can also migrate memory around that's misplaced for functional and not just performance reasons.
Are you sure? This doesn't seem to what any hmm_range_fault() user should be doing. hmm_range_fault() is to help mirror the page table to a secondary, that is all. Migration policy shouldn't be part of it, just mirroring doesn't necessarily mean any access was performed, for instance.
And mirroring doesn't track any access done by non-faulting cases either.
Quoted 2 lines
The plan I discussed with Thomas a while back at least for gpus was to have that as a
drm_devpagemaplibrary,
I would not be happy to see this. Please improve pagemap directly if you think you need more things.
Quoted 2 lines
which would have a common owner (or
maybe per driver or so as Thomas suggested).
Neither really match the expected design here. The owner should be entirely based on reachability. Devices that cannot reach each other directly should have different owners.
Quoted 2 lines
But upfront speccing all this out doesn't seem like a good idea to, because I honestly don't know what we all need.
This is why it is currently just void *owner :)
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 32 lines · Jason Gunthorpe
Quoted 5 lines · Simona Vetter
So one aspect where I don't like the
pgmap->ownerapproach much is that
it's a big thing to get right, and it feels a bit to me that we don't yet
know the right questions.Well, I would say it isn't really complete yet. No driver has yet attempted to use a private interconnect with these scheme. Probably it
needs more work.Quoted 8 lines
A bit related is that we'll have to do some driver-specific migration
afterhmm_range_faultanyway for allocation policies. With coherent interconnect that'd be up to numactl, but for driver private it's all up
to the driver. And once we have that, we can also migrate memory around
that's misplaced for functional and not just performance reasons.Are you sure? This doesn't seem to what any
hmm_range_fault()user should be doing.hmm_range_fault()is to help mirror the page table to a secondary, that is all. Migration policy shouldn't be part of it,
just mirroring doesn't necessarily mean any access was performed, for instance.And mirroring doesn't track any access done by non-faulting cases either.
Quoted 6 lines
Quoted 3 lines
The plan I discussed with Thomas a while back at least for gpus was to
have that as adrm_devpagemaplibrary,I would not be happy to see this. Please improve pagemap directly if you think you need more things.
These are mainly helpers to migrate and populate a range of cpu memory space (struct mm_struct) with GPU device_private memory, migrate to system on gpu memory shortage and implement the migrate_to_vram pagemap op, tied to gpu device memory allocations, so I don't think there is anything we should be exposing at the dev_pagemap level at this point?
Quoted 7 lines
Quoted 3 lines
which would have a common owner (or
maybe per driver or so as Thomas suggested).Neither really match the expected design here. The owner should be entirely based on reachability. Devices that cannot reach each other directly should have different owners.
Actually what I'm putting together is a small helper to allocate and assign an "owner" based on devices that are previously registered to a "registry". The caller has to indicate using a callback function for each struct device pair whether there is a fast interconnect available, and this is expected to be done at pagemap creation time, so I think this aligns with the above. Initially a "registry" (which is a list of device-owner pairs) will be driver-local, but could easily have a wider scope.
This means we handle access control, unplug checks and similar at migration time, typically before hmm_range_fault(), and the role of hmm_range_fault() will be to over pfns whose backing memory is directly accessible to the device, else migrate to system.
Device unplug would then be handled by refusing migrations to the device (gpu drivers would probably use drm_dev_enter()), and then evict all device memory after a drm_dev_unplug(). This of course relies on that eviction is more or less failsafe.
/Thomas
Quoted 5 lines
Quoted 3 lines
But upfront speccing all this out doesn't seem like a good idea to, because I honestly don't know what we all need.
This is why it is currently just void *owner :)
Again, with the above I think we are good for now, but having experimented a lot with the callback, I'm still not convinced by the performance argument, for the following reasons.
1) Existing users would never use the callback. They can still rely on the owner check, only if that fails we check for callback existence.
2) By simply caching the result from the last checked dev_pagemap, most callback calls could typically be eliminated.
3) As mentioned before, a callback call would typically always be followed by either migration to ram or a page-table update. Compared to these, the callback overhead would IMO be unnoticeable.
4) pcie_p2p is already planning a dev_pagemap callback?
/Thomas
Quoted 2 lines
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 8 lines · Thomas Hellström
Quoted 2 lines
I would not be happy to see this. Please improve pagemap directly if you think you need more things.
These are mainly helpers to migrate and populate a range of cpu memory space (struct
mm_struct) with GPUdevice_privatememory, migrate to system on gpu memory shortage and implement themigrate_to_vrampagemap op, tied to gpu device memory allocations, so I don't think there is anything we should be exposing at thedev_pagemaplevel at this point?
Maybe that belongs in mm/hmm then?
Quoted 4 lines
Quoted 3 lines
Neither really match the expected design here. The owner should be entirely based on reachability. Devices that cannot reach each other directly should have different owners.
Quoted 8 lines
Actually what I'm putting together is a small helper to allocate and assign an "owner" based on devices that are previously registered to a "registry". The caller has to indicate using a callback function for each struct device pair whether there is a fast interconnect available, and this is expected to be done at pagemap creation time, so I think this aligns with the above. Initially a "registry" (which is a list of device-owner pairs) will be driver-local, but could easily have a wider scope.
Yeah, that seems like a workable idea
Quoted 4 lines
This means we handle access control, unplug checks and similar at migration time, typically before
hmm_range_fault(), and the role ofhmm_range_fault()will be to over pfns whose backing memory is directly accessible to the device, else migrate to system.
Yes, that sound right
Quoted 4 lines
1) Existing users would never use the callback. They can still rely on the owner check, only if that fails we check for callback existence.
2) By simply caching the result from the last checkeddev_pagemap, most callback calls could typically be eliminated.
But then you are not in the locked region so your cache is racy and invalid.
Quoted 3 lines
3) As mentioned before, a callback call would typically always be followed by either migration to ram or a page-table update. Compared to these, the callback overhead would IMO be unnoticeable.
Why? Surely the normal case should be a callback saying the memory can be accessed?
Quoted 1 line
4)
pcie_p2pis already planning adev_pagemapcallback?
Yes, but it is not a racy validation callback, and it already is creating a complicated lifecycle problem inside the exporting the driver.
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Thomas Hellström
1) Existing users would never use the callback. They can still rely on
the owner check, only if that fails we check for callback existence.
2) By simply caching the result from the last checkeddev_pagemap, most
callback calls could typically be eliminated.
Quoted 2 lines
But then you are not in the locked region so your cache is racy and invalid.
I'm not sure I follow? If a device private pfn handed back to the caller is dependent on dev_pagemap A having a fast interconnect to the client, then subsequent pfns in the same hmm_range_fault() call must be able to make the same assumption (pagemap A having a fast interconnect), else the whole result is invalid?
Quoted 9 lines
Quoted 5 lines
3) As mentioned before, a callback call would typically always be followed by either migration to ram or a page-table update. Compared to
these, the callback overhead would IMO be unnoticeable.Why? Surely the normal case should be a callback saying the memory can
be accessed?
Sure, but at least on the xe driver, that means page-table repopulation since the hmm_range_fault() typically originated from a page-fault.
Quoted 6 lines
Quoted 2 lines
4)
pcie_p2pis already planning adev_pagemapcallback?Yes, but it is not a racy validation callback, and it already is creating a complicated lifecycle problem inside the exporting the driver.
Yeah, I bet there are various reasons against a callback. I just don't see the performance argument being a main concern.
Quoted 2 lines
Jason
/Thomas
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 12 lines · Thomas Hellström
Quoted 11 lines · Jason Gunthorpe
Quoted 9 lines · Thomas Hellström
1) Existing users would never use the callback. They can still rely on
the owner check, only if that fails we check for callback existence.
2) By simply caching the result from the last checkeddev_pagemap, most
callback calls could typically be eliminated.
Quoted 8 lines
Quoted 2 lines
But then you are not in the locked region so your cache is racy and invalid.
I'm not sure I follow? If a device private pfn handed back to the caller is dependent on
dev_pagemapA having a fast interconnect to the client, then subsequent pfns in the samehmm_range_fault()call must be able to make the same assumption (pagemap A having a fast interconnect), else the whole result is invalid?
But what is the receiver going to do with this device private page? Relock it again and check again if it is actually OK? Yuk.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
3) As mentioned before, a callback call would typically always be followed by either migration to ram or a page-table update. Compared to
these, the callback overhead would IMO be unnoticeable.Why? Surely the normal case should be a callback saying the memory can
be accessed?Sure, but at least on the xe driver, that means page-table repopulation since the
hmm_range_fault()typically originated from a page-fault.
Yes, I expect all hmm_range_fault()'s to be on page fault paths, and we'd like it to be as fast as we can in the CPU present case..
Jason
Re: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages
Quoted 27 lines · Jason Gunthorpe
Quoted 18 lines · Thomas Hellström
Quoted 13 lines · Jason Gunthorpe
Quoted 11 lines · Thomas Hellström
1) Existing users would never use the callback. They can still rely
on
the owner check, only if that fails we check for callback existence.
2) By simply caching the result from the last checkeddev_pagemap,
most
callback calls could typically be eliminated.Quoted 3 lines
But then you are not in the locked region so your cache is racy and
invalid.Quoted 7 lines
I'm not sure I follow? If a device private pfn handed back to the caller is dependent on
dev_pagemapA having a fast interconnect to the
client, then subsequent pfns in the samehmm_range_fault()call must be
able to make the same assumption (pagemap A having a fast interconnect), else the whole result is invalid?
Quoted 2 lines
But what is the receiver going to do with this device private page? Relock it again and check again if it is actually OK? Yuk.
I'm still lost as to what would be the possible race-condition that can't be handled in the usual way using mmu invalidations + notifier seqno bump? Is it the fast interconnect being taken down?
/Thomas
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
3) As mentioned before, a callback call would typically always be
followed by either migration to ram or a page-table update. Compared to
these, the callback overhead would IMO be unnoticeable.Quoted 8 lines
Quoted 4 lines
Why? Surely the normal case should be a callback saying the memory
can
be accessed?Sure, but at least on the xe driver, that means page-table repopulation
since thehmm_range_fault()typically originated from a page-fault.Yes, I expect all hmm_range_fault()'s to be on page fault paths, and we'd like it to be as fast as we can in the CPU present case..
Jason
RFC 2/5 nouveau/dmem: HMM P2P DMA for private dev pages
In reply to Yonatan MamanFrom: Yonatan Maman <[email protected]>
Enabling Peer-to-Peer DMA (P2P DMA) access in GPU-centric applications is crucial for minimizing data transfer overhead (e.g., for RDMA use- case).
This change aims to enable that capability for Nouveau over HMM device private pages. P2P DMA for private device pages allows the GPU to directly exchange data with other devices (e.g., NICs) without needing to traverse system RAM.
To fully support Peer-to-Peer for device private pages, the following changes are made:
- Introduce struct nouveau_dmem_hmm_p2p within struct nouveau_dmem to manage BAR1 PCI P2P memory. p2p_start_addr holds the virtual address allocated with pci_alloc_p2pmem(), and p2p_size represents the allocated size of the PCI P2P memory.
- nouveau_dmem_init - Ensure BAR1 accessibility and assign struct pages (PCI_P2P_PAGE) for all BAR1 pages. Introduce nouveau_alloc_bar1_pci_p2p_mem in nouveau_dmem to expose BAR1 for use as P2P memory via pci_p2pdma_add_resource and implement static allocation and assignment of struct pages using pci_alloc_p2pmem. This function will be called from nouveau_dmem_init, and failure triggers a warning message instead of driver failure.
- nouveau_dmem_fini - Ensure BAR1 PCI P2P memory is properly destroyed during driver cleanup. Introduce nouveau_destroy_bar1_pci_p2p_mem to handle freeing of PCI P2P memory associated with Nouveau BAR1. Modify nouveau_dmem_fini to call nouveau_destroy_bar1_pci_p2p_mem.
- Implement Nouveau p2p_page callback function - Implement BAR1 mapping for the chunk using io_mem_reserve if no mapping exists. Retrieve the pre-allocated P2P virtual address and size from hmm_p2p. Calculate the page offset within BAR1 and return the corresponding P2P page.
--- drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++ 1 file changed, 110 insertions(+)
drivers/gpu/drm/nouveau/nouveau_dmem.c +109 -0
@@ -40,6 +40,9 @@#include <linux/hmm.h> #include <linux/memremap.h> #include <linux/migrate.h> +#include <linux/pci-p2pdma.h> +#include <nvkm/core/pci.h> /* * FIXME: this is ugly right now we are using TTM to allocate vram and we pin
@@ -77,9 +80,15 @@ struct nouveau_dmem_migrate {struct nouveau_channel *chan; }; +struct nouveau_dmem_hmm_p2p { +size_t p2p_size; +void *p2p_start_addr; +}; +struct nouveau_dmem {struct nouveau_drm *drm;struct nouveau_dmem_migrate migrate; +struct nouveau_dmem_hmm_p2p hmm_p2p;struct list_head chunks;struct mutex mutex;struct page *free_pages;
@@ -158,6 +167,60 @@ static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage,return 0 ; } +static int nouveau_dmem_bar1_mapping (struct nouveau_bo *nvbo, +unsigned long long *bus_addr) +{ +int ret; +struct ttm_resource *mem = nvbo->bo.resource; + +if (mem->bus.offset) { + *bus_addr = mem->bus.offset; +return 0 ; + } + +if (PFN_UP (nvbo->bo.base.size) >PFN_UP (nvbo->bo.resource->size)) +return -EINVAL ; + + ret =ttm_bo_reserve (&nvbo->bo, false, false,NULL ); +if (ret) +return ret; + + ret = nvbo->bo.bdev->funcs->io_mem_reserve (nvbo->bo.bdev, mem); + *bus_addr = mem->bus.offset; + +ttm_bo_unreserve (&nvbo->bo); +return ret; +} + +static int nouveau_dmem_get_dma_pfn (struct page *private_page, +unsigned long *dma_pfn) +{ +int ret; +unsigned long long offset_in_chunk; +unsigned long long chunk_bus_addr; +unsigned long long bar1_base_addr; +struct nouveau_drm *drm =page_to_drm (private_page); +struct nouveau_bo *nvbo =nouveau_page_to_chunk (private_page)->bo; +struct nvkm_device *nv_device =nvxx_device (drm); +size_t p2p_size = drm->dmem->hmm_p2p.p2p_size; + + bar1_base_addr = nv_device->func->resource_addr (nv_device,1 ); + offset_in_chunk = + (page_to_pfn (private_page) <<PAGE_SHIFT ) - +nouveau_page_to_chunk (private_page)->pagemap.range.start; + + ret =nouveau_dmem_bar1_mapping (nvbo, &chunk_bus_addr); +if (ret) +return ret; + + *dma_pfn = chunk_bus_addr + offset_in_chunk; +if (!p2p_size || *dma_pfn > bar1_base_addr + p2p_size || + *dma_pfn < bar1_base_addr) +return -ENOMEM ; + +return 0 ; +} +static vm_fault_t nouveau_dmem_migrate_to_ram (struct vm_fault *vmf) {struct nouveau_drm *drm =page_to_drm (vmf->page);
@@ -221,6 +284,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf)static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = { .page_free = nouveau_dmem_page_free, .migrate_to_ram = nouveau_dmem_migrate_to_ram, + .get_dma_pfn_for_device = nouveau_dmem_get_dma_pfn, }; static int
@@ -413,14 +477,31 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk)kvfree (dma_addrs); } +static void nouveau_destroy_bar1_pci_p2p_mem (struct nouveau_drm *drm, +struct pci_dev *pdev, +void *p2p_start_addr, +size_t p2p_size) +{ +if (p2p_size) +pci_free_p2pmem (pdev, p2p_start_addr, p2p_size); + +NV_INFO (drm,"PCI P2P memory freed(%p)\n" , p2p_start_addr); +} +void nouveau_dmem_fini (struct nouveau_drm *drm) {struct nouveau_dmem_chunk *chunk, *tmp; +struct nvkm_device *nv_device =nvxx_device (drm);if (drm->dmem ==NULL )return ; +nouveau_destroy_bar1_pci_p2p_mem (drm, + nv_device->func->pci (nv_device)->pdev, + drm->dmem->hmm_p2p.p2p_start_addr, + drm->dmem->hmm_p2p.p2p_size); +mutex_lock (&drm->dmem->mutex);list_for_each_entry_safe (chunk, tmp, &drm->dmem->chunks, list) {
@@ -586,10 +667,28 @@ nouveau_dmem_migrate_init(struct nouveau_drm *drm)return -ENODEV ; } +static int nouveau_alloc_bar1_pci_p2p_mem (struct nouveau_drm *drm, +struct pci_dev *pdev,size_t size, +void **pp2p_start_addr) +{ +int ret; + + ret =pci_p2pdma_add_resource (pdev,1 , size,0 ); +if (ret) +return ret; + + *pp2p_start_addr =pci_alloc_p2pmem (pdev, size); + +NV_INFO (drm,"PCI P2P memory allocated(%p)\n" , *pp2p_start_addr); +return 0 ; +} +void nouveau_dmem_init (struct nouveau_drm *drm) {int ret; +struct nvkm_device *nv_device =nvxx_device (drm); +size_t bar1_size;/* This only make sense on PASCAL or newer */ if (drm->client.device.info.family <NV_DEVICE_INFO_V0_PASCAL )
@@ -610,6 +709,17 @@ nouveau_dmem_init(struct nouveau_drm *drm)kfree (drm->dmem); drm->dmem =NULL ; } + +/* Expose BAR1 for HMM P2P Memory */ + bar1_size = nv_device->func->resource_size (nv_device,1 ); + ret =nouveau_alloc_bar1_pci_p2p_mem (drm, + nv_device->func->pci (nv_device)->pdev, + bar1_size, + &drm->dmem->hmm_p2p.p2p_start_addr); + drm->dmem->hmm_p2p.p2p_size = (ret) ?0 : bar1_size; +if (ret) +NV_WARN (drm, +"PCI P2P memory allocation failed, HMM P2P won't be supported\n" ); } static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm,
Signature · 1 line
2.34.1
RFC 3/5 IB/core: P2P DMA for device private pages
In reply to Yonatan MamanFrom: Yonatan Maman <[email protected]>
Add Peer-to-Peer (P2P) DMA request for hmm_range_fault calling, utilizing capabilities introduced in mm/hmm. By setting range.default_flags to HMM_PFN_REQ_FAULT | HMM_PFN_REQ_TRY_P2P, HMM attempts to initiate P2P DMA connections for device private pages (instead of page fault handling).
This enhancement utilizes P2P DMA to reduce performance overhead during data migration between devices (e.g., GPU) and system memory, providing performance benefits for GPU-centric applications that utilize RDMA and device private pages.
--- drivers/infiniband/core/umem_odp.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/infiniband/core/umem_odp.c +4 -0
@@ -332,6 +332,10 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, range.default_flags |=HMM_PFN_REQ_WRITE ; } +if (access_mask &HMM_PFN_ALLOW_P2P ) + range.default_flags |=HMM_PFN_ALLOW_P2P ; + + range.pfn_flags_mask =HMM_PFN_ALLOW_P2P ; range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]); timeout = jiffies +msecs_to_jiffies (HMM_RANGE_DEFAULT_TIMEOUT );
Signature · 1 line
2.34.1
RFC 4/5 RDMA/mlx5: Add fallback for P2P DMA errors
In reply to Yonatan MamanFrom: Yonatan Maman <[email protected]>
Handle P2P DMA mapping errors when the transaction requires traversing an inaccessible host bridge that is not in the allowlist:
- In populate_mtt, if a P2P mapping fails, the HMM_PFN_ALLOW_P2P flag is cleared only for the PFNs that returned a mapping error.
- In pagefault_real_mr, if a P2P mapping error occurs, the mapping is retried with the HMM_PFN_ALLOW_P2P flag only for the PFNs that didn't fail, ensuring a fallback to standard DMA(host memory) for the rest,
if possible.
--- drivers/infiniband/hw/mlx5/odp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +21 -3
@@ -169,6 +169,7 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries,struct pci_p2pdma_map_state p2pdma_state = {};struct ib_device *dev = odp->umem.ibdev;size_t i; +int ret =0 ;if (flags &MLX5_IB_UPD_XLT_ZAP )return 0 ;
@@ -184,8 +185,11 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries, dma_addr =hmm_dma_map_pfn (dev->dma_device, &odp->map, start + i, &p2pdma_state); -if (ib_dma_mapping_error (dev, dma_addr)) -return -EFAULT ; +if (ib_dma_mapping_error (dev, dma_addr)) { + odp->map.pfn_list[start + i] &= ~(HMM_PFN_ALLOW_P2P ); + ret = -EFAULT ; +continue ; + } dma_addr |=MLX5_IB_MTT_READ ;if ((pfn &HMM_PFN_WRITE ) && !downgrade)
@@ -194,7 +198,7 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries, pas[i] =cpu_to_be64 (dma_addr); odp->npages++; } -return 0 ; +return ret; } int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
@@ -696,6 +700,10 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp,if (odp->umem.writable && !downgrade) access_mask |=HMM_PFN_WRITE ; +/* +* try fault with HMM_PFN_ALLOW_P2P flag +*/ + access_mask |=HMM_PFN_ALLOW_P2P ; np =ib_umem_odp_map_dma_and_lock (odp, user_va, bcnt, access_mask, fault);if (np <0 )return np;
@@ -705,6 +713,16 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp,* ib_umem_odp_map_dma_and_lock already checks this. */ ret =mlx5r_umr_update_xlt (mr, start_idx, np, page_shift, xlt_flags); +if (ret == -EFAULT ) { +/* +* Indicate P2P Mapping Error, retry with no HMM_PFN_ALLOW_P2P +*/ + access_mask &= ~HMM_PFN_ALLOW_P2P ; + np =ib_umem_odp_map_dma_and_lock (odp, user_va, bcnt, access_mask, fault); +if (np <0 ) +return np; + ret =mlx5r_umr_update_xlt (mr, start_idx, np, page_shift, xlt_flags); + }mutex_unlock (&odp->umem_mutex);if (ret <0 ) {
Signature · 1 line
2.34.1
RFC 5/5 RDMA/mlx5: Enabling ATS for ODP memory
In reply to Yonatan MamanFrom: Yonatan Maman <[email protected]>
ATS (Address Translation Services) mainly utilized to optimize PCI Peer-to-Peer transfers and prevent bus failures. This change employed ATS usage for ODP memory, to optimize DMA P2P for ODP memory. (e.g DMA P2P for private device pages - ODP memory).
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -3
@@ -1705,9 +1705,9 @@ static inline bool rt_supported(int ts_cap)static inline bool mlx5_umem_needs_ats (struct mlx5_ib_dev *dev,struct ib_umem *umem,int access_flags) { -if (!MLX5_CAP_GEN (dev->mdev, ats) || !umem->is_dmabuf) -return false; -return access_flags &IB_ACCESS_RELAXED_ORDERING ; +if (MLX5_CAP_GEN (dev->mdev, ats) && (umem->is_dmabuf || umem->is_odp)) +return access_flags &IB_ACCESS_RELAXED_ORDERING ; +return false; } int set_roce_addr(struct mlx5_ib_dev *dev, u32 port_num,
Signature · 1 line
2.34.1