Skip to contentSkip to conversation

drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

PATCH In review Ramesh Errabolu 2 participants 3 messages May 31, 2022 – Jun 3, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

Extend current kernel config requirements of amdgpu by adding config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer communication, in both data and control planes, among AMD GPU devices that are connected via PCIe and have large BAR vBIOS

---
 drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/gpu/drm/amd/amdkfd/Kconfig +8 -0
index 8cc0a76ddf9f..26614f5f20ea 100644
@@ -25,3 +25,11 @@ config HSA_AMD_SVM
 	  preemptions and one based on page faults. To enable page fault
 	  based memory management on most GFXv9 GPUs, set the module
 	  parameter amdgpu.noretry=0.
+
+config HSA_AMD_P2P
+	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
+	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY 
+	help
+	  Enable this if you want to access AMD GPU peer devices, in both data
+	  and control planes, that are connected via PCIe and have large BAR vBIOS
+
Signature · 1 line
2.35.1
maintainer
#1 lore raw

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:

Quoted 15 lines

Extend current kernel config requirements of amdgpu by adding config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer communication, in both data and control planes, among AMD GPU devices that are connected via PCIe and have large BAR vBIOS

---
  drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/gpu/drm/amd/amdkfd/Kconfig +0 -0
index 8cc0a76ddf9f..26614f5f20ea 100644

drivers/gpu/drm/amd/amdkfd/Kconfig config HSA_AMD_SVM · line 34

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/Kconfig +7 -0
@@ -25,3 +25,11 @@ config HSA_AMD_SVM
  	  preemptions and one based on page faults. To enable page fault
  	  based memory management on most GFXv9 GPUs, set the module
  	  parameter amdgpu.noretry=0.
+
+config HSA_AMD_P2P
+	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
+	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
+	help
+	  Enable this if you want to access AMD GPU peer devices, in both data
+	  and control planes, that are connected via PCIe and have large BAR vBIOS

I have not seen the terms "data plane" and "control plane" used in the context of GPUs. As far as I can tell, this terminology is more common in the context network routing. I think it could cause confusion to introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems to address someone writing an application. This help message is meant for users and admins building a kernel, who may want to run compute applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids uncommon terminology and addresses the concerns of users rather than application developers:

Enable peer-to-peer (P2P) communication between AMD GPUs over the PCIe bus. This can improve performance of multi-GPU compute applications and libraries by enabling GPUs to access data directly in peer GPUs' memory without intermediate copies in system memory.

This P2P feature is only enabled on compatible chipsets, and between GPUs with large memory BARs that expose the entire VRAM in PCI bus address space within the physical address limits of the GPUs.

Regards,
  Felix

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/Kconfig +1 -0
+
#2 lore raw

[AMD Official Use Only - General]

Update the both the description and commit message per review comments. Will post new patch shortly.

Regards,
Ramesh

-----Original Message-----
From: Kuehling, Felix <[email protected]>
Sent: Wednesday, June 1, 2022 9:15 PM
To: Errabolu, Ramesh <[email protected]>; [email protected]
Subject: Re: [PATCH 1/3] drm/amdkfd: Define config HSA_AMD_P2P to support peer-to-peer

Am 2022-05-31 um 13:02 schrieb Ramesh Errabolu:

Quoted 17 lines

Extend current kernel config requirements of amdgpu by adding config HSA_AMD_P2P. Enabling HSA_AMD_P2P is REQUIRED to support peer-to-peer communication, in both data and control planes, among AMD GPU devices that are connected via PCIe and have large BAR vBIOS

---
  drivers/gpu/drm/amd/amdkfd/Kconfig | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/gpu/drm/amd/amdkfd/Kconfig +0 -0
index 8cc0a76ddf9f..26614f5f20ea 100644
@@ -25,3 +25,11 @@ config HSA_AMD_SVM
  	  preemptions and one based on page faults. To enable page fault

drivers/gpu/drm/amd/amdkfd/Kconfig config HSA_AMD_SVM · line 35

Quoted 10 lines
drivers/gpu/drm/amd/amdkfd/Kconfig +8 -0
  	  based memory management on most GFXv9 GPUs, set the module
  	  parameter amdgpu.noretry=0.
+
+config HSA_AMD_P2P
+	bool "HSA kernel driver support for peer-to-peer for AMD GPU devices"
+	depends on HSA_AMD && PCI_P2PDMA && DMABUF_MOVE_NOTIFY
+	help
+	  Enable this if you want to access AMD GPU peer devices, in both data
+	  and control planes, that are connected via PCIe and have large BAR 
+vBIOS

I have not seen the terms "data plane" and "control plane" used in the context of GPUs. As far as I can tell, this terminology is more common in the context network routing. I think it could cause confusion to introduce these terms without an explanation to users.

The sentence "... if you want to access AMD GPU peer devices ..." seems to address someone writing an application. This help message is meant for users and admins building a kernel, who may want to run compute applications, not for compute application developers.

I would also not mention large-BAR VBIOSes because the BAR can often be resized even with a small-BAR VBIOS.

Therefore I would recommend an alternative text here that avoids uncommon terminology and addresses the concerns of users rather than application developers:

Enable peer-to-peer (P2P) communication between AMD GPUs over the PCIe bus. This can improve performance of multi-GPU compute applications and libraries by enabling GPUs to access data directly in peer GPUs' memory without intermediate copies in system memory.

This P2P feature is only enabled on compatible chipsets, and between GPUs with large memory BARs that expose the entire VRAM in PCI bus address space within the physical address limits of the GPUs.

Regards,
  Felix

Quoted 1 line

+