Skip to contentSkip to conversation

drm/amdgpu: Add kernel parameter to force no xgmi

PATCH In review Alex Sierra 4 participants 8 messages Oct 28, 2020 – Oct 29, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Add kernel parameter to force no xgmi

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
 extern int amdgpu_mes;
 extern int amdgpu_noretry;
 extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
 #ifdef CONFIG_HSA_AMD
 extern int sched_policy;
 extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
index 1fe850e0a94d..0a5d97a84017 100644
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	if (adev->gmc.xgmi.num_physical_nodes > 1)
+	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)
 		amdgpu_xgmi_add_device(adev);
 	amdgpu_amdkfd_device_init(adev);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +9 -0
index 4b78ecfd35f7..22485067cf31 100644
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = 0;
 int amdgpu_reset_method = -1; /* auto */
 int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
 
 struct amdgpu_mgpu_info mgpu_info = {
 	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
 MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
 module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
 
+/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
+
 /**
  * DOC: si_support (int)
  * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

Am 28.10.20 um 15:55 schrieb Alex Sierra:

Quoted 2 lines

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

Quoted 25 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
  3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
  extern int amdgpu_mes;
  extern int amdgpu_noretry;
  extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
  #ifdef CONFIG_HSA_AMD
  extern int sched_policy;
  extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct amdgpu_device *adev) · line 2260

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
  	if (r)
  		goto init_failed;
 
-	if (adev->gmc.xgmi.num_physical_nodes > 1)
+	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)

Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.

Apart from that sounds like a good idea in general.

Christian.

Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
  		amdgpu_xgmi_add_device(adev);
  	amdgpu_amdkfd_device_init(adev);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +2 -0
index 4b78ecfd35f7..22485067cf31 100644
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
  int amdgpu_tmz = 0;
  int amdgpu_reset_method = -1; /* auto */
  int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
 
  struct amdgpu_mgpu_info mgpu_info = {
  	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
  module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
 
+/**

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 536

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +7 -0
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
+
  /**
   * DOC: si_support (int)
   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw
Quoted 7 lines · Christian König

Am 28.10.20 um 15:55 schrieb Alex Sierra:

Quoted 2 lines

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

Ack

Regards,
Alex Sierra

Quoted 26 lines
Quoted 26 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++

  3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
   extern int amdgpu_mes;
   extern int amdgpu_noretry;
   extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
   #ifdef CONFIG_HSA_AMD
   extern int sched_policy;
   extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644
Quoted 8 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct 
 amdgpu_device *adev)
       if (r)
           goto init_failed;
   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
+    if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)

Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.

What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.

Regards,
Alex Sierra

Quoted 25 lines

Apart from that sounds like a good idea in general.

Christian.

Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
           amdgpu_xgmi_add_device(adev);
       amdgpu_amdkfd_device_init(adev);

  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 4b78ecfd35f7..22485067cf31 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
   int amdgpu_tmz = 0;

  int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
    struct amdgpu_mgpu_info mgpu_info = { .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -522,6 +523,14 @@ module_param_named(ras_enable, 
 amdgpu_ras_enable, int, 0444);
   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);

  +/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).

Quoted 9 lines
Quoted 8 lines

+ * Default is 0 (disabled).
+ */

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +3 -0
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
+

  /**
   * DOC: si_support (int)
   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

[AMD Public Use]

Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.

Regards,
Alex Sierra

Quoted 110 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

Quoted 7 lines · Christian König

Am 28.10.20 um 15:55 schrieb Alex Sierra:

Quoted 2 lines

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

Ack

Regards,
Alex Sierra

Quoted 35 lines
Quoted 29 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++

  3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
   extern int amdgpu_mes;
   extern int amdgpu_noretry;
   extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
   #ifdef CONFIG_HSA_AMD
   extern int sched_policy;
   extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
 amdgpu_device *adev)
       if (r)
           goto init_failed;
   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
+    if (!amdgpu_force_no_xgmi && adev-

gmc.xgmi.num_physical_nodes >

Quoted 4 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262

Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
+1)

Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.

What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.

Regards,
Alex Sierra

Quoted 14 lines

Apart from that sounds like a good idea in general.

Christian.

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
           amdgpu_xgmi_add_device(adev);

      amdgpu_amdkfd_device_init(adev);
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 4b78ecfd35f7..22485067cf31 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
   int amdgpu_tmz = 0;
Quoted 8 lines
Quoted 8 lines

  int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
    struct amdgpu_mgpu_info mgpu_info = { .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -522,6 +523,14 @@ module_param_named(ras_enable,
 amdgpu_ras_enable, int, 0444);

  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable

(default

Quoted 8 lines
Quoted 8 lines

0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi

interface");

Quoted 1 line
Quoted 1 line

+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,

0600);

Quoted 6 lines
Quoted 5 lines

+
  /**
   * DOC: si_support (int)
   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

Quoted 10 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata= s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw
Quoted 3 lines · Alejandro (Alex)

[AMD Public Use]

Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.

No problem, but if you do add it elsewhere, please use something more specific, like

amdgpu_xgmi_p2p

as the (positive-controlled) flag, since more generic flags could be added later, to control a more encompassing behaviour.

Regards,
Luben

Quoted 124 lines

Regards,
Alex Sierra

Quoted 110 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

Quoted 7 lines · Christian König

Am 28.10.20 um 15:55 schrieb Alex Sierra:

Quoted 2 lines

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

Ack

Regards,
Alex Sierra

Quoted 35 lines
Quoted 29 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++

  3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
   extern int amdgpu_mes;
   extern int amdgpu_noretry;
   extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
   #ifdef CONFIG_HSA_AMD
   extern int sched_policy;
   extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
 amdgpu_device *adev)
       if (r)
           goto init_failed;
   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
+    if (!amdgpu_force_no_xgmi && adev-

gmc.xgmi.num_physical_nodes >

Quoted 4 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262

Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
+1)

Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.

What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.

Regards,
Alex Sierra

Quoted 14 lines

Apart from that sounds like a good idea in general.

Christian.

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
           amdgpu_xgmi_add_device(adev);

      amdgpu_amdkfd_device_init(adev);
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 4b78ecfd35f7..22485067cf31 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
   int amdgpu_tmz = 0;
Quoted 8 lines
Quoted 8 lines

  int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
    struct amdgpu_mgpu_info mgpu_info = { .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -522,6 +523,14 @@ module_param_named(ras_enable,
 amdgpu_ras_enable, int, 0444);

  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable

(default

Quoted 8 lines
Quoted 8 lines

0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi

interface");

Quoted 1 line
Quoted 1 line

+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,

0600);

Quoted 6 lines
Quoted 5 lines

+
  /**
   * DOC: si_support (int)
   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

Quoted 10 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata= s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0

Quoted 5 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=04%7C01%7Cluben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1uU%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw

[AMD Public Use]

Will do Luben. Thanks for the recommendation.

Quoted 167 lines

-----Original Message-----
From: Tuikov, Luben <[email protected]>
Sent: Thursday, October 29, 2020 12:15 PM
To: Sierra Guiza, Alejandro (Alex) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

Quoted 3 lines · Alejandro (Alex)

[AMD Public Use]

Please ignore this patch, it should be in a different branch. As PCIe p2p is

not supported in upstream.

No problem, but if you do add it elsewhere, please use something more specific, like

amdgpu_xgmi_p2p

as the (positive-controlled) flag, since more generic flags could be added later, to control a more encompassing behaviour.

Regards,
Luben

Quoted 108 lines

Regards,
Alex Sierra

Quoted 104 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

Quoted 7 lines · Christian König

Am 28.10.20 um 15:55 schrieb Alex Sierra:

Quoted 2 lines

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

Ack

Regards,
Alex Sierra

Quoted 35 lines
Quoted 29 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++

  3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
   extern int amdgpu_mes;
   extern int amdgpu_noretry;
   extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
   #ifdef CONFIG_HSA_AMD
   extern int sched_policy;
   extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
 amdgpu_device *adev)
       if (r)
           goto init_failed;
   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
+    if (!amdgpu_force_no_xgmi && adev-

gmc.xgmi.num_physical_nodes >

Quoted 4 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262

Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0
+1)

Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.

What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.

Regards,
Alex Sierra

Quoted 14 lines

Apart from that sounds like a good idea in general.

Christian.

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
           amdgpu_xgmi_add_device(adev);

      amdgpu_amdkfd_device_init(adev);
  diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 4b78ecfd35f7..22485067cf31 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
   int amdgpu_tmz = 0;
Quoted 8 lines
Quoted 8 lines

  int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
    struct amdgpu_mgpu_info mgpu_info = { .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
@@ -522,6 +523,14 @@ module_param_named(ras_enable,
 amdgpu_ras_enable, int, 0444);

  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable

(default

Quoted 8 lines
Quoted 8 lines

0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi

interface");

Quoted 1 line
Quoted 1 line

+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi,

int,

Quoted 6 lines
Quoted 6 lines

0600);

Quoted 5 lines
Quoted 5 lines

+
  /**
   * DOC: si_support (int)
   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is

enabled,

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

_______________________________________________
amd-gfx mailing list
[email protected]

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.

Quoted 2 lines
Quoted 2 lines

freedesktop.org%2Fmailman%2Flistinfo%2Famd-

gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865

Quoted 1 line
Quoted 1 line

91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63

Quoted 1 line
Quoted 1 line

7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD

Quoted 1 line
Quoted 1 line

AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=

Quoted 1 line
Quoted 1 line

s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0

Quoted 5 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.freedesktop.org%2Fmailman%2Flistinfo%2Famd-

gfx&amp;data=04%7C01%7Clu

Quoted 1 line

ben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd896 1fe488

Quoted 1 line
Quoted 10 lines

4e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7 CTWFpbGZs

Quoted 1 line

b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D

Quoted 1 line

%7C1000&amp;sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1 uU%3D&amp

Quoted 2 lines

;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw
In reply to Alex Sierra
Quoted 48 lines · Alex Sierra

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
 extern int amdgpu_mes;
 extern int amdgpu_noretry;
 extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
 #ifdef CONFIG_HSA_AMD
 extern int sched_policy;
 extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
index 1fe850e0a94d..0a5d97a84017 100644
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	if (adev->gmc.xgmi.num_physical_nodes > 1)
+	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)
 		amdgpu_xgmi_add_device(adev);
 	amdgpu_amdkfd_device_init(adev);
 
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
index 4b78ecfd35f7..22485067cf31 100644
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = 0;
 int amdgpu_reset_method = -1; /* auto */
 int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
 
 struct amdgpu_mgpu_info mgpu_info = {
 	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 528

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +3 -0
@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
 MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
 module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
 
+/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).

How about using human English here? Perhaps something like,

"Forces not to use" ==> "Disables the use of the XGMI interface ..."

Or, if you use the suggestion Christian posted, you could say something like,

"Enable XGMI for P2P transactions."

Regards,
Luben

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 536

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +5 -0
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
+
 /**
  * DOC: si_support (int)
  * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#7 lore raw

[AMD Official Use Only - Internal Distribution Only]

Quoted 72 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Luben Tuikov
Sent: Wednesday, October 28, 2020 11:12 AM
To: Sierra Guiza, Alejandro (Alex) <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[CAUTION: External Email]

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct

Quoted 28 lines · Alex Sierra

By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -0
index ba65d4f2ab67..3645f00e9f61 100644
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;  extern int

amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type;

drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
+extern int amdgpu_force_no_xgmi;
 #ifdef CONFIG_HSA_AMD
 extern int sched_policy;
 extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 1fe850e0a94d..0a5d97a84017 100644
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct

amdgpu_device *adev)

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c int amdgpu_force_asic_type = -1; int amdgpu_tmz

Quoted 16 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
      if (r)
              goto init_failed;
 
-     if (adev->gmc.xgmi.num_physical_nodes > 1)
+     if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes
 
+ 1)
              amdgpu_xgmi_add_device(adev);
      amdgpu_amdkfd_device_init(adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0 -0
index 4b78ecfd35f7..22485067cf31 100644
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;  int amdgpu_tmz

=

drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable,

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1 -0
 0;  int amdgpu_reset_method = -1; /* auto */  int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
 
 struct amdgpu_mgpu_info mgpu_info = {
      .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -522,6 +523,14 @@ module_param_named(ras_enable,

amdgpu_ras_enable,

Quoted 7 lines

int, 0444); MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);

+/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).

Quoted 7 lines

How about using human English here? Perhaps something like,

"Forces not to use" ==> "Disables the use of the XGMI interface ..."

Or, if you use the suggestion Christian posted, you could say something like,

"Enable XGMI for P2P transactions."

I second the use of enable. The fact that we can prevent xGMI init means we enable it by default.

Thanks,

Jon

Quoted 17 lines

Regards,
Luben

Quoted 3 lines

+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi

interface");

Quoted 1 line

+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,

0600);

Quoted 6 lines

+
/**
* DOC: si_support (int)
* Set SI support driver. This parameter works after set config
CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

Quoted 11 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&amp;data=04%7C01%7Cjonathan.kim%40amd.com%7C7e0b2ac9a2ac4c7 8353008d87b53e1b3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7 C637394947518556523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sda ta=ve9KxUmOXYYYS8MQJBkzW7V%2FRxCORSVWxlUzhFRYxzw%3D&amp;re served=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx