drm/amdgpu: Add kernel parameter to force no xgmi
Subsystems (2)
PATCH 1/1 drm/amdgpu: Add kernel parameter to force no xgmi
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ 3 files changed, 11 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;extern int amdgpu_mes;extern int amdgpu_noretry;extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi;#ifdef CONFIG_HSA_AMD extern int sched_policy;extern bool debug_evictions;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -1
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)if (r) goto init_failed; -if (adev->gmc.xgmi.num_physical_nodes >1 ) +if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes >1 )amdgpu_xgmi_add_device (adev);amdgpu_amdkfd_device_init (adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +9 -0
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;int amdgpu_tmz =0 ;int amdgpu_reset_method =-1 ;/* auto */ int amdgpu_num_kcq =-1 ; +int amdgpu_force_no_xgmi =0 ;struct amdgpu_mgpu_info mgpu_info = { .mutex =__MUTEX_INITIALIZER (mgpu_info.mutex),
@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC (ras_mask,"Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1" );module_param_named (ras_mask, amdgpu_ras_mask, uint,0444 ); +/** +* DOC: force_no_xgmi (uint) +* Forces not to use xgmi interface (0 = disable, 1 = enable). +* Default is 0 (disabled). +*/ +MODULE_PARM_DESC (force_no_xgmi,"Force not to use xgmi interface" ); +module_param_named (force_no_xgmi, amdgpu_force_no_xgmi, int,0600 ); +/** * DOC: si_support (int) * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Am 28.10.20 um 15:55 schrieb Alex Sierra:
Quoted 2 lines
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Better name that amdgpu_xgmi with a default value of enabled.
Or maybe add another bit value for amdgpu_vm_debug instead.
Quoted 25 lines
Signed-off-by Alex Sierra <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ 3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery;extern int amdgpu_mes;extern int amdgpu_noretry;extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi;#ifdef CONFIG_HSA_AMD extern int sched_policy;extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct amdgpu_device *adev) · line 2260
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)if (r) goto init_failed; -if (adev->gmc.xgmi.num_physical_nodes >1 ) +if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes >1 )
Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.
Apart from that sounds like a good idea in general.
Christian.
Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0amdgpu_xgmi_add_device (adev);amdgpu_amdkfd_device_init (adev);drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +2
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;int amdgpu_tmz =0 ;int amdgpu_reset_method =-1 ;/* auto */ int amdgpu_num_kcq =-1 ; +int amdgpu_force_no_xgmi =0 ;struct amdgpu_mgpu_info mgpu_info = { .mutex =__MUTEX_INITIALIZER (mgpu_info.mutex),@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC (ras_mask,"Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1" );module_param_named (ras_mask, amdgpu_ras_mask, uint,0444 ); +/**
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 536
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +7
-0+* DOC: force_no_xgmi (uint) +* Forces not to use xgmi interface (0 = disable, 1 = enable). +* Default is 0 (disabled). +*/ +MODULE_PARM_DESC (force_no_xgmi,"Force not to use xgmi interface" ); +module_param_named (force_no_xgmi, amdgpu_force_no_xgmi, int,0600 ); +/** * DOC: si_support (int) * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 7 lines · Christian König
Am 28.10.20 um 15:55 schrieb Alex Sierra:
Quoted 2 lines
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Better name that
amdgpu_xgmiwith a default value of enabled.Or maybe add another bit value for
amdgpu_vm_debuginstead.
Ack
Regards,
Alex Sierra
Quoted 26 lines
Quoted 26 lines
Signed-off-by Alex Sierra <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi; #ifdef CONFIG_HSA_AMD extern int sched_policy; extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) if (r) goto init_failed; - if (adev->gmc.xgmi.num_physical_nodes > 1) + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.
What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.
Regards,
Alex Sierra
Quoted 25 lines
Apart from that sounds like a good idea in general.
Christian.
Quoted 20 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0amdgpu_xgmi_add_device(adev); amdgpu_amdkfd_device_init(adev);diff --git
a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cindex4b78ecfd35f7..22485067cf31100644drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz = 0;int
amdgpu_reset_method= -1; /* auto */ intamdgpu_num_kcq= -1;
+intamdgpu_force_no_xgmi= 0;
structamdgpu_mgpu_infomgpu_info= { .mutex =__MUTEX_INITIALIZER(mgpu_info.mutex),drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);+/**
+ * DOC:force_no_xgmi(uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
Quoted 9 lines
Quoted 8 lines
+ * Default is 0 (disabled).
+ */drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +3
-0+MODULE_PARM_DESC (force_no_xgmi,"Force not to use xgmi interface" ); +module_param_named (force_no_xgmi, amdgpu_force_no_xgmi, int,0600 ); +/**
* DOC:si_support(int)
* Set SI support driver. This parameter works after set configCONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[AMD Public Use]
Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.
Regards,
Alex Sierra
Quoted 110 lines
-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmiQuoted 7 lines · Christian König
Am 28.10.20 um 15:55 schrieb Alex Sierra:
Quoted 2 lines
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Better name that
amdgpu_xgmiwith a default value of enabled.Or maybe add another bit value for
amdgpu_vm_debuginstead.Ack
Regards,
Alex SierraQuoted 35 lines
Quoted 29 lines
Signed-off-by Alex Sierra <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi; #ifdef CONFIG_HSA_AMD extern int sched_policy; extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) if (r) goto init_failed; - if (adev->gmc.xgmi.num_physical_nodes > 1) + if (!amdgpu_force_no_xgmi && adev-gmc.xgmi.num_physical_nodes >
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0+1)Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.
What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.
Regards,
Alex SierraQuoted 14 lines
Apart from that sounds like a good idea in general.
Christian.
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0amdgpu_xgmi_add_device(adev);
amdgpu_amdkfd_device_init(adev);
diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cindex4b78ecfd35f7..22485067cf31100644drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz = 0;Quoted 8 lines
Quoted 8 lines
int
amdgpu_reset_method= -1; /* auto */ intamdgpu_num_kcq= -1;
+intamdgpu_force_no_xgmi= 0;
structamdgpu_mgpu_infomgpu_info= { .mutex =__MUTEX_INITIALIZER(mgpu_info.mutex),drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
(default
Quoted 8 lines
Quoted 8 lines
0xffffffff), only valid whenras_enable== 1");module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC:force_no_xgmi(uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmiinterface");
Quoted 1 line
Quoted 1 line
+module_param_named(force_no_xgmi,
amdgpu_force_no_xgmi, int,0600);
Quoted 6 lines
Quoted 5 lines
+
/**
* DOC:si_support(int)
* Set SI support driver. This parameter works after set configCONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
Quoted 10 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata= s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 3 lines · Alejandro (Alex)
[AMD Public Use]
Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.
No problem, but if you do add it elsewhere, please use something more specific, like
amdgpu_xgmi_p2p
as the (positive-controlled) flag, since more generic flags could be added later, to control a more encompassing behaviour.
Regards,
Luben
Quoted 124 lines
Regards,
Alex SierraQuoted 110 lines
-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmiQuoted 7 lines · Christian König
Am 28.10.20 um 15:55 schrieb Alex Sierra:
Quoted 2 lines
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Better name that
amdgpu_xgmiwith a default value of enabled.Or maybe add another bit value for
amdgpu_vm_debuginstead.Ack
Regards,
Alex SierraQuoted 35 lines
Quoted 29 lines
Signed-off-by Alex Sierra <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi; #ifdef CONFIG_HSA_AMD extern int sched_policy; extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) if (r) goto init_failed; - if (adev->gmc.xgmi.num_physical_nodes > 1) + if (!amdgpu_force_no_xgmi && adev-gmc.xgmi.num_physical_nodes >
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0+1)Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.
What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.
Regards,
Alex SierraQuoted 14 lines
Apart from that sounds like a good idea in general.
Christian.
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0amdgpu_xgmi_add_device(adev);
amdgpu_amdkfd_device_init(adev);
diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cindex4b78ecfd35f7..22485067cf31100644drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz = 0;Quoted 8 lines
Quoted 8 lines
int
amdgpu_reset_method= -1; /* auto */ intamdgpu_num_kcq= -1;
+intamdgpu_force_no_xgmi= 0;
structamdgpu_mgpu_infomgpu_info= { .mutex =__MUTEX_INITIALIZER(mgpu_info.mutex),drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
(default
Quoted 8 lines
Quoted 8 lines
0xffffffff), only valid whenras_enable== 1");module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC:force_no_xgmi(uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmiinterface");
Quoted 1 line
Quoted 1 line
+module_param_named(force_no_xgmi,
amdgpu_force_no_xgmi, int,0600);
Quoted 6 lines
Quoted 5 lines
+
/**
* DOC:si_support(int)
* Set SI support driver. This parameter works after set configCONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,Quoted 10 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata= s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0
Quoted 5 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Cluben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1uU%3D&reserved=0
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[AMD Public Use]
Will do Luben. Thanks for the recommendation.
Quoted 167 lines
-----Original Message-----
From: Tuikov, Luben <[email protected]>
Sent: Thursday, October 29, 2020 12:15 PM
To: Sierra Guiza, Alejandro (Alex) <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmiQuoted 3 lines · Alejandro (Alex)
[AMD Public Use]
Please ignore this patch, it should be in a different branch. As PCIe p2p is
not supported in upstream.
No problem, but if you do add it elsewhere, please use something more specific, like
amdgpu_xgmi_p2pas the (positive-controlled) flag, since more generic flags could be added later, to control a more encompassing behaviour.
Regards,
LubenQuoted 108 lines
Regards,
Alex SierraQuoted 104 lines
-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Sierra Guiza, Alejandro (Alex)
Sent: Wednesday, October 28, 2020 1:09 PM
To: Koenig, Christian <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmiQuoted 7 lines · Christian König
Am 28.10.20 um 15:55 schrieb Alex Sierra:
Quoted 2 lines
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Better name that
amdgpu_xgmiwith a default value of enabled.Or maybe add another bit value for
amdgpu_vm_debuginstead.Ack
Regards,
Alex SierraQuoted 35 lines
Quoted 29 lines
Signed-off-by Alex Sierra <[email protected]>---
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 +drivers/gpu/drm/amd/amdgpu/amdgpu_device.c| 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi; #ifdef CONFIG_HSA_AMD extern int sched_policy; extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2261
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) if (r) goto init_failed; - if (adev->gmc.xgmi.num_physical_nodes > 1) + if (!amdgpu_force_no_xgmi && adev-gmc.xgmi.num_physical_nodes >
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2262
Quoted 1 line
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0+1)Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work.
What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set.
Regards,
Alex SierraQuoted 14 lines
Apart from that sounds like a good idea in general.
Christian.
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0amdgpu_xgmi_add_device(adev);
amdgpu_amdkfd_device_init(adev);
diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.cindex4b78ecfd35f7..22485067cf31100644drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz = 0;Quoted 8 lines
Quoted 8 lines
int
amdgpu_reset_method= -1; /* auto */ intamdgpu_num_kcq= -1;
+intamdgpu_force_no_xgmi= 0;
structamdgpu_mgpu_infomgpu_info= { .mutex =__MUTEX_INITIALIZER(mgpu_info.mutex),drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
(default
Quoted 8 lines
Quoted 8 lines
0xffffffff), only valid whenras_enable== 1");module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/**
+ * DOC:force_no_xgmi(uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmiinterface");
Quoted 1 line
Quoted 1 line
+module_param_named(force_no_xgmi,
amdgpu_force_no_xgmi,int,
Quoted 6 lines
Quoted 6 lines
0600);
Quoted 5 lines
Quoted 5 lines
+
/**
* DOC:si_support(int)
* Set SI support driver. This parameter works after set configCONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver isenabled,
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
_______________________________________________
amd-gfx mailing list
[email protected]https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
Quoted 2 lines
Quoted 2 lines
freedesktop.org%2Fmailman%2Flistinfo%2Famd-
gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865
Quoted 1 line
Quoted 1 line
91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
Quoted 1 line
Quoted 1 line
7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
Quoted 1 line
Quoted 1 line
AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=
Quoted 1 line
Quoted 1 line
s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0
Quoted 5 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clu
Quoted 1 line
ben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd896 1fe488
Quoted 1 line
Quoted 10 lines
4e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7 CTWFpbGZs
Quoted 1 line
b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn 0%3D
Quoted 1 line
%7C1000&sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1 uU%3D&
Quoted 2 lines
;reserved=0
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 48 lines · Alex Sierra
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Signed-off-by Alex Sierra <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ 3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery;extern int amdgpu_mes;extern int amdgpu_noretry;extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi;#ifdef CONFIG_HSA_AMD extern int sched_policy;extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-1@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)if (r) goto init_failed; -if (adev->gmc.xgmi.num_physical_nodes >1 ) +if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes >1 )amdgpu_xgmi_add_device (adev);amdgpu_amdkfd_device_init (adev);drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;int amdgpu_tmz =0 ;int amdgpu_reset_method =-1 ;/* auto */ int amdgpu_num_kcq =-1 ; +int amdgpu_force_no_xgmi =0 ;struct amdgpu_mgpu_info mgpu_info = { .mutex =__MUTEX_INITIALIZER (mgpu_info.mutex),
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 528
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +3
-0@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);MODULE_PARM_DESC (ras_mask,"Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1" );module_param_named (ras_mask, amdgpu_ras_mask, uint,0444 ); +/** +* DOC: force_no_xgmi (uint) +* Forces not to use xgmi interface (0 = disable, 1 = enable).
How about using human English here? Perhaps something like,
"Forces not to use" ==> "Disables the use of the XGMI interface ..."
Or, if you use the suggestion Christian posted, you could say something like,
"Enable XGMI for P2P transactions."
Regards,
Luben
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); · line 536
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +5
-0+* Default is 0 (disabled). +*/ +MODULE_PARM_DESC (force_no_xgmi,"Force not to use xgmi interface" ); +module_param_named (force_no_xgmi, amdgpu_force_no_xgmi, int,0600 ); +/** * DOC: si_support (int) * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
[AMD Official Use Only - Internal Distribution Only]
Quoted 72 lines
-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Luben Tuikov
Sent: Wednesday, October 28, 2020 11:12 AM
To: Sierra Guiza, Alejandro (Alex) <[email protected]>; amd- [email protected]
Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi[CAUTION: External Email]
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct
Quoted 28 lines · Alex Sierra
By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions.
Signed-off-by Alex Sierra <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ 3 files changed, 11 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +0
-0@@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int
amdgpu_mes; extern intamdgpu_noretry; extern intamdgpu_force_asic_type;drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0+extern int amdgpu_force_no_xgmi;#ifdef CONFIG_HSA_AMD extern int sched_policy;extern bool debug_evictions;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
amdgpu_device*adev)drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c int amdgpu_force_asic_type = -1; int amdgpu_tmz
Quoted 16 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2
-1if (r) goto init_failed; -if (adev->gmc.xgmi.num_physical_nodes >1 ) +if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes + 1)amdgpu_xgmi_add_device (adev);amdgpu_amdkfd_device_init (adev);drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +0
-0@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz=
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c module_param_named(ras_enable,
Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +1
-00 ;int amdgpu_reset_method =-1 ;/* auto */ int amdgpu_num_kcq =-1 ; +int amdgpu_force_no_xgmi =0 ;struct amdgpu_mgpu_info mgpu_info = { .mutex =__MUTEX_INITIALIZER (mgpu_info.mutex),@@ -522,6 +523,14 @@ module_param_named(ras_enable,
amdgpu_ras_enable,Quoted 7 lines
int, 0444);
MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);+/**
+ * DOC:force_no_xgmi(uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
Quoted 7 lines
How about using human English here? Perhaps something like,
"Forces not to use" ==> "Disables the use of the XGMI interface ..."
Or, if you use the suggestion Christian posted, you could say something like,
"Enable XGMI for P2P transactions."
I second the use of enable. The fact that we can prevent xGMI init means we enable it by default.
Thanks,
Jon
Quoted 17 lines
Regards,
LubenQuoted 3 lines
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmiinterface");
Quoted 1 line
+module_param_named(force_no_xgmi,
amdgpu_force_no_xgmi, int,0600);
Quoted 6 lines
+
/**
* DOC:si_support(int)
* Set SI support driver. This parameter works after set configCONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
Quoted 11 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
freedesktop.org%2Fmailman%2Flistinfo%2Famd- gfx&data=04%7C01%7Cjonathan.kim%40amd.com%7C7e0b2ac9a2ac4c7 8353008d87b53e1b3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7 C637394947518556523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sda ta=ve9KxUmOXYYYS8MQJBkzW7V%2FRxCORSVWxlUzhFRYxzw%3D&re served=0
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx