drm/amdkfd: fix partition query when setting up recommended sdma engines
Subsystems (3)
PATCH 1/1 drm/amdkfd: fix partition query when setting up recommended sdma engines
When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.
--- drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1 -1
@@ -1287,7 +1287,7 @@ static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;bool support_rec_eng = !amdgpu_sriov_vf (adev) && to_dev->gpu && adev->aid_mask && num_xgmi_nodes && - (amdgpu_xcp_query_partition_mode (adev->xcp_mgr,AMDGPU_XCP_FL_NONE ) == + (amdgpu_xcp_query_partition_mode (adev->xcp_mgr,AMDGPU_XCP_FL_LOCKED ) ==AMDGPU_SPX_PARTITION_MODE ) && (!(adev->flags &AMD_IS_APU ) && num_xgmi_nodes ==8 );
Signature · 1 line
2.34.1
Quoted 18 lines · Jonathan Kim
When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.Fixes a0f548d7871e ("drm/amdkfd: allow users to target recommended SDMA engines")Signed-off-by Jonathan Kim <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev, · line 1291
Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1
-1@@ -1287,7 +1287,7 @@ static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;bool support_rec_eng = !amdgpu_sriov_vf (adev) && to_dev->gpu && adev->aid_mask && num_xgmi_nodes && - (amdgpu_xcp_query_partition_mode (adev->xcp_mgr,AMDGPU_XCP_FL_NONE ) == + (amdgpu_xcp_query_partition_mode (adev->xcp_mgr,AMDGPU_XCP_FL_LOCKED ) == AMDGPU_SPX_PARTITION_MODE) &&
Replacing with (gpu->kfd->num_nodes == 1) may be better.
Thanks,
Lijo
Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-0(!(adev->flags &AMD_IS_APU ) && num_xgmi_nodes ==8 );
[Public]
Quoted 36 lines
-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, August 7, 2024 11:46 PM
To: Kim, Jonathan <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>
Subject: Re: [PATCH] drm/amdkfd: fix partition query when setting up recommended sdma enginesQuoted 8 lines · Jonathan Kim
When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.Fixes a0f548d7871e ("drm/amdkfd: allow users to target recommendedSDMA engines")
Review
Quoted 6 lines
Signed-off-by Jonathan Kim <[email protected]>--- drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)(quoted diff) +0
-0
b/drivers/gpu/drm/amd/amdkfd/kfd_topology.cdrivers/gpu/drm/amd/amdkfd/kfd_topology.c static void
Quoted 4 lines
index
40771f8752cb..8fee89b8dd67100644drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-0@@ -1287,7 +1287,7 @@ static voidkfd_set_recommended_sdma_engines(struct
kfd_topology_device*to_dev,drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,
Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-1int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;bool support_rec_eng = !amdgpu_sriov_vf (adev) && to_dev->gpu && adev->aid_mask && num_xgmi_nodes && - (amdgpu_xcp_query_partition_mode(adev->xcp_mgr,
Quoted 6 lines
AMDGPU_XCP_FL_NONE) ==drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void · line 1290
Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1
-0+ (amdgpu_xcp_query_partition_mode(adev->xcp_mgr,
AMDGPU_XCP_FL_LOCKED) ==drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev, · line 1291
Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-0AMDGPU_SPX_PARTITION_MODE) &&Replacing with (
gpu->kfd->num_nodes== 1) may be better.
Thanks. That seems a lot simpler. Also another assumption is that all 14 SDMA xGMI engines are present, but that may or may not always be the case for all dGPU SPX-mode devices. I'll add that as a hard condition check as well.
Jon
Quoted 6 lines
Thanks,
LijoQuoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0
-0(!(adev->flags &AMD_IS_APU ) && num_xgmi_nodes ==8 );