Skip to contentSkip to conversation

drm/amdkfd: fix partition query when setting up recommended sdma engines

PATCH In review Jonathan Kim 2 participants 3 messages Aug 7, 2024 – Aug 8, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdkfd: fix partition query when setting up recommended sdma engines

When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.

Fixes a0f548d7871e ("drm/amdkfd: allow users to target recommended SDMA engines")
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1 -1
index 40771f8752cb..8fee89b8dd67 100644
@@ -1287,7 +1287,7 @@ static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,
 	int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;
 	bool support_rec_eng = !amdgpu_sriov_vf(adev) && to_dev->gpu &&
 		adev->aid_mask && num_xgmi_nodes &&
-		(amdgpu_xcp_query_partition_mode(adev->xcp_mgr, AMDGPU_XCP_FL_NONE) ==
+		(amdgpu_xcp_query_partition_mode(adev->xcp_mgr, AMDGPU_XCP_FL_LOCKED) ==
 		      AMDGPU_SPX_PARTITION_MODE) &&
 		(!(adev->flags & AMD_IS_APU) && num_xgmi_nodes == 8);
 
Signature · 1 line
2.34.1
#1 lore raw
Quoted 18 lines · Jonathan Kim

When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.

Fixes a0f548d7871e ("drm/amdkfd: allow users to target recommended SDMA engines")
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -0
index 40771f8752cb..8fee89b8dd67 100644

drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev, · line 1291

Quoted 6 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1 -1
@@ -1287,7 +1287,7 @@ static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,
 	int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;
 	bool support_rec_eng = !amdgpu_sriov_vf(adev) && to_dev->gpu &&
 		adev->aid_mask && num_xgmi_nodes &&
-		(amdgpu_xcp_query_partition_mode(adev->xcp_mgr, AMDGPU_XCP_FL_NONE) ==
+		(amdgpu_xcp_query_partition_mode(adev->xcp_mgr, AMDGPU_XCP_FL_LOCKED) ==
 		      AMDGPU_SPX_PARTITION_MODE) &&

Replacing with (gpu->kfd->num_nodes == 1) may be better.

Thanks,
Lijo

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -0
 		(!(adev->flags & AMD_IS_APU) && num_xgmi_nodes == 8);
 
#2 lore raw

[Public]

Quoted 36 lines

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Wednesday, August 7, 2024 11:46 PM
To: Kim, Jonathan <[email protected]>; [email protected]
Cc: Kuehling, Felix <[email protected]>
Subject: Re: [PATCH] drm/amdkfd: fix partition query when setting up recommended sdma engines

Quoted 8 lines · Jonathan Kim

When users dynamically set the partition mode through sysfs writes, this can lead to a double lock situation where the KFD is trying to take the partition lock when updating the recommended SDMA engines. Have the KFD do a lockless query instead to avoid this.
This should work since the KFD always initializes synchronously after the KGD partition mode is set regardless of user or system setup.

Fixes a0f548d7871e ("drm/amdkfd: allow users to target recommended

SDMA engines")

Review

Quoted 6 lines
---
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c

drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void

Quoted 4 lines

index 40771f8752cb..8fee89b8dd67 100644

drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -0
@@ -1287,7 +1287,7 @@ static void

kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,

drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev,

Quoted 4 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -1
    int num_xgmi_nodes = adev->gmc.xgmi.num_physical_nodes;
    bool support_rec_eng = !amdgpu_sriov_vf(adev) && to_dev->gpu &&
            adev->aid_mask && num_xgmi_nodes &&
-           (amdgpu_xcp_query_partition_mode(adev->xcp_mgr,
Quoted 6 lines

AMDGPU_XCP_FL_NONE) ==

drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void · line 1290

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +1 -0
+           (amdgpu_xcp_query_partition_mode(adev->xcp_mgr,

AMDGPU_XCP_FL_LOCKED) ==

drivers/gpu/drm/amd/amdkfd/kfd_topology.c static void kfd_set_recommended_sdma_engines(struct kfd_topology_device *to_dev, · line 1291

Quoted 1 line
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -0
                  AMDGPU_SPX_PARTITION_MODE) &&

Replacing with (gpu->kfd->num_nodes == 1) may be better.

Thanks. That seems a lot simpler. Also another assumption is that all 14 SDMA xGMI engines are present, but that may or may not always be the case for all dGPU SPX-mode devices. I'll add that as a hard condition check as well.

Jon

Quoted 6 lines

Thanks,
Lijo

Quoted 2 lines
drivers/gpu/drm/amd/amdkfd/kfd_topology.c +0 -0
            (!(adev->flags & AMD_IS_APU) && num_xgmi_nodes == 8);