Skip to contentSkip to conversation

drm/ttm: Helper function to get TTM mem limit

PATCH In review Mukul Joshi 3 participants 6 messages Apr 26, 2023

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 drm/ttm: Helper function to get TTM mem limit

Add a helper function to get TTM memory limit. This is needed by KFD to set its own internal memory limits.

---
 drivers/gpu/drm/ttm/ttm_tt.c | 6 ++++++
 include/drm/ttm/ttm_tt.h     | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_tt.c +6 -0
index d505603930a7..1f765dd7792c 100644
@@ -449,3 +449,9 @@ ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 	return &iter_tt->base;
 }
 EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
+
+unsigned long ttm_tt_pages_limit(void)
+{
+	return ttm_pages_limit;
+}
+EXPORT_SYMBOL(ttm_tt_pages_limit);
include/drm/ttm/ttm_tt.h +1 -1
index b7d3f3843f1e..d54b2dc05d71 100644
@@ -222,7 +222,7 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 
 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 					    struct ttm_tt *tt);
-
+unsigned long ttm_tt_pages_limit(void);
 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>
 
Signature · 1 line
2.35.1
#1 lore raw

PATCH 2/3 drm/amdgpu: Set GTT size equal to TTM mem limit

Use the helper function in TTM to get TTM mem limit and set GTT size to be equal to TTL mem limit.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25 ++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6 -19
index ce34b73d05bc..ac220c779fc8 100644
@@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either based on 1/2 the size of RAM size
-	 * or whatever the user passed on module init */
-	if (amdgpu_gtt_size == -1) {
-		struct sysinfo si;
-
-		si_meminfo(&si);
-		/* Certain GL unit tests for large textures can cause problems
-		 * with the OOM killer since there is no way to link this memory
-		 * to a process.  This was originally mitigated (but not necessarily
-		 * eliminated) by limiting the GTT size.  The problem is this limit
-		 * is often too low for many modern games so just make the limit 1/2
-		 * of system memory which aligns with TTM. The OOM accounting needs
-		 * to be addressed, but we shouldn't prevent common 3D applications
-		 * from being usable just to potentially mitigate that corner case.
-		 */
-		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-			       (u64)si.totalram * si.mem_unit / 2);
-	} else {
+	/* Compute GTT size, either based on TTM limit
+	 * or whatever the user passed on module init.
+	 */
+	if (amdgpu_gtt_size == -1)
+		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
+	else
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
-	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
Signature · 1 line
2.35.1
#2 lore raw

RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

Looks you can drop macro 'AMDGPU_DEFAULT_GTT_SIZE_MB' as well.

Regards,
Guchun

Quoted 65 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Mukul Joshi
Sent: Wednesday, April 26, 2023 9:53 AM
To: [email protected]; [email protected]
Cc: Joshi, Mukul <[email protected]>; Kuehling, Felix <[email protected]>; Koenig, Christian <[email protected]>
Subject: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

Use the helper function in TTM to get TTM mem limit and set GTT size to be equal to TTL mem limit.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25 ++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -8
index ce34b73d05bc..ac220c779fc8 100644
@@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device
 *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either based on 1/2 the size of RAM size
-	 * or whatever the user passed on module init */
-	if (amdgpu_gtt_size == -1) {
-		struct sysinfo si;
-
-		si_meminfo(&si);
-		/* Certain GL unit tests for large textures can cause problems
-		 * with the OOM killer since there is no way to link this

memory

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * to a process.  This was originally mitigated (but not

necessarily

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * eliminated) by limiting the GTT size.  The problem is this

limit

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * is often too low for many modern games so just make the

limit 1/2

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * of system memory which aligns with TTM. The OOM

accounting needs

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * to be addressed, but we shouldn't prevent common 3D

applications

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * from being usable just to potentially mitigate that corner

case.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6 -6
-		 */
-		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-			       (u64)si.totalram * si.mem_unit / 2);
-	} else {
+	/* Compute GTT size, either based on TTM limit
+	 * or whatever the user passed on module init.
+	 */
+	if (amdgpu_gtt_size == -1)
+		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
+	else
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
-	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.1

#3 lore raw

RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

[AMD Official Use Only - General]

Quoted 11 lines

-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, April 26, 2023 2:00 AM
To: Joshi, Mukul <[email protected]>; [email protected]; [email protected]
Cc: Joshi, Mukul <[email protected]>; Kuehling, Felix <[email protected]>; Koenig, Christian <[email protected]>
Subject: RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

Looks you can drop macro 'AMDGPU_DEFAULT_GTT_SIZE_MB' as well.

Thanks for catching this. Sorry I missed it. I will remove and send a v2.

Regards,
Mukul

Quoted 70 lines

Regards,
Guchun

Quoted 7 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Mukul Joshi
Sent: Wednesday, April 26, 2023 9:53 AM
To: [email protected]; [email protected]
Cc: Joshi, Mukul <[email protected]>; Kuehling, Felix <[email protected]>; Koenig, Christian

<[email protected]>

Quoted 59 lines

Subject: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

Use the helper function in TTM to get TTM mem limit and set GTT size to be equal to TTL mem limit.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25
++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -8
index ce34b73d05bc..ac220c779fc8 100644
@@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device
 *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either based on 1/2 the size of RAM size
-	 * or whatever the user passed on module init */
-	if (amdgpu_gtt_size == -1) {
-		struct sysinfo si;
-
-		si_meminfo(&si);
-		/* Certain GL unit tests for large textures can cause problems
-		 * with the OOM killer since there is no way to link this

memory

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * to a process.  This was originally mitigated (but not

necessarily

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * eliminated) by limiting the GTT size.  The problem is this

limit

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * is often too low for many modern games so just make the

limit 1/2

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * of system memory which aligns with TTM. The OOM

accounting needs

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * to be addressed, but we shouldn't prevent common 3D

applications

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -1
-		 * from being usable just to potentially mitigate that corner

case.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +6 -6
-		 */
-		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-			       (u64)si.totalram * si.mem_unit / 2);
-	} else {
+	/* Compute GTT size, either based on TTM limit
+	 * or whatever the user passed on module init.
+	 */
+	if (amdgpu_gtt_size == -1)
+		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
+	else
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
-	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.1

#4 lore raw

PATCH 3/3 drm/amdkfd: Update KFD TTM mem limit

In reply to Mukul Joshi

Use the helper function in TTM to get TTM memory limit and set KFD's internal mem limit. This ensures that KFD's TTM mem limit and actual TTM mem limit are exactly same.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c       | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 +++++-
 2 files changed, 7 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +2 -1
index fed8bb9a721f..a46285841d17 100644
@@ -53,7 +53,6 @@ int amdgpu_amdkfd_init(void)
 	amdgpu_amdkfd_total_mem_size *= si.mem_unit;
 
 	ret = kgd2kfd_init();
-	amdgpu_amdkfd_gpuvm_init_mem_limits();
 	kfd_initialized = !ret;
 
 	return ret;
@@ -143,6 +142,8 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
 	int i;
 	int last_valid_bit;
 
+	amdgpu_amdkfd_gpuvm_init_mem_limits();
+
 	if (adev->kfd.dev) {
 		struct kgd2kfd_shared_resources gpu_resources = {
 			.compute_vmid_bitmap =
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +5 -1
index 862e94fbf53c..1002c7834386 100644
@@ -35,6 +35,7 @@
 #include <uapi/linux/kfd_ioctl.h>
 #include "amdgpu_xgmi.h"
 #include "kfd_smi_events.h"
+#include <drm/ttm/ttm_tt.h>
 
 /* Userptr restore delay, just long enough to allow consecutive VM
  * changes to accumulate
@@ -109,13 +110,16 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
 	struct sysinfo si;
 	uint64_t mem;
 
+	if (kfd_mem_limit.max_system_mem_limit)
+		return;
+
 	si_meminfo(&si);
 	mem = si.freeram - si.freehigh;
 	mem *= si.mem_unit;
 
 	spin_lock_init(&kfd_mem_limit.mem_limit_lock);
 	kfd_mem_limit.max_system_mem_limit = mem - (mem >> 4);
-	kfd_mem_limit.max_ttm_mem_limit = (mem >> 1) - (mem >> 3);
+	kfd_mem_limit.max_ttm_mem_limit = ttm_tt_pages_limit() << PAGE_SHIFT;
 	pr_debug("Kernel memory limit %lluM, TTM limit %lluM\n",
 		(kfd_mem_limit.max_system_mem_limit >> 20),
 		(kfd_mem_limit.max_ttm_mem_limit >> 20));
Signature · 1 line
2.35.1
maintainer Reviewed-by on v1 1/3
#5 lore raw
In reply to Mukul Joshi

Am 26.04.23 um 03:52 schrieb Mukul Joshi:

Quoted 4 lines

Add a helper function to get TTM memory limit. This is needed by KFD to set its own internal memory limits.

Quoted 32 lines
---
  drivers/gpu/drm/ttm/ttm_tt.c | 6 ++++++
  include/drm/ttm/ttm_tt.h     | 2 +-
  2 files changed, 7 insertions(+), 1 deletion(-)
drivers/gpu/drm/ttm/ttm_tt.c +6 -0
index d505603930a7..1f765dd7792c 100644
@@ -449,3 +449,9 @@ ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
  	return &iter_tt->base;
  }
  EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
+
+unsigned long ttm_tt_pages_limit(void)
+{
+	return ttm_pages_limit;
+}
+EXPORT_SYMBOL(ttm_tt_pages_limit);
include/drm/ttm/ttm_tt.h +1 -1
index b7d3f3843f1e..d54b2dc05d71 100644
@@ -222,7 +222,7 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 
  struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
  					    struct ttm_tt *tt);
-
+unsigned long ttm_tt_pages_limit(void);
  #if IS_ENABLED(CONFIG_AGP)
  #include <linux/agp_backend.h>