Skip to contentSkip to conversation

drm/amdgpu: Adjust logic around GTT size (v3)

PATCH In review Alex Deucher 6 participants 7 messages May 20, 2022 – Jun 15, 2022

Subsystems (2)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

PATCH 1/1 drm/amdgpu: Adjust logic around GTT size (v3)

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +14 -6
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either bsaed on 3/4th the size of RAM size
+	/* Compute GTT size, either bsaed on 1/2 the size of RAM size
 	 * or whatever the user passed on module init */
 	if (amdgpu_gtt_size == -1) {
 		struct sysinfo si;
 
 		si_meminfo(&si);
-		gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-			       adev->gmc.mc_vram_size),
-			       ((uint64_t)si.totalram * si.mem_unit * 3/4));
-	}
-	else
+		/* Certain GL unit tests for large textures can cause problems
+		 * with the OOM killer since there is no way to link this memory
+		 * to a process.  This was originally mitigated (but not necessarily
+		 * eliminated) by limiting the GTT size.  The problem is this limit
+		 * is often too low for many modern games so just make the limit 1/2
+		 * of system memory which aligns with TTM. The OOM accounting needs
+		 * to be addressed, but we shouldn't prevent common 3D applications
+		 * from being usable just to potentially mitigate that corner case.
+		 */
+		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+			       (u64)si.totalram * si.mem_unit / 2);
+	} else {
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
Signature · 1 line
2.35.3
#1 lore raw

[AMD Official Use Only - General]

I'll defer to Felix/Christian for the actual change, but a small typo in a comment:

Quoted 33 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Alex Deucher
Sent: Friday, May 20, 2022 11:09 AM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>
Subject: [PATCH] drm/amdgpu: Adjust logic around GTT size (v3)

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +0 -0
index d2b5cccb45c3..7195ed77c85a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c int amdgpu_ttm_init(struct amdgpu_device · line 1802

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -1
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device
 *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either bsaed on 3/4th the size of RAM size
+	/* Compute GTT size, either bsaed on 1/2 the size of RAM size

^ s/bsaed/based

Kent

Quoted 37 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -5
 	 * or whatever the user passed on module init */
 	if (amdgpu_gtt_size == -1) {
 		struct sysinfo si;
 
 		si_meminfo(&si);
-		gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB <<
 20),
-			       adev->gmc.mc_vram_size),
-			       ((uint64_t)si.totalram * si.mem_unit * 3/4));
-	}
-	else
+		/* Certain GL unit tests for large textures can cause problems
+		 * with the OOM killer since there is no way to link this

memory

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * to a process.  This was originally mitigated (but not

necessarily

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * eliminated) by limiting the GTT size.  The problem is this

limit

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * is often too low for many modern games so just make the

limit 1/2

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * of system memory which aligns with TTM. The OOM

accounting needs

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * to be addressed, but we shouldn't prevent common 3D

applications

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +1 -0
+		 * from being usable just to potentially mitigate that corner

case.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5 -1
+		 */
+		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+			       (u64)si.totalram * si.mem_unit / 2);
+	} else {
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3

Reviewed-by on v1 1/1
#2 lore raw
In reply to Alex Deucher

Marek

Quoted 70 lines · Alex Deucher

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2 -6
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
        DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
                 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-       /* Compute GTT size, either bsaed on 3/4th the size of RAM size
+       /* Compute GTT size, either bsaed on 1/2 the size of RAM size
         * or whatever the user passed on module init */
        if (amdgpu_gtt_size == -1) {
                struct sysinfo si;
 
                si_meminfo(&si);
-               gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-                              adev->gmc.mc_vram_size),
-                              ((uint64_t)si.totalram * si.mem_unit *
 3/4));
-       }
-       else
+               /* Certain GL unit tests for large textures can cause

problems
+ * with the OOM killer since there is no way to link this
memory
+ * to a process. This was originally mitigated (but not
necessarily
+ * eliminated) by limiting the GTT size. The problem is
this limit
+ * is often too low for many modern games so just make the
limit 1/2
+ * of system memory which aligns with TTM. The OOM
accounting needs
+ * to be addressed, but we shouldn't prevent common 3D
applications
+ * from being usable just to potentially mitigate that
corner case.

drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5 -1
+                */
+               gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+                              (u64)si.totalram * si.mem_unit / 2);
+       } else {
                gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+       }
 
        /* Initialize GTT memory pool */
        r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3

#3 lore raw
In reply to Alex Deucher

@Christian Koenig
Any objections to this? I realize that fixing the OOM killer is ultimately the right approach, but I don't really see how this makes things worse. The current scheme is biased towards dGPUs as they have lots of on board memory so on dGPUs we can end up setting gtt size to 3/4 of system memory already in a lot of cases since there is often as much vram as system memory. Due to the limits in ttm, we can't use more than half at the moment anway, so this shouldn't make things worse on dGPUs and would help a lot of APUs. Once could make the argument that with more vram there is less need for gtt so less chance for OOM, but I think it is more of a scale issue. E.g., on dGPUs you'll generally be running higher resolutions and texture quality, etc. so the overall memory footprint is just scaled up.

Alex

Quoted 60 lines · Alex Deucher

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +14 -7
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
        DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
                 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-       /* Compute GTT size, either bsaed on 3/4th the size of RAM size
+       /* Compute GTT size, either bsaed on 1/2 the size of RAM size
         * or whatever the user passed on module init */
        if (amdgpu_gtt_size == -1) {
                struct sysinfo si;
 
                si_meminfo(&si);
-               gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-                              adev->gmc.mc_vram_size),
-                              ((uint64_t)si.totalram * si.mem_unit * 3/4));
-       }
-       else
+               /* Certain GL unit tests for large textures can cause problems
+                * with the OOM killer since there is no way to link this memory
+                * to a process.  This was originally mitigated (but not necessarily
+                * eliminated) by limiting the GTT size.  The problem is this limit
+                * is often too low for many modern games so just make the limit 1/2
+                * of system memory which aligns with TTM. The OOM accounting needs
+                * to be addressed, but we shouldn't prevent common 3D applications
+                * from being usable just to potentially mitigate that corner case.
+                */
+               gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+                              (u64)si.totalram * si.mem_unit / 2);
+       } else {
                gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+       }
 
        /* Initialize GTT memory pool */
        r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3

maintainer
#4 lore raw

I totally agree on the reasoning, but I have the strong feeling that this will blow up in our face once more.

I've tried to raise this limit twice already and had to revert it both times. And the reasons why I had to revert it haven't changed since them.

Christian.

Am 02.06.22 um 18:40 schrieb Alex Deucher:

Quoted 76 lines

@Christian Koenig
Any objections to this? I realize that fixing the OOM killer is ultimately the right approach, but I don't really see how this makes things worse. The current scheme is biased towards dGPUs as they have lots of on board memory so on dGPUs we can end up setting gtt size to 3/4 of system memory already in a lot of cases since there is often as much vram as system memory. Due to the limits in ttm, we can't use more than half at the moment anway, so this shouldn't make things worse on dGPUs and would help a lot of APUs. Once could make the argument that with more vram there is less need for gtt so less chance for OOM, but I think it is more of a scale issue. E.g., on dGPUs you'll generally be running higher resolutions and texture quality, etc. so the overall memory footprint is just scaled up.

Alex

Quoted 59 lines · Alex Deucher

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
  1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +14 -7
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
         DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
                  (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-       /* Compute GTT size, either bsaed on 3/4th the size of RAM size
+       /* Compute GTT size, either bsaed on 1/2 the size of RAM size
          * or whatever the user passed on module init */
         if (amdgpu_gtt_size == -1) {
                 struct sysinfo si;
 
                 si_meminfo(&si);
-               gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-                              adev->gmc.mc_vram_size),
-                              ((uint64_t)si.totalram * si.mem_unit * 3/4));
-       }
-       else
+               /* Certain GL unit tests for large textures can cause problems
+                * with the OOM killer since there is no way to link this memory
+                * to a process.  This was originally mitigated (but not necessarily
+                * eliminated) by limiting the GTT size.  The problem is this limit
+                * is often too low for many modern games so just make the limit 1/2
+                * of system memory which aligns with TTM. The OOM accounting needs
+                * to be addressed, but we shouldn't prevent common 3D applications
+                * from being usable just to potentially mitigate that corner case.
+                */
+               gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+                              (u64)si.totalram * si.mem_unit / 2);
+       } else {
                 gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+       }
 
         /* Initialize GTT memory pool */
         r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3

#5 lore raw

I don't see how this is worse than the current behavior. We have some bug reports where we have games that use a lot of memory and with the lower limit the system ends up dying due to swapping and the behavior is actually better with the patch.

Alex

Quoted 87 lines · Christian König

I totally agree on the reasoning, but I have the strong feeling that this will blow up in our face once more.

I've tried to raise this limit twice already and had to revert it both times. And the reasons why I had to revert it haven't changed since them.

Christian.

Am 02.06.22 um 18:40 schrieb Alex Deucher:

Quoted 76 lines

@Christian Koenig
Any objections to this? I realize that fixing the OOM killer is ultimately the right approach, but I don't really see how this makes things worse. The current scheme is biased towards dGPUs as they have lots of on board memory so on dGPUs we can end up setting gtt size to 3/4 of system memory already in a lot of cases since there is often as much vram as system memory. Due to the limits in ttm, we can't use more than half at the moment anway, so this shouldn't make things worse on dGPUs and would help a lot of APUs. Once could make the argument that with more vram there is less need for gtt so less chance for OOM, but I think it is more of a scale issue. E.g., on dGPUs you'll generally be running higher resolutions and texture quality, etc. so the overall memory footprint is just scaled up.

Alex

Quoted 59 lines · Alex Deucher

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
  1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +14 -7
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
         DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
                  (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-       /* Compute GTT size, either bsaed on 3/4th the size of RAM size
+       /* Compute GTT size, either bsaed on 1/2 the size of RAM size
          * or whatever the user passed on module init */
         if (amdgpu_gtt_size == -1) {
                 struct sysinfo si;
 
                 si_meminfo(&si);
-               gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-                              adev->gmc.mc_vram_size),
-                              ((uint64_t)si.totalram * si.mem_unit * 3/4));
-       }
-       else
+               /* Certain GL unit tests for large textures can cause problems
+                * with the OOM killer since there is no way to link this memory
+                * to a process.  This was originally mitigated (but not necessarily
+                * eliminated) by limiting the GTT size.  The problem is this limit
+                * is often too low for many modern games so just make the limit 1/2
+                * of system memory which aligns with TTM. The OOM accounting needs
+                * to be addressed, but we shouldn't prevent common 3D applications
+                * from being usable just to potentially mitigate that corner case.
+                */
+               gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+                              (u64)si.totalram * si.mem_unit / 2);
+       } else {
                 gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+       }
 
         /* Initialize GTT memory pool */
         r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3

#6 lore raw

Well the key point is it isn't better. It's just that different people will start complaining.

I'm fine with trying to change it once more, just keep in mind that we decide between two evils and somebody could start complaining.

Christian.

Am 14.06.22 um 15:43 schrieb Alex Deucher:

Quoted 93 lines

I don't see how this is worse than the current behavior. We have some bug reports where we have games that use a lot of memory and with the lower limit the system ends up dying due to swapping and the behavior is actually better with the patch.

Alex

Quoted 85 lines · Christian König

I totally agree on the reasoning, but I have the strong feeling that this will blow up in our face once more.

I've tried to raise this limit twice already and had to revert it both times. And the reasons why I had to revert it haven't changed since them.

Christian.

Am 02.06.22 um 18:40 schrieb Alex Deucher:

Quoted 76 lines

@Christian Koenig
Any objections to this? I realize that fixing the OOM killer is ultimately the right approach, but I don't really see how this makes things worse. The current scheme is biased towards dGPUs as they have lots of on board memory so on dGPUs we can end up setting gtt size to 3/4 of system memory already in a lot of cases since there is often as much vram as system memory. Due to the limits in ttm, we can't use more than half at the moment anway, so this shouldn't make things worse on dGPUs and would help a lot of APUs. Once could make the argument that with more vram there is less need for gtt so less chance for OOM, but I think it is more of a scale issue. E.g., on dGPUs you'll generally be running higher resolutions and texture quality, etc. so the overall memory footprint is just scaled up.

Alex

Quoted 59 lines · Alex Deucher

Certain GL unit tests for large textures can cause problems with the OOM killer since there is no way to link this memory to a process. This was originally mitigated (but not necessarily eliminated) by limiting the GTT size. The problem is this limit is often too low for many modern games so just make the limit 1/2 of system memory. The OOM accounting needs to be addressed, but we shouldn't prevent common 3D applications from being usable just to potentially mitigate that corner case.

Set default GTT size to max(3G, 1/2 of system ram) by default.

v2: drop previous logic and default to 3/4 of ram
v3: default to half of ram to align with ttm

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 ++++++++++++++------
   1 file changed, 14 insertions(+), 6 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +14 -7
index d2b5cccb45c3..7195ed77c85a 100644
@@ -1798,18 +1798,26 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
          DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
                   (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-       /* Compute GTT size, either bsaed on 3/4th the size of RAM size
+       /* Compute GTT size, either bsaed on 1/2 the size of RAM size
           * or whatever the user passed on module init */
          if (amdgpu_gtt_size == -1) {
                  struct sysinfo si;
 
                  si_meminfo(&si);
-               gtt_size = min(max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-                              adev->gmc.mc_vram_size),
-                              ((uint64_t)si.totalram * si.mem_unit * 3/4));
-       }
-       else
+               /* Certain GL unit tests for large textures can cause problems
+                * with the OOM killer since there is no way to link this memory
+                * to a process.  This was originally mitigated (but not necessarily
+                * eliminated) by limiting the GTT size.  The problem is this limit
+                * is often too low for many modern games so just make the limit 1/2
+                * of system memory which aligns with TTM. The OOM accounting needs
+                * to be addressed, but we shouldn't prevent common 3D applications
+                * from being usable just to potentially mitigate that corner case.
+                */
+               gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
+                              (u64)si.totalram * si.mem_unit / 2);
+       } else {
                  gtt_size = (uint64_t)amdgpu_gtt_size << 20;
+       }
 
          /* Initialize GTT memory pool */
          r = amdgpu_gtt_mgr_init(adev, gtt_size);
--

2.35.3