Skip to contentSkip to conversation

drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()

PATCH Superseded Ashutosh Desai 3 participants 6 messages Apr 9 – Apr 20

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()

drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:

  unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
  unsigned int height = mode_cmd->height / (i ? info->vsub : 1);

However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.

For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.

Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().

---
 drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
index 9166c353f..88808e972 100644
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
 	}
 
 	for (i = 0; i < info->num_planes; i++) {
-		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
-		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
+		unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i);
+		unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i);
 		unsigned int min_size;
 
 		objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]);
Signature · 1 line
2.34.1
#1 lore raw
Quoted 19 lines · Ashutosh Desai

drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:

  unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
  unsigned int height = mode_cmd->height / (i ? info->vsub : 1);

However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.

For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.

Quoted 5 lines

Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().

Hello Ashutosh -

Receiving a handful of highly polished patches in quick succession on various parts of the drm subsystem from someone who has no commits in the kernel and has no previous interactions on the mailing lists is virtually unheard of.

I have to ask, did you use AI coding assistants? Please read the kernel documentation on AI coding assistants and attribution [1].

BR,
Jani.

[1] https://docs.kernel.org/process/coding-assistants.html

Quoted 19 lines
---
 drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
index 9166c353f..88808e972 100644
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
 	}
 
 	for (i = 0; i < info->num_planes; i++) {
-		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
-		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
+		unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i);
+		unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i);
 		unsigned int min_size;
 
 		objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]);
Signature · 1 line
Jani Nikula, Intel
#2 lore raw

Hi Jani,

Thank you for your question. It is a valid one. I have used AI to a small extent to help me understand the subsystem because it was something new for me. I haven't used AI to identify any bugs or create patches. This type of bug is something that I am always on the lookout for (size checking and boundary checks). So, these became evident during my review. Since AI wasn’t used for code generation, I didn’t think attribution was required, but please let me know if you’d prefer otherwise.

Would be happy to walk through any of the patches if it is helpful.

Best regards,
Ashutosh

Quoted 75 lines · Jani Nikula
Quoted 19 lines · Ashutosh Desai

drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:

  unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
  unsigned int height = mode_cmd->height / (i ? info->vsub : 1);

However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.

For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.

Quoted 1 line

Fix by replacing the open-coded divisions with

drm_format_info_plane_width()

Quoted 4 lines

and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().

Hello Ashutosh -

Receiving a handful of highly polished patches in quick succession on various parts of the drm subsystem from someone who has no commits in the kernel and has no previous interactions on the mailing lists is virtually unheard of.

I have to ask, did you use AI coding assistants? Please read the kernel documentation on AI coding assistants and attribution [1].

BR,
Jani.

[1] https://docs.kernel.org/process/coding-assistants.html

Review

Quoted 5 lines
---
 drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/drivers/gpu/drm/drm_gem_framebuffer_helper.c

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device

Quoted 4 lines

index 9166c353f..88808e972 100644

drivers/gpu/drm/drm_gem_framebuffer_helper.c +0 -0
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device

*dev,

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device *dev,

Quoted 4 lines
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0 -1
      }
 
      for (i = 0; i < info->num_planes; i++) {
-             unsigned int width = mode_cmd->width / (i ? info->hsub :

1);

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 176

Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0 -1
-             unsigned int height = mode_cmd->height / (i ? info->vsub :

1);

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 175

Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +1 -0
+             unsigned int width = drm_format_info_plane_width(info,

mode_cmd->width, i);

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 176

Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +1 -0
+             unsigned int height = drm_format_info_plane_height(info,

mode_cmd->height, i);

drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device *dev,

Quoted 3 lines
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0 -0
              unsigned int min_size;
 
              objs[i] = drm_gem_object_lookup(file,

mode_cmd->handles[i]);

--
Jani Nikula, Intel

#3 lore raw
In reply to Ashutosh Desai

Hi Jani,

Thank you for your question. It is a valid one. I have used AI to a small extent to help me understand the subsystem because it was something new for me. I haven't used AI to identify any bugs or create patches. This type of bug is something that I am always on the lookout for (size checking and boundary checks). So, these became evident during my review. Since AI wasn't used for code generation, I didn't think attribution was required, but please let me know if you'd prefer otherwise.

Would be happy to walk through any of the patches if it is helpful.

Best regards,
Ashutosh

maintainer Reviewed-by on v1 1/1
#4 lore raw
In reply to Ashutosh Desai

Am 09.04.26 um 18:41 schrieb Ashutosh Desai:

Quoted 19 lines

drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:

   unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
   unsigned int height = mode_cmd->height / (i ? info->vsub : 1);

However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.

For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.

Quoted 5 lines

Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().

Thanks for the fix.

Fixes 4c3dbb2c312c ("drm: Add GEM backed framebuffer library")
Cc <[email protected]> # v4.14+
Quoted 19 lines
---
  drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
index 9166c353f..88808e972 100644
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
  	}
 
  	for (i = 0; i < info->num_planes; i++) {
-		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
-		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
+		unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i);
+		unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i);
  		unsigned int min_size;
 
  		objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]);
Signature · 6 lines
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich, (HRB 36809, AG Nürnberg)
#5 lore raw

PATCH v2 1/1 drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()

drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:

  unsigned int width  = mode_cmd->width  / (i ? info->hsub : 1);
  unsigned int height = mode_cmd->height / (i ? info->vsub : 1);

However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.

For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.

Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().

Fixes 4c3dbb2c312c ("drm: Add GEM backed framebuffer library")
Cc [email protected] # v4.14+

---
V1 -> V2: add Fixes: tag, Cc: [email protected], incorporate
Reviewed-by from Thomas Zimmermann.

 drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
index 9166c353f131..88808e972cc1 100644
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev,
 	}
 
 	for (i = 0; i < info->num_planes; i++) {
-		unsigned int width = mode_cmd->width / (i ? info->hsub : 1);
-		unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
+		unsigned int width = drm_format_info_plane_width(info, mode_cmd->width, i);
+		unsigned int height = drm_format_info_plane_height(info, mode_cmd->height, i);
 		unsigned int min_size;
 
 		objs[i] = drm_gem_object_lookup(file, mode_cmd->handles[i]);
Signature · 1 line
2.34.1