drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()
Subsystems (3)
PATCH 1/1 drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()
drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:
unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.
For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.
Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().
--- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, }for (i =0 ; i < info->num_planes; i++) { -unsigned int width = mode_cmd->width / (i ? info->hsub :1 ); -unsigned int height = mode_cmd->height / (i ? info->vsub :1 ); +unsigned int width =drm_format_info_plane_width (info, mode_cmd->width, i); +unsigned int height =drm_format_info_plane_height (info, mode_cmd->height, i);unsigned int min_size; objs[i] =drm_gem_object_lookup (file, mode_cmd->handles[i]);
Signature · 1 line
2.34.1
Quoted 19 lines · Ashutosh Desai
drm_gem_fb_init_with_funcs()computes sub-sampled plane dimensions using plain integer division:unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1);However, the ioctl-level
framebuffer_check()indrm_framebuffer.cuses drm_format_info_plane_width/height() which round up dimensions viaDIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to
UINT_MAXas an unsigned int, causingmin_sizeto overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.
Quoted 5 lines
Fix by replacing the open-coded divisions with
drm_format_info_plane_width()anddrm_format_info_plane_height(), which useDIV_ROUND_UP()and match the calculation already used inframebuffer_check().Signed-off-by Ashutosh Desai <[email protected]>
Hello Ashutosh -
Receiving a handful of highly polished patches in quick succession on various parts of the drm subsystem from someone who has no commits in the kernel and has no previous interactions on the mailing lists is virtually unheard of.
I have to ask, did you use AI coding assistants? Please read the kernel documentation on AI coding assistants and attribution [1].
BR,
Jani.
[1] https://docs.kernel.org/process/coding-assistants.html
Quoted 19 lines
--- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_gem_framebuffer_helper.c +2
-2@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, }for (i =0 ; i < info->num_planes; i++) { -unsigned int width = mode_cmd->width / (i ? info->hsub :1 ); -unsigned int height = mode_cmd->height / (i ? info->vsub :1 ); +unsigned int width =drm_format_info_plane_width (info, mode_cmd->width, i); +unsigned int height =drm_format_info_plane_height (info, mode_cmd->height, i);unsigned int min_size; objs[i] =drm_gem_object_lookup (file, mode_cmd->handles[i]);
Signature · 1 line
Jani Nikula, Intel
Hi Jani,
Thank you for your question. It is a valid one. I have used AI to a small extent to help me understand the subsystem because it was something new for me. I haven't used AI to identify any bugs or create patches. This type of bug is something that I am always on the lookout for (size checking and boundary checks). So, these became evident during my review. Since AI wasn’t used for code generation, I didn’t think attribution was required, but please let me know if you’d prefer otherwise.
Would be happy to walk through any of the patches if it is helpful.
Best regards,
Ashutosh
Quoted 75 lines · Jani Nikula
Quoted 19 lines · Ashutosh Desai
drm_gem_fb_init_with_funcs()computes sub-sampled plane dimensions using plain integer division:unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1);However, the ioctl-level
framebuffer_check()indrm_framebuffer.cuses drm_format_info_plane_width/height() which round up dimensions viaDIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to
UINT_MAXas an unsigned int, causingmin_sizeto overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.Quoted 1 line
Fix by replacing the open-coded divisions with
drm_format_info_plane_width()Quoted 4 lines
and
drm_format_info_plane_height(), which useDIV_ROUND_UP()and match the calculation already used inframebuffer_check().Signed-off-by Ashutosh Desai <[email protected]>Hello Ashutosh -
Receiving a handful of highly polished patches in quick succession on various parts of the drm subsystem from someone who has no commits in the kernel and has no previous interactions on the mailing lists is virtually unheard of.
I have to ask, did you use AI coding assistants? Please read the kernel documentation on AI coding assistants and attribution [1].
BR,
Jani.[1] https://docs.kernel.org/process/coding-assistants.html
Review
Quoted 5 lines
--- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/drm_gem_framebuffer_helper.cdrivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device
Quoted 4 lines
index
9166c353f..88808e972100644drivers/gpu/drm/drm_gem_framebuffer_helper.c +0
-0@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device*dev,
drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device *dev,
Quoted 4 lines
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0
-1}for (i =0 ; i < info->num_planes; i++) { - unsigned int width = mode_cmd->width / (i ? info->hsub :1);
drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 176
Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0
-1- unsigned int height = mode_cmd->height / (i ? info->vsub :1);
drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 175
Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +1
-0+ unsigned int width = drm_format_info_plane_width(info,
mode_cmd->width, i);drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device · line 176
Quoted 1 line
drivers/gpu/drm/drm_gem_framebuffer_helper.c +1
-0+ unsigned int height = drm_format_info_plane_height(info,
mode_cmd->height, i);drivers/gpu/drm/drm_gem_framebuffer_helper.c int drm_gem_fb_init_with_funcs(struct drm_device *dev,
Quoted 3 lines
drivers/gpu/drm/drm_gem_framebuffer_helper.c +0
-0unsigned int min_size; objs[i] = drm_gem_object_lookup(file,mode_cmd->handles[i]);
--
Jani Nikula, Intel
Hi Jani,
Thank you for your question. It is a valid one. I have used AI to a small extent to help me understand the subsystem because it was something new for me. I haven't used AI to identify any bugs or create patches. This type of bug is something that I am always on the lookout for (size checking and boundary checks). So, these became evident during my review. Since AI wasn't used for code generation, I didn't think attribution was required, but please let me know if you'd prefer otherwise.
Would be happy to walk through any of the patches if it is helpful.
Best regards,
Ashutosh
Am 09.04.26 um 18:41 schrieb Ashutosh Desai:
Quoted 19 lines
drm_gem_fb_init_with_funcs()computes sub-sampled plane dimensions using plain integer division:unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1);However, the ioctl-level
framebuffer_check()indrm_framebuffer.cuses drm_format_info_plane_width/height() which round up dimensions viaDIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to
UINT_MAXas an unsigned int, causingmin_sizeto overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.
Quoted 5 lines
Fix by replacing the open-coded divisions with
drm_format_info_plane_width()anddrm_format_info_plane_height(), which useDIV_ROUND_UP()and match the calculation already used inframebuffer_check().Signed-off-by Ashutosh Desai <[email protected]>
Thanks for the fix.
Quoted 19 lines
--- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_gem_framebuffer_helper.c +2
-2@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, }for (i =0 ; i < info->num_planes; i++) { -unsigned int width = mode_cmd->width / (i ? info->hsub :1 ); -unsigned int height = mode_cmd->height / (i ? info->vsub :1 ); +unsigned int width =drm_format_info_plane_width (info, mode_cmd->width, i); +unsigned int height =drm_format_info_plane_height (info, mode_cmd->height, i);unsigned int min_size; objs[i] =drm_gem_object_lookup (file, mode_cmd->handles[i]);
Signature · 6 lines
-- Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Frankenstr. 146, 90461 Nürnberg, Germany, www.suse.com GF: Jochen Jaser, Andrew McDonald, Werner Knoblich, (HRB 36809, AG Nürnberg)
PATCH v2 1/1 drm/gem: Fix inconsistent plane dimension calculation in drm_gem_fb_init_with_funcs()
drm_gem_fb_init_with_funcs() computes sub-sampled plane dimensions using plain integer division:
unsigned int width = mode_cmd->width / (i ? info->hsub : 1); unsigned int height = mode_cmd->height / (i ? info->vsub : 1);
However, the ioctl-level framebuffer_check() in drm_framebuffer.c uses drm_format_info_plane_width/height() which round up dimensions via DIV_ROUND_UP(). This inconsistency corrupts the subsequent GEM object size check for certain pixel format and dimension combinations.
For example, with NV12 (vsub=2) and a 1-pixel-tall framebuffer the GEM size validation path sees height=0 instead of height=1. The expression (height - 1) then wraps to UINT_MAX as an unsigned int, causing min_size to overflow and wrap back to a small value. A tiny GEM object therefore passes the size guard, yet when the GPU accesses the chroma plane it will read or write memory beyond the object's bounds.
Fix by replacing the open-coded divisions with drm_format_info_plane_width() and drm_format_info_plane_height(), which use DIV_ROUND_UP() and match the calculation already used in framebuffer_check().
---
V1 -> V2: add Fixes: tag, Cc: [email protected], incorporate
Reviewed-by from Thomas Zimmermann.
drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_framebuffer_helper.c +2 -2
@@ -172,8 +172,8 @@ int drm_gem_fb_init_with_funcs(struct drm_device *dev, }for (i =0 ; i < info->num_planes; i++) { -unsigned int width = mode_cmd->width / (i ? info->hsub :1 ); -unsigned int height = mode_cmd->height / (i ? info->vsub :1 ); +unsigned int width =drm_format_info_plane_width (info, mode_cmd->width, i); +unsigned int height =drm_format_info_plane_height (info, mode_cmd->height, i);unsigned int min_size; objs[i] =drm_gem_object_lookup (file, mode_cmd->handles[i]);
Signature · 1 line
2.34.1