Skip to contentSkip to conversation

video: fbdev: kyro: Validate overlay viewport coordinates

PATCH Applied Danila Chernetsov 3 participants 4 messages Jul 24 – Jul 25 Framebuffer layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 video: fbdev: kyro: Validate overlay viewport coordinates

The overlay viewport end coordinates are computed from the viewport origin and dimensions using 32-bit unsigned arithmetic. Large input values can cause these calculations to wrap around before the resulting coordinates are passed to SetOverlayViewPort().

SetOverlayViewPort() packs the viewport coordinates into 16-bit register fields. The X coordinates are additionally adjusted by +2 and +1 before being written. Validate the coordinate calculations for 32-bit wraparound and ensure that the adjusted coordinates fit within their 16-bit register fields before calling SetOverlayViewPort().

Found by Linux Verification Center (linuxtesting.org) with SVACE.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
---
 drivers/video/fbdev/kyro/fbdev.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
drivers/video/fbdev/kyro/fbdev.c +23 -1
index d756b3603fa6..133c91716c49 100644
@@ -369,6 +369,9 @@ static int kyro_dev_overlay_create(u32 ulWidth,
 
 static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight)
 {
+	u32 right;
+	u32 bottom;
+
 	if (deviceInfo.ulOverlayOffset == 0)
 		/* probably haven't called CreateOverlay yet */
 		return -EINVAL;
@@ -378,11 +381,30 @@ static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight
 	    (x < 2 && ulWidth + 2 == 0))
 		return -EINVAL;
 
+	/*
+	 * SetOverlayViewPort() adjusts X coordinates by +2 (left) and +1
+	 * (right) before packing them into 16-bit register fields.
+	 */
+	if (x > U16_MAX - 2 || y > U16_MAX)
+		return -EINVAL;
+
+	right = x + ulWidth;
+	bottom = y + ulHeight;
+
+	if (right < x || bottom < y)
+		return -EINVAL;
+
+	right--;
+	bottom--;
+
+	if (right > U16_MAX - 1 || bottom > U16_MAX)
+		return -EINVAL;
+
 	/* Stop Ramdac Output */
 	DisableRamdacOutput(deviceInfo.pSTGReg);
 
 	SetOverlayViewPort(deviceInfo.pSTGReg,
-			   x, y, x + ulWidth - 1, y + ulHeight - 1);
+			x, y, right, bottom);
 
 	EnableOverlayPlane(deviceInfo.pSTGReg);
 	/* Start Ramdac Output */
Signature · 1 line
2.25.1
#1 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing validation for ulHeight == 1 causes a division by zero panic in SetOverlayViewPort. --

--- Patch [1]: [PATCH] video: fbdev: kyro: Validate overlay viewport coordinates --- commit eccd0fc9ac01567bf3583eb92ff9c8b899a5e6b3
Author: Danila Chernetsov <[email protected]>

video: fbdev: kyro: Validate overlay viewport coordinates

This commit adds coordinate validation in kyro_dev_overlay_viewport_set to prevent 32-bit wraparound before passing the values to SetOverlayViewPort. It explicitly checks for overflows when adding dimensions and restricts the adjusted coordinates to fit into 16-bit register fields.

Quoted 3 lines
drivers/video/fbdev/kyro/fbdev.c +0 -0

[ ... ]

Quoted 13 lines
drivers/video/fbdev/kyro/fbdev.c +9 -0
@@ -378,11 +381,30 @@ static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight
 	    (x < 2 && ulWidth + 2 == 0))
 		return -EINVAL;
 
+	/*
+	 * SetOverlayViewPort() adjusts X coordinates by +2 (left) and +1
+	 * (right) before packing them into 16-bit register fields.
+	 */
+	if (x > U16_MAX - 2 || y > U16_MAX)
+		return -EINVAL;
+
+	right = x + ulWidth;
+	bottom = y + ulHeight;

drivers/video/fbdev/kyro/fbdev.c static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight · line 402

Quoted 10 lines
drivers/video/fbdev/kyro/fbdev.c +10 -0
+
+	if (right < x || bottom < y)
+		return -EINVAL;
+
+	right--;
+	bottom--;
+
+	if (right > U16_MAX - 1 || bottom > U16_MAX)
+		return -EINVAL;
+

[Severity: High]
This is a pre-existing issue, but does this validation block still allow a divide-by-zero panic if ulHeight is 1?

If a user passes ySize = 1 via the KYRO_IOCTL_OVERLAY_VIEWPORT_SET ioctl, ulHeight will be 1. The code calculates bottom = y + ulHeight, then bottom--, which results in bottom == y.

When these values are passed to SetOverlayViewPort(), it calculates ulDest as the difference between the y coordinates (bottom - top). Since they are equal, ulDest evaluates to 0:

drivers/video/fbdev/kyro/STG4000OverlayDevice.c:SetOverlayViewPort() {
    ...
    ulDest = srcDest.lDstY2 - srcDest.lDstY1;
    ...
    ulFxScale = (ulDest << 11) / ulSrc;
    ...
    ulSrcBottom = ulSrcBottom - (ulFxOffset / ulFxScale);
    ...
}

Because ulDest is 0, ulFxScale evaluates to 0. The subsequent division by ulFxScale then causes a kernel panic.

Should ulHeight == 1 be rejected alongside the ulHeight == 0 check earlier in kyro_dev_overlay_viewport_set()?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
author
#2 lore raw
Quoted 69 lines · On 24/07/2026 03:54, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing validation for ulHeight == 1 causes a division by zero panic in SetOverlayViewPort. --

--- Patch [1]: [PATCH] video: fbdev: kyro: Validate overlay viewport coordinates --- commit eccd0fc9ac01567bf3583eb92ff9c8b899a5e6b3
Author: Danila Chernetsov <[email protected]>

video: fbdev: kyro: Validate overlay viewport coordinates

This commit adds coordinate validation in kyro_dev_overlay_viewport_set to prevent 32-bit wraparound before passing the values to SetOverlayViewPort. It explicitly checks for overflows when adding dimensions and restricts the adjusted coordinates to fit into 16-bit register fields.

Quoted 3 lines
drivers/video/fbdev/kyro/fbdev.c +0 -0

[ ... ]

Quoted 13 lines
drivers/video/fbdev/kyro/fbdev.c +9 -0
@@ -378,11 +381,30 @@ static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight
  	    (x < 2 && ulWidth + 2 == 0))
  		return -EINVAL;
 
+	/*
+	 * SetOverlayViewPort() adjusts X coordinates by +2 (left) and +1
+	 * (right) before packing them into 16-bit register fields.
+	 */
+	if (x > U16_MAX - 2 || y > U16_MAX)
+		return -EINVAL;
+
+	right = x + ulWidth;
+	bottom = y + ulHeight;

drivers/video/fbdev/kyro/fbdev.c static int kyro_dev_overlay_viewport_set(u32 x, u32 y, u32 ulWidth, u32 ulHeight · line 402

Quoted 10 lines
drivers/video/fbdev/kyro/fbdev.c +10 -0
+
+	if (right < x || bottom < y)
+		return -EINVAL;
+
+	right--;
+	bottom--;
+
+	if (right > U16_MAX - 1 || bottom > U16_MAX)
+		return -EINVAL;
+

[Severity: High]
This is a pre-existing issue, but does this validation block still allow a divide-by-zero panic if ulHeight is 1?

If a user passes ySize = 1 via the KYRO_IOCTL_OVERLAY_VIEWPORT_SET ioctl, ulHeight will be 1. The code calculates bottom = y + ulHeight, then bottom--, which results in bottom == y.

When these values are passed to SetOverlayViewPort(), it calculates ulDest as the difference between the y coordinates (bottom - top). Since they are equal, ulDest evaluates to 0:

drivers/video/fbdev/kyro/STG4000OverlayDevice.c:SetOverlayViewPort() {
     ...
     ulDest = srcDest.lDstY2 - srcDest.lDstY1;
     ...
     ulFxScale = (ulDest << 11) / ulSrc;
     ...
     ulSrcBottom = ulSrcBottom - (ulFxOffset / ulFxScale);
     ...
}
Quoted 6 lines

Because ulDest is 0, ulFxScale evaluates to 0. The subsequent division by ulFxScale then causes a kernel panic.

Should ulHeight == 1 be rejected alongside the ulHeight == 0 check earlier in kyro_dev_overlay_viewport_set()?

Thanks for the review. The reported division-by-zero issue is pre-existing in the codebase and is completely unrelated to my changes. This patch does not introduce, touch, or modify this logic.

I request the maintainers to review my patch as is.

Best regards,
Danila Chernetsov

maintainer Applied
#3 lore raw
In reply to Danila Chernetsov
Quoted 12 lines · Danila Chernetsov

The overlay viewport end coordinates are computed from the viewport origin and dimensions using 32-bit unsigned arithmetic. Large input values can cause these calculations to wrap around before the resulting coordinates are passed to SetOverlayViewPort().

SetOverlayViewPort() packs the viewport coordinates into 16-bit register fields. The X coordinates are additionally adjusted by +2 and +1 before being written. Validate the coordinate calculations for 32-bit wraparound and ensure that the adjusted coordinates fit within their 16-bit register fields before calling SetOverlayViewPort().

Quoted 7 lines

Found by Linux Verification Center (linuxtesting.org) with SVACE.

Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")
---
  drivers/video/fbdev/kyro/fbdev.c | 24 +++++++++++++++++++++++-
  1 file changed, 23 insertions(+), 1 deletion(-)

applied.

Thanks!
Helge