vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()

PATCH RESEND In review Xi Wang 3 participants 6 messages Dec 20, 2011 – Dec 21, 2011

Subsystems (3)

DRM drivers DRM misc DRM driver for VMware virtual GPU

lore

Share & tools

lore

Xi Wang

author

2011-12-20 21:08 UTC #0 lore raw

PATCH RESEND 1/1 vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()

The previous commit didn't correctly fix the integer overflow issue.

http://git.kernel.org/linus/e133e737

- unsigned int required_size;

+	u64 required_size;
	...
	required_size = mode_cmd->pitch * mode_cmd->height;

- if (unlikely(required_size > dev_priv->vram_size)) {
+ if (unlikely(required_size > (u64) dev_priv->vram_size)) {

Note that both pitch and height are u32, their product is still u32 and would overflow before being assigned to required_size. A correct way is to convert pitch and height to u64 before the multiplication.

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls an existing function vmw_kms_validate_mode_vram() for validation.

Signed-off-by Xi Wang <[email protected]>

---
 vmwgfx_kms.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

vmwgfx_kms.c +3 -3

index b87afdf..6b8857e 100644
@@ -1101,7 +1101,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
 	struct vmw_surface *surface = NULL;
 	struct vmw_dma_buffer *bo = NULL;
 	struct ttm_base_object *user_obj;
-	u64 required_size;
 	int ret;
 
 	/**

@@ -1110,8 +1109,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
 	 * requested framebuffer.
 	 */
 
-	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size > (u64) dev_priv->vram_size)) {
+	if (!vmw_kms_validate_mode_vram(dev_priv,
+					mode_cmd->pitch,
+					mode_cmd->height)) {
 		DRM_ERROR("VRAM size is too small for requested mode.\n");
 		return ERR_PTR(-ENOMEM);
 	}

Signature · 1 line

1.7.5.4

David Airlie

2011-12-21 09:30 UTC #1 lore raw

----- Original Message -----

Quoted 5 lines

From: "Xi Wang" <[email protected]>
To: "Jakob Bornecrantz" <[email protected]>, "Thomas Hellstrom" <[email protected]>
Cc: [email protected], "Dave Airlie" <[email protected]>
Sent: Tuesday, 20 December, 2011 9:08:32 PM
Subject: [PATCH RESEND] vmwgfx: fix incorrect vram size check in vmw_kms_fb_create()

This patch doesn't apply with git am, please regenerate it,

Thomas, can you ack as well please?

Is this for -fixes or -next?

Dave.

Xi Wang

author

2011-12-21 09:34 UTC #2 lore raw

Quoted 6 lines · David Airlie

This patch doesn't apply with git am, please regenerate it,
Thomas, can you ack as well please?
Is this for -fixes or -next?

Sorry this patch is for vmwgfx.git. I will redo a version that can be applied to the mainline kernel.

- xi

Xi Wang

author

2011-12-21 10:18 UTC #3 lore raw

PATCH 1/1 vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()

In reply to David Airlie

Commit e133e737 didn't correctly fix the integer overflow issue.

- unsigned int required_size;

+	u64 required_size;
	...
	required_size = mode_cmd->pitch * mode_cmd->height;

- if (unlikely(required_size > dev_priv->vram_size)) {
+ if (unlikely(required_size > (u64) dev_priv->vram_size)) {

Note that both pitch and height are u32. Their product is still u32 and would overflow before being assigned to required_size. A correct way is to convert pitch and height to u64 before the multiplication.

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for validation.

Signed-off-by Xi Wang <[email protected]>

---
 drivers/gpu/drm/vmwgfx/vmwgfx_kms.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

drivers/gpu/drm/vmwgfx/vmwgfx_kms.c +3 -3

index 8aa1dbb..f94b33a 100644
@@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
 	struct vmw_surface *surface = NULL;
 	struct vmw_dma_buffer *bo = NULL;
 	struct ttm_base_object *user_obj;
-	u64 required_size;
 	int ret;
 
 	/**

@@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
 	 * requested framebuffer.
 	 */
 
-	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size > (u64) dev_priv->vram_size)) {
+	if (!vmw_kms_validate_mode_vram(dev_priv,
+					mode_cmd->pitch,
+					mode_cmd->height)) {
 		DRM_ERROR("VRAM size is too small for requested mode.\n");
 		return ERR_PTR(-ENOMEM);
 	}

Signature · 1 line

1.7.5.4

Thomas Hellstrom

2011-12-21 12:26 UTC #4 lore raw

Re: [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()

This looks good, although I want to do a quick test to verify that it doesn't break anything.

I'll get back as soon as possible.

/Thomas

Quoted 48 lines · Xi Wang

Commit e133e737 didn't correctly fix the integer overflow issue.

(quoted diff) +1 -1

-	unsigned int required_size;
+	u64 required_size;
...

	required_size = mode_cmd->pitch * mode_cmd->height;

(quoted diff) +1 -1

-	if (unlikely(required_size>  dev_priv->vram_size)) {
+	if (unlikely(required_size>  (u64) dev_priv->vram_size)) {

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for validation.

Signed-off-by Xi Wang<[email protected]>

---
  drivers/gpu/drm/vmwgfx/vmwgfx_kms.c |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

drivers/gpu/drm/vmwgfx/vmwgfx_kms.c +3 -3

index 8aa1dbb..f94b33a 100644
@@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
  	struct vmw_surface *surface = NULL;
  	struct vmw_dma_buffer *bo = NULL;
  	struct ttm_base_object *user_obj;
-	u64 required_size;
  	int ret;
 
  	/**

@@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
  	 * requested framebuffer.
  	 */
 
-	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size>  (u64) dev_priv->vram_size)) {
+	if (!vmw_kms_validate_mode_vram(dev_priv,
+					mode_cmd->pitch,
+					mode_cmd->height)) {
  		DRM_ERROR("VRAM size is too small for requested mode.\n");
  		return ERR_PTR(-ENOMEM);
  	}

Thomas Hellstrom

2011-12-21 21:36 UTC #5 lore raw

Re: [PATCH -fixes] vmwgfx: fix incorrect VRAM size check in vmw_kms_fb_create()

In reply to Xi Wang

Reviewed- and tested by

Thomas Hellstrom <[email protected]>

Thanks,
Thomas

Quoted 48 lines · Xi Wang

Commit e133e737 didn't correctly fix the integer overflow issue.

(quoted diff) +1 -1

-	unsigned int required_size;
+	u64 required_size;
...

	required_size = mode_cmd->pitch * mode_cmd->height;

(quoted diff) +1 -1

-	if (unlikely(required_size>  dev_priv->vram_size)) {
+	if (unlikely(required_size>  (u64) dev_priv->vram_size)) {

	required_size = (u64)mode_cmd->pitch * (u64)mode_cmd->height;

This patch calls the existing vmw_kms_validate_mode_vram() for validation.

Signed-off-by Xi Wang<[email protected]>

---
  drivers/gpu/drm/vmwgfx/vmwgfx_kms.c |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

drivers/gpu/drm/vmwgfx/vmwgfx_kms.c +3 -3

index 8aa1dbb..f94b33a 100644
@@ -1093,7 +1093,6 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
  	struct vmw_surface *surface = NULL;
  	struct vmw_dma_buffer *bo = NULL;
  	struct ttm_base_object *user_obj;
-	u64 required_size;
  	int ret;
 
  	/**

@@ -1102,8 +1101,9 @@ static struct drm_framebuffer *vmw_kms_fb_create(struct drm_device *dev,
  	 * requested framebuffer.
  	 */
 
-	required_size = mode_cmd->pitch * mode_cmd->height;
-	if (unlikely(required_size>  (u64) dev_priv->vram_size)) {
+	if (!vmw_kms_validate_mode_vram(dev_priv,
+					mode_cmd->pitch,
+					mode_cmd->height)) {
  		DRM_ERROR("VRAM size is too small for requested mode.\n");
  		return ERR_PTR(-ENOMEM);
  	}