Skip to contentSkip to conversation

Another memory leak in drivers/char/vt.c

PATCH In review Catalin Marinas 4 participants 7 messages Jul 29, 2009 – Aug 1, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

There was a memory leak fixed recently by commit 1a8f458f6d. However, there seems to be another with this kmemleak trace:

unreferenced object 0xde158000 (size 12288):
comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4

The problem happens in the vc_allocate() function where vc->vc_screenbuf is set to the kmalloc() returned value. However, the visual_init() function called 3 lines before also allocates the vc->vc_screenbuf.

One solution is below (another would be to kfree the vc_screenbuf and reallocate):

drivers/char/vt.c +3 -1
index 404f4c1..1da75ef 100644
@@ -770,7 +770,9 @@ int vc_allocate(unsigned int currcons)	/* return 0 on success */
 	    visual_init(vc, currcons, 1);
 	    if (!*vc->vc_uni_pagedir_loc)
 		con_set_default_unimap(vc);
-	    vc->vc_screenbuf = kmalloc(vc->vc_screenbuf_size, GFP_KERNEL);
+	    if (!vc->vc_screenbuf)
+		    vc->vc_screenbuf = kmalloc(vc->vc_screenbuf_size,
+					       GFP_KERNEL);
 	    if (!vc->vc_screenbuf) {
 		kfree(vc);
 		vc_cons[currcons].d = NULL;

Thanks.

Signature · 1 line
Catalin
#1 lore raw

Hello Catalin,

Quoted 28 lines · Catalin Marinas

Hi,

There was a memory leak fixed recently by commit 1a8f458f6d. However, there seems to be another with this kmemleak trace:

unreferenced object 0xde158000 (size 12288):
comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4
Quoted 3 lines

The problem happens in the vc_allocate() function where vc->vc_screenbuf is set to the kmalloc() returned value. However, the visual_init() function called 3 lines before also allocates the vc->vc_screenbuf.

The common way seems to be that ->con_init(init=1) just sets the dimensions manually (instead of using vc_resize()) and vc_allocate() uses them to actually allocate a properly sized screen buffer.

So it seems like fbcon is at fault here. It calls vc_resize() from ->con_init(init=1) and updates the console dimensions manually on init=0 (after calling vc_resize()) which is completely mixed up.

I think the quick fix is something like the appended (untested!).

In the long run it is probably good to re-evaluate whether vc_resize() can be called unconditionally from ->con_init() and remove the allocation from vc_allocate().

Hannes

drivers/video/console/fbcon.c +3 -3
index 471a9a6..3a44695 100644
@@ -1082,7 +1082,6 @@ static void fbcon_init(struct vc_data *vc, int init)
 	new_rows = FBCON_SWAP(ops->rotate, info->var.yres, info->var.xres);
 	new_cols /= vc->vc_font.width;
 	new_rows /= vc->vc_font.height;
-	vc_resize(vc, new_cols, new_rows);
 
 	/*
 	 * We must always set the mode. The mode of the previous console
@@ -1111,10 +1110,11 @@ static void fbcon_init(struct vc_data *vc, int init)
 	 *  vc_{cols,rows}, but we must not set those if we are only
 	 *  resizing the console.
 	 */
-	if (!init) {
+	if (init) {
 		vc->vc_cols = new_cols;
 		vc->vc_rows = new_rows;
-	}
+	} else
+		vc_resize(vc, new_cols, new_rows);
 
 	if (logo)
 		fbcon_prepare_logo(vc, info, cols, rows, new_cols, new_rows);
author Tested-by
#2 lore raw
Quoted 35 lines · Johannes Weiner
Quoted 26 lines · Catalin Marinas

There was a memory leak fixed recently by commit 1a8f458f6d. However, there seems to be another with this kmemleak trace:

unreferenced object 0xde158000 (size 12288):
comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4
Quoted 3 lines

The problem happens in the vc_allocate() function where vc->vc_screenbuf is set to the kmalloc() returned value. However, the visual_init() function called 3 lines before also allocates the vc->vc_screenbuf.

The common way seems to be that ->con_init(init=1) just sets the dimensions manually (instead of using vc_resize()) and vc_allocate() uses them to actually allocate a properly sized screen buffer.

Quoted 5 lines

So it seems like fbcon is at fault here. It calls vc_resize() from ->con_init(init=1) and updates the console dimensions manually on init=0 (after calling vc_resize()) which is completely mixed up.

I think the quick fix is something like the appended (untested!).

This patch seems to fix the leak, you can add

Thanks.

Signature · 1 line
Catalin
#3 lore raw

PATCH 1/1 fbcon: don't use vc_resize() on initialization

Catalin and kmemleak spotted a leak of a VC screen buffer in vc_allocate() due to the following chain of events:

	vc_allocate()
	  visual_init(init=1)
	    vc->vc_sw->con_init(init=1)
              fbcon_init()
	        vc_resize()
	          vc->screen_buf = kmalloc()
	  vc->screen_buf = kmalloc()

The common way for the VC drivers is to set the screen dimension parameters manually in the init case and only call vc_resize() for !init - which allocates a screen buffer according to the new dimensions.

fbcon instead would do vc_resize() unconditionally and afterwards set the dimensions manually (again) for !init - i.e. completely upside down. The vc_resize() allocated buffer would then get lost by vc_allocate() allocating a fresh one.

Use vc_resize() only for actual resizing to close the leak.

Set the dimensions manually only in initialization mode to remove the redundant setting in resize mode.

The kmemleak trace from Catalin:

unreferenced object 0xde158000 (size 12288):
comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4
---
 drivers/video/console/fbcon.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/video/console/fbcon.c +3 -3
index 471a9a6..3a44695 100644
@@ -1082,7 +1082,6 @@ static void fbcon_init(struct vc_data *vc, int init)
 	new_rows = FBCON_SWAP(ops->rotate, info->var.yres, info->var.xres);
 	new_cols /= vc->vc_font.width;
 	new_rows /= vc->vc_font.height;
-	vc_resize(vc, new_cols, new_rows);
 
 	/*
 	 * We must always set the mode. The mode of the previous console
@@ -1111,10 +1110,11 @@ static void fbcon_init(struct vc_data *vc, int init)
 	 *  vc_{cols,rows}, but we must not set those if we are only
 	 *  resizing the console.
 	 */
-	if (!init) {
+	if (init) {
 		vc->vc_cols = new_cols;
 		vc->vc_rows = new_rows;
-	}
+	} else
+		vc_resize(vc, new_cols, new_rows);
 
 	if (logo)
 		fbcon_prepare_logo(vc, info, cols, rows, new_cols, new_rows);
Signature · 1 line
1.6.3
#4 lore raw

Re: [patch] fbcon: don't use vc_resize() on initialization

Quoted 21 lines · On Wed, 29 Jul 2009 19:21:23 +0200 Johannes Weiner <[email protected]> wrote:

Catalin and kmemleak spotted a leak of a VC screen buffer in vc_allocate() due to the following chain of events:

	vc_allocate()
	  visual_init(init=1)
	    vc->vc_sw->con_init(init=1)
              fbcon_init()
	        vc_resize()
	          vc->screen_buf = kmalloc()
	  vc->screen_buf = kmalloc()

The common way for the VC drivers is to set the screen dimension parameters manually in the init case and only call vc_resize() for !init - which allocates a screen buffer according to the new dimensions.

fbcon instead would do vc_resize() unconditionally and afterwards set the dimensions manually (again) for !init - i.e. completely upside down. The vc_resize() allocated buffer would then get lost by vc_allocate() allocating a fresh one.

Quoted 8 lines

Use vc_resize() only for actual resizing to close the leak.

Set the dimensions manually only in initialization mode to remove the redundant setting in resize mode.

The kmemleak trace from Catalin:

unreferenced object 0xde158000 (size 12288):

That's a big leak!

What sequence of user actions would cause it to occur?

Quoted 56 lines

comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4
---
 drivers/video/console/fbcon.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/video/console/fbcon.c +3 -3
index 471a9a6..3a44695 100644
@@ -1082,7 +1082,6 @@ static void fbcon_init(struct vc_data *vc, int init)
 	new_rows = FBCON_SWAP(ops->rotate, info->var.yres, info->var.xres);
 	new_cols /= vc->vc_font.width;
 	new_rows /= vc->vc_font.height;
-	vc_resize(vc, new_cols, new_rows);
 
 	/*
 	 * We must always set the mode. The mode of the previous console
@@ -1111,10 +1110,11 @@ static void fbcon_init(struct vc_data *vc, int init)
 	 *  vc_{cols,rows}, but we must not set those if we are only
 	 *  resizing the console.
 	 */
-	if (!init) {
+	if (init) {
 		vc->vc_cols = new_cols;
 		vc->vc_rows = new_rows;
-	}
+	} else
+		vc_resize(vc, new_cols, new_rows);
 
 	if (logo)
 		fbcon_prepare_logo(vc, info, cols, rows, new_cols, new_rows);
#5 lore raw

Re: [patch] fbcon: don't use vc_resize() on initialization

Quoted 29 lines · Andrew Morton
Quoted 26 lines · On Wed, 29 Jul 2009 19:21:23 +0200 Johannes Weiner <[email protected]> wrote:

Catalin and kmemleak spotted a leak of a VC screen buffer in vc_allocate() due to the following chain of events:

	vc_allocate()
	  visual_init(init=1)
	    vc->vc_sw->con_init(init=1)
              fbcon_init()
	        vc_resize()
	          vc->screen_buf = kmalloc()
	  vc->screen_buf = kmalloc()

The common way for the VC drivers is to set the screen dimension parameters manually in the init case and only call vc_resize() for !init - which allocates a screen buffer according to the new dimensions.

fbcon instead would do vc_resize() unconditionally and afterwards set the dimensions manually (again) for !init - i.e. completely upside down. The vc_resize() allocated buffer would then get lost by vc_allocate() allocating a fresh one.

Use vc_resize() only for actual resizing to close the leak.

Set the dimensions manually only in initialization mode to remove the redundant setting in resize mode.

Quoted 7 lines
Quoted 3 lines

The kmemleak trace from Catalin:

unreferenced object 0xde158000 (size 12288):

That's a big leak!

What sequence of user actions would cause it to occur?

In my case, I think anything that causes a tty_open() call with a framebuffer console. I get a leak when logging in on the console and another when starting the X server.

Signature · 1 line
Catalin
Tested-by on v1 1/1
#6 lore raw

Re: [patch] fbcon: don't use vc_resize() on initialization

In reply to Johannes Weiner
Quoted 92 lines · Johannes Weiner

Catalin and kmemleak spotted a leak of a VC screen buffer in vc_allocate() due to the following chain of events:

	vc_allocate()
	  visual_init(init=1)
	    vc->vc_sw->con_init(init=1)
              fbcon_init()
	        vc_resize()
	          vc->screen_buf = kmalloc()
	  vc->screen_buf = kmalloc()

The common way for the VC drivers is to set the screen dimension parameters manually in the init case and only call vc_resize() for !init - which allocates a screen buffer according to the new dimensions.

fbcon instead would do vc_resize() unconditionally and afterwards set the dimensions manually (again) for !init - i.e. completely upside down. The vc_resize() allocated buffer would then get lost by vc_allocate() allocating a fresh one.

Use vc_resize() only for actual resizing to close the leak.

Set the dimensions manually only in initialization mode to remove the redundant setting in resize mode.

The kmemleak trace from Catalin:

unreferenced object 0xde158000 (size 12288):
comm "Xorg", pid 1439, jiffies 4294961016
hex dump (first 32 bytes):

    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
    20 00 20 00 20 00 20 00 20 00 20 00 20 00 20 00   . . . . . . . .
  backtrace:
    [<c006f74b>] __save_stack_trace+0x17/0x1c
    [<c006f81d>] create_object+0xcd/0x188
    [<c01f5457>] kmemleak_alloc+0x1b/0x3c
    [<c006e303>] __kmalloc+0xdb/0xe8
    [<c012cc4b>] vc_do_resize+0x73/0x1e0
    [<c012cdf1>] vc_resize+0x15/0x18
    [<c011afc1>] fbcon_init+0x1f9/0x2b8
    [<c0129e87>] visual_init+0x9f/0xdc
    [<c012aff3>] vc_allocate+0x7f/0xfc
    [<c012b087>] con_open+0x17/0x80
    [<c0120e43>] tty_open+0x1f7/0x2e4
    [<c0072fa1>] chrdev_open+0x101/0x118
    [<c006ffad>] __dentry_open+0x105/0x1cc
    [<c00700fd>] nameidata_to_filp+0x2d/0x38
    [<c00788cd>] do_filp_open+0x2c1/0x54c
    [<c006fdff>] do_sys_open+0x3b/0xb4
---
 drivers/video/console/fbcon.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
drivers/video/console/fbcon.c +3 -3
index 471a9a6..3a44695 100644
@@ -1082,7 +1082,6 @@ static void fbcon_init(struct vc_data *vc, int init)
 	new_rows = FBCON_SWAP(ops->rotate, info->var.yres, info->var.xres);
 	new_cols /= vc->vc_font.width;
 	new_rows /= vc->vc_font.height;
-	vc_resize(vc, new_cols, new_rows);
 
 	/*
 	 * We must always set the mode. The mode of the previous console
@@ -1111,10 +1110,11 @@ static void fbcon_init(struct vc_data *vc, int init)
 	 *  vc_{cols,rows}, but we must not set those if we are only
 	 *  resizing the console.
 	 */
-	if (!init) {
+	if (init) {
 		vc->vc_cols = new_cols;
 		vc->vc_rows = new_rows;
-	}
+	} else
+		vc_resize(vc, new_cols, new_rows);
 
 	if (logo)
 		fbcon_prepare_logo(vc, info, cols, rows, new_cols, new_rows);
Signature · 6 lines
1.6.3
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/