Fix the graphic corruption issue on IA64 machines

Hi,

I encountered the graphic corruption issue on IA64 machines by the following operations.

0. enable VGA console(default setting). disable serial console setting.
1. boot the system at run level 3 and login via text-mode console
(/dev/tty1)
as the root user.
2. disable console blanking.

    # setterm -blank 0
 3. start X.
 4. shutdown X.
 5. start X again.
 6. open the gnome-terminal and write someting to /dev/console.
    # ls -l  > /dev/console

The cause of this problem may be VGA console driver's misunderstanding mode(text/graphic).

I confirmed this problem is fixed by the attached patch, but I don't know this is the correct fix.

--------------------------------------------------------------------

VGA console driver can misunderstand the current mode(Text/Graphic) under "disable console blanking" setting.

- start X
--> vt_ioctl() drivers/char/vt_ioctl.c

        do_blank_screen(1)     drivers/char/vt.c
           sw->con_blank(vc_cons[currcons].d, 1, 0)  drivers/char/vt.c
       vgacon_blank()     drivers/video/console/vgacon.c

　 vga_is_gfx = 1 /* enter Graphic mode */

- shutdown X
--> vt_ioctl()

          do_unblank_screen(1)    drivers/char/vt.c
            sw->con_blank(vc_cons[currcons].d, 0, leaving_gfx)
        vgacon_blank()

　 vga_is_gfx = 0 /* leave Graphic mode */

When "disable console blank" is set (=> blankinterval=0), "do_unblank_screen()" function
returns without changing "blank_state", and when "blank_state" is "blank_off",
"do_blank_screen() function returns without invoking sw->con_blank() function.
That's why VGA console driver can misunderstand the current mode.

-----------------------------------------------------------------------

Regards,
Taku Izumi

Fix the graphic corruption issue on IA64 machines.
VGA console driver can misunderstand the current mode(Text/Graphic) under
"disable console blanking" setting. When "disable console blank" is set (blankinterval=0), "do_unblank_screen()" function returns without changing "blank_state", and when "blank_state" is "blank_off", "do_blank_screen() function returns without invoking sw->con_blank() function. That's why VGA console driver can misunderstand the current mode.

---
 drivers/char/vt.c |    2 +-
 1 files changed, 1 insertion(+), 1 deletion(-)

Index: linux-2.6.21.5/drivers/char/vt.c
========================================================================
@@ -3419,12 +3419,12 @@ void do_unblank_screen(int leaving_gfx)
 		return;
 	}
 	vc = vc_cons[fg_console].d;
+	blank_state = blank_normal_wait;
 	if (vc->vc_mode != KD_TEXT)
 		return; /* but leave console_blanked != 0 */
 
 	if (blankinterval) {
 		mod_timer(&console_timer, jiffies + blankinterval);
-		blank_state = blank_normal_wait;
 	}
 
 	console_blanked = 0;

HI,

As a result of the discussion with Pete Zaitcev([email protected]), I re-create a patch. This attached patch is revised version.

He pointed out that the former patch may violate the assumptions and was not safe. Concretely speaking, he concerned that an unexpected problem may arise somewhere if "blank_state", which is intended to reflect the state of timer, was shuffled arround.

This revised patch reflects his pointed out. I confirmed this also fixed the problem.

Regards,
Taku Izumi <[email protected]>

Fix the graphic corruption issue on IA64 machines.
VGA console driver can misunderstand the current mode(Text/Graphic) under "disable console blanking" setting. When "disable console blank" is set (blankinterval=0), "do_unblank_screen()" function returns without changing "blank_state", and when "blank_state" is "blank_off", "do_blank_screen() function returns without invoking sw->con_blank() function. That's why VGA console driver can misunderstand the current mode.

---
 drivers/char/vt.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)

Index: linux-2.6.22/drivers/char/vt.c
=============================================================================
@@ -3491,9 +3491,6 @@ void do_blank_screen(int entering_gfx)
 		}
 		return;
 	}
-	if (blank_state != blank_normal_wait)
-		return;
-	blank_state = blank_off;
 
 	/* entering graphics mode? */
 	if (entering_gfx) {

@@ -3501,10 +3498,15 @@ void do_blank_screen(int entering_gfx)
 		save_screen(vc);
 		vc->vc_sw->con_blank(vc, -1, 1);
 		console_blanked = fg_console + 1;
+		blank_state = blank_off;
 		set_origin(vc);
 		return;
 	}
 
+	if (blank_state != blank_normal_wait)
+		return;
+	blank_state = blank_off;
+
 	/* don't blank graphics */
 	if (vc->vc_mode != KD_TEXT) {
 		console_blanked = fg_console + 1;