vgacon: Disallow console operations when in KD_GRAPHICS mode

Reported by James Pearson as:

boot to run level 3

if not root, then make sure /dev/console is writeable

login and type:

setterm -blank 0

start X

type into an xterm:

while true; do echo "" > /dev/console; usleep 100000; done

while the above loop is running switch to the text console and back again (Ctrl-Alt-F1 then Ctrl-Alt-F7)

... and the screen will be shifting (and wrapping) to the left.

This problem stems from continuously writing text to the system console (which is in KD_TEXT mode) while the foreground console is in KD_GRAPHICS mode. Somewhere along the way, console printing got confused and omitted the KD_GRAPHICS/KD_TEXT test. Thus, vgacon attempted to scroll the screen of X, which causes X to shift.

Fix by disallowing vgacon to touch the hardware when the vc is in KD_GRAPHICS mode. A definitive fix entails a full audit of the console code.

Tony
---

 drivers/video/console/vgacon.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

index d0d2733..2460b82 100644
@@ -660,6 +660,9 @@ #endif
 
 static void vgacon_cursor(struct vc_data *c, int mode)
 {
+	if (c->vc_mode != KD_TEXT)
+		return;
+
 	vgacon_restore_screen(c);
 
 	switch (mode) {

@@ -1318,7 +1321,7 @@ static int vgacon_scroll(struct vc_data 
 	unsigned long oldo;
 	unsigned int delta;
 
-	if (t || b != c->vc_rows || vga_is_gfx)
+	if (t || b != c->vc_rows || vga_is_gfx || c->vc_mode != KD_TEXT)
 		return 0;
 
 	if (!vga_hardscroll_enabled || lines >= c->vc_rows / 2)

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/