Skip to contentSkip to conversation

Kernel utf-8 handling

PATCH New DervishD 7 participants 19 messages Jun 1, 2007 – Jun 2, 2007

Subsystems (2)
lore
Share & tools
lore

DervishD

author
#0 lore raw

Hi all :)

I have a do-it-yourself Linux box, and I'm planning to move to UTF8
(currently I'm using es_ES locale, with latin1 encoding). One of my main concerns (apart from programs with little or no utf8 support, which I will have to suffer) is kernel handling, because I only use the console; I only use X and a terminal emulator if I can't avoid it.

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like 'ñ' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

Will the console work as it works now if I can live with latin1
accented characters only? Is there any terminal emulator *for the console*, not for X, that handles utf8? Will I be sentenced to X to be able to use my computer with utf8?

Don't take me bad, I really would love to spend 100MiB of RAM just
to run mutt under xterm, but for the time being I prefer the console to work even if I must run X...

Thanks a lot! :))

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#1 lore raw
Quoted 8 lines · DervishD

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like '?' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

Hmmm. I've not yet played with moving to utf-8 but would

man console_codes

be of help to you?

Signature · 3 lines
    "To the extent that we overreact, we proffer the terrorists the
    greatest tribute."
    	- High Court Judge Michael Kirby
author
#2 lore raw

Hi CaT :)

* CaT <[email protected]> dixit:

Quoted 10 lines · DervishD
Quoted 8 lines

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like '?' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

Quoted 5 lines

Hmmm. I've not yet played with moving to utf-8 but would

man console_codes

be of help to you?

Not, because I already know the escape sequence to go back and forth
to utf-8 mode O:)) I'm going to carry some tests using Ubuntu, but since that is a distro I'm not sure if I will be able to apply the same solutions and mechanisms to my do-it-yourself box.

Thanks anyway for answering :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#3 lore raw
In reply to DervishD
Quoted 1 line · DervishD wrote/

Hi all :)

Hi!

Quoted 3 lines

Will the console work as it works now if I can live with latin1
accented characters only?

Just tested here, it _seems_ to work right on the console with Spanish and French accentuated characters.

Quoted 2 lines

Is there any terminal emulator *for the
console*, not for X, that handles utf8?

fbiterm, I never dared to try though...

See you,
Eric

author
#4 lore raw

Hi Éric :)

* Éric Piel <[email protected]> dixit:

Quoted 5 lines
Quoted 2 lines · DervishD wrote/

Will the console work as it works now if I can live with latin1
accented characters only?

Just tested here, it _seems_ to work right on the console with Spanish and French accentuated characters.

Enough for me, thanks!

Quoted 3 lines
Quoted 2 lines

Is there any terminal emulator *for the
console*, not for X, that handles utf8?

fbiterm, I never dared to try though...

I'll give it a try ;), thanks a lot :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#5 lore raw
In reply to DervishD
Quoted 7 lines · DervishD

Hi all :)

I have a do-it-yourself Linux box, and I'm planning to move to UTF8
(currently I'm using es_ES locale, with latin1 encoding). One of my main concerns (apart from programs with little or no utf8 support, which I will have to suffer) is kernel handling, because I only use the console; I only use X and a terminal emulator if I can't avoid it.

The switch is possible. You could try the latest development LFS LiveCD (http://ums.usu.ru/~patrakov/test/lfslivecd-x86-6.3-pre2-r1897.iso) and see if it works for you (be sure to CC: me if you post any feedback). It will automatically configure the console using a simple menu-driven interface. Some interesting programs to try there: mutt, tin, lynx, finch.

Quoted 8 lines

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like 'ñ' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

Yes.

Quoted 4 lines

Will the console work as it works now if I can live with latin1
accented characters only? Is there any terminal emulator *for the console*, not for X, that handles utf8? Will I be sentenced to X to be able to use my computer with utf8?

screen, jfbterm, iterm (jfbterm and iterm are not on my CD, but you can build them from source there - the filesystem on the CD can be written to). Note that jfbterm and iterm expect X fonts (or unifont) to be available, and the CD doesn't have these fonts (fully switched to Xft).

Signature · 1 line
Alexander E. Patrakov
#6 lore raw
Quoted 1 line · I wrote:

The switch is possible. You could try the latest development LFS LiveCD (http://ums.usu.ru/~patrakov/test/lfslivecd-x86-6.3-pre2-r1897.iso) and see if it works for you (be sure to CC: me if you post any feedback).

I was wrong. The problem is that input works, output works, but copying-and-pasting with gpm doesn't work (the stable version of the CD, http://ums.usu.ru/~patrakov/x86/lfslivecd-x86-6.2-5.iso, contains a rejected-upstream kernel patch that hides the issue). This means that for all serious Unicode work, you must use X.

Signature · 1 line
Alexander E. Patrakov
author
#7 lore raw

Hi Alexander :)

* Alexander E. Patrakov <[email protected]> dixit:

Quoted 9 lines
Quoted 3 lines

The switch is possible. You could try the latest development LFS LiveCD
(http://ums.usu.ru/~patrakov/test/lfslivecd-x86-6.3-pre2-r1897.iso) and see if it works for you (be sure to CC: me if you post any feedback).

I was wrong. The problem is that input works, output works, but copying-and-pasting with gpm doesn't work (the stable version of the CD,
http://ums.usu.ru/~patrakov/x86/lfslivecd-x86-6.2-5.iso, contains a rejected-upstream kernel patch that hides the issue). This means that for all serious Unicode work, you must use X.

Not a problem for me because I don't use GPM. I hate mice, and while
under X I'm forced (more or less) to use them, I don't use a mouse on the console. I can blind type, so I prefer to keep my hands on the keyboard. I work much more fast that way, and I haven't needed to copy and paste between consoles ever.

Thanks anyway for the warning ;)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#8 lore raw
Quoted 3 lines · DervishD

Not a problem for me because I don't use GPM. I hate mice, and while
under X I'm forced (more or less) to use them,

Use the ratpoison WM then.

Jan

author
#9 lore raw
In reply to Alexander E. Patrakov

Hi Alexander :)

* Alexander E. Patrakov <[email protected]> dixit:

Quoted 7 lines
Quoted 5 lines

I have a do-it-yourself Linux box, and I'm planning to move to UTF8
(currently I'm using es_ES locale, with latin1 encoding). One of my main concerns (apart from programs with little or no utf8 support, which I will have to suffer) is kernel handling, because I only use the console; I only use X and a terminal emulator if I can't avoid it.

The switch is possible.

The switch to utf-8 is possible, I know, but I wasn't so sure about
the switch to utf-8 *in the console only*.

Quoted 6 lines

You could try the latest development LFS LiveCD
(http://ums.usu.ru/~patrakov/test/lfslivecd-x86-6.3-pre2-r1897.iso)
and see if it works for you (be sure to CC: me if you post any feedback). It will automatically configure the console using a simple menu-driven interface. Some interesting programs to try there: mutt, tin, lynx, finch.

I was going to take a look at Linux From Scratch (and BLFS, too) for
help, but before trying anything there I wanted to be sure that the kernel will work. Otherwise I will put my effort in building a minimal X setup with a suitable terminal emulator.

Anyway, if everything works in the console in the LFS LiveCD, I will
be sure that I can make the change and then I'll look directly in the LFS book for further instructions.

Thanks a lot! :)

Quoted 6 lines
Quoted 4 lines

Will the console work as it works now if I can live with latin1
accented characters only? Is there any terminal emulator *for the console*, not for X, that handles utf8? Will I be sentenced to X to be able to use my computer with utf8?

screen,

GNU screen? I'll give it a try.

Quoted 2 lines

jfbterm, iterm (jfbterm and iterm are not on my CD, but you can build them from source there - the filesystem on the CD can be written to).

There are more terminal emulators for console than I supposed.

Quoted 2 lines

Note that jfbterm and iterm expect X fonts (or unifont) to be available, and the CD doesn't have these fonts (fully switched to Xft).

Well, I'll deal with that issue if I like them more than screen or
plain console.

Thanks again :)

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#10 lore raw
In reply to DervishD
Quoted 9 lines · DervishD

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like 'ñ' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

so not sure where your problem is. I do however have a patch that you could try should a problem arise. I should repost and ask around again and beat until it's in :)

Jan

#11 lore raw
Quoted 5 lines · Jan Engelhardt

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

Both of those are in the 0-255 range, though. I thought the issue was with characters 256+, like č. At least on my FC6 system that doesn't work with gpm.

-hpa

author
#12 lore raw

Hi H. Peter :)

* H. Peter Anvin <[email protected]> dixit:

Quoted 8 lines
Quoted 3 lines · Jan Engelhardt

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

Both of those are in the 0-255 range, though. I thought the issue was with characters 256+, like ??. At least on my FC6 system that doesn't work with gpm.

By now, to fully switch to utf-8, I only need to have áéíóúñ and
their uppercase counterparts (for people who cannot see the chars, they are the acute accented vowels and the ntilde). I really hope that, in the future, there are a way of fully use Unicode (and I mean the entire range) using framebuffer in console (there probably is an emulator for that job already, I have to check), but for the time being that's enough.

I really don't mind if I have to use an userspace program to have
unicode support on the console (the kernel shouldn't mess with encodings, or have utf-8 only; I don't know the current status, that's why I asked), but if the kernel does directly the job in the console driver, that's not bad either.

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#13 lore raw
Quoted 10 lines · DervishD

* H. Peter Anvin <[email protected]> dixit:

Quoted 8 lines
Quoted 3 lines · Jan Engelhardt

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

Both of those are in the 0-255 range, though. I thought the issue was with characters 256+, like ??. At least on my FC6 system that doesn't work with gpm.

I've got a cp437 "DOS" font loaded[1], and giving a glance at /usr/share/kbd/unimaps/cp437.uni tells me right away:

	0x01 U+263A

So I create U+263A in joe (ESC ' x 2 6 3 a), it displays fine (the smiley character), and I can copy/paste it using GPM without problems. Though, I have that patch[2] in my kernel. Can you check if it works if you apply it?

[1] https://dev.computergmbh.de/svn/hxtools/trunk/kbd/ahnv.fnt
[2] http://lkml.org/lkml/2007/6/1/339

    (I also have it for 2.6.18 and 2.6.20 if you need)



	Jan
author
#14 lore raw

Hi Jan :)

* Jan Engelhardt <[email protected]> dixit:

Quoted 15 lines · DervishD
Quoted 10 lines

* H. Peter Anvin <[email protected]> dixit:

Quoted 8 lines
Quoted 3 lines · Jan Engelhardt

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

Both of those are in the 0-255 range, though. I thought the issue was with characters 256+, like ??. At least on my FC6 system that doesn't work with gpm.

I've got a cp437 "DOS" font loaded[1], and giving a glance at /usr/share/kbd/unimaps/cp437.uni tells me right away:

Quoted 6 lines
	0x01 U+263A

So I create U+263A in joe (ESC ' x 2 6 3 a), it displays fine (the smiley character), and I can copy/paste it using GPM without problems. Though, I have that patch[2] in my kernel. Can you check if it works if you apply it?

I cannot reboot my machine right now, and I don't have GPM
installed, but if you give me a few days I think I'll be able to test. I have to upgrade the kernel too (I'm using 2.6.19.5), so make it a week O:)) I'll test ASAP.

And thanks for the patch :))

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
author
#15 lore raw
In reply to Jan Engelhardt

Hi Jan :)

* Jan Engelhardt <[email protected]> dixit:

Quoted 11 lines · DervishD
Quoted 9 lines

This said, I know that the console will give me no problems
regarding character representantion (heck, I'm pretty sure that I will be able to use even the same font I'm using right now in the console if I get the proper unicode map), but probably will give me problems when *entering* characters. I've read that the kernel handles accented chars, and things like 'ñ' (ntilde) because it assumes that any composed character (composed using dead keys, for example) is in the latin1 range. While this is not a perfect behaviour, it will work for me.

Quoted 3 lines

(1) I can do <Compose><~><n> just fine on vt
(2) I can do <ö> just fine on vt too
(3) And copy+paste them both using GPM too, again w/o probs

    Cool! :)))
Quoted 1 line

so not sure where your problem is.

Right now my only problem was lack of reliable information, but that
has been fully solved by all people who answered, so my next step is to upgrade kbd and start making tests :))

Quoted 2 lines

I do however have a patch that you could try should a problem arise. I should repost and ask around again and beat until it's in :)

Any patch regarding a better support of utf-8 is very welcome, and I
can help you beat anybody if that will make the patch in XD

Thanks a lot for the patch (I'll try it if I face problems) and for
your answer.

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!
#16 lore raw

PATCH 1/1 Kernel utf-8 handling

In reply to DervishD

Hello to all,

the following patch still lingers in my tree, can we have it merged?

This patch fixes dead keys and copy/paste of non-ASCII characters in UTF-8 mode on Linux console. See more details about the original patch at: http://chris.heathens.co.nz/linux/utf8.html

Already posted on

	(Oldest) http://lkml.org/lkml/2003/5/31/148
	         http://lkml.org/lkml/2005/12/24/69
	(Recent) http://lkml.org/lkml/2006/8/7/75
Cc Alexander E. Patrakov <[email protected]>
---
 drivers/char/consolemap.c  |   78 +++++++++++++++++++++++++++++++++++++++++----
 drivers/char/keyboard.c    |   26 ++++++++++-----
 drivers/char/selection.c   |   48 +++++++++++++++++++++++----
 include/linux/consolemap.h |    5 ++
 4 files changed, 134 insertions(+), 23 deletions(-)
drivers/char/consolemap.c +72 -6
Index: linux-2.6.22-rc3-git6/drivers/char/consolemap.c
===================================================================
@@ -177,6 +177,7 @@ struct uni_pagedir {
 	unsigned long	refcount;
 	unsigned long	sum;
 	unsigned char	*inverse_translations[4];
+	u16		*inverse_trans_unicode;
 	int		readonly;
 };
 
@@ -207,6 +208,41 @@ static void set_inverse_transl(struct vc
 	}
 }
 
+static void set_inverse_trans_unicode(struct vc_data *conp,
+				      struct uni_pagedir *p)
+{
+	int i, j, k, glyph;
+	u16 **p1, *p2;
+	u16 *q;
+
+	if (!p) return;
+	q = p->inverse_trans_unicode;
+	if (!q) {
+		q = p->inverse_trans_unicode =
+			kmalloc(MAX_GLYPH * sizeof(u16), GFP_KERNEL);
+		if (!q)
+			return;
+	}
+	memset(q, 0, MAX_GLYPH * sizeof(u16));
+
+	for (i = 0; i < 32; i++) {
+		p1 = p->uni_pgdir[i];
+		if (!p1)
+			continue;
+		for (j = 0; j < 32; j++) {
+			p2 = p1[j];
+			if (!p2)
+				continue;
+			for (k = 0; k < 64; k++) {
+				glyph = p2[k];
+				if (glyph >= 0 && glyph < MAX_GLYPH
+					       && q[glyph] < 32)
+		  			q[glyph] = (i << 11) + (j << 6) + k;
+			}
+		}
+	}
+}
+
 unsigned short *set_translate(int m, struct vc_data *vc)
 {
 	inv_translate[vc->vc_num] = m;
@@ -217,19 +253,29 @@ unsigned short *set_translate(int m, str
  * Inverse translation is impossible for several reasons:
  * 1. The font<->character maps are not 1-1.
  * 2. The text may have been written while a different translation map
- *    was active, or using Unicode.
+ *    was active.
  * Still, it is now possible to a certain extent to cut and paste non-ASCII.
  */
-unsigned char inverse_translate(struct vc_data *conp, int glyph)
+u16 inverse_translate(struct vc_data *conp, int glyph, int use_unicode)
 {
 	struct uni_pagedir *p;
+	int m;
 	if (glyph < 0 || glyph >= MAX_GLYPH)
 		return 0;
-	else if (!(p = (struct uni_pagedir *)*conp->vc_uni_pagedir_loc) ||
-		 !p->inverse_translations[inv_translate[conp->vc_num]])
+	else if (!(p = (struct uni_pagedir *)*conp->vc_uni_pagedir_loc))
 		return glyph;
-	else
-		return p->inverse_translations[inv_translate[conp->vc_num]][glyph];
+	else if (use_unicode) {
+		if (!p->inverse_trans_unicode)
+			return glyph;
+		else
+			return p->inverse_trans_unicode[glyph];
+	} else {
+		m = inv_translate[conp->vc_num];
+		if (!p->inverse_translations[m])
+			return glyph;
+		else
+			return p->inverse_translations[m][glyph];
+	}
 }
 
 static void update_user_maps(void)
@@ -243,6 +289,7 @@ static void update_user_maps(void)
 		p = (struct uni_pagedir *)*vc_cons[i].d->vc_uni_pagedir_loc;
 		if (p && p != q) {
 			set_inverse_transl(vc_cons[i].d, p, USER_MAP);
+			set_inverse_trans_unicode(vc_cons[i].d, p);
 			q = p;
 		}
 	}
@@ -353,6 +400,10 @@ static void con_release_unimap(struct un
 		kfree(p->inverse_translations[i]);
 		p->inverse_translations[i] = NULL;
 	}
+	if (p->inverse_trans_unicode) {
+		kfree(p->inverse_trans_unicode);
+		p->inverse_trans_unicode = NULL;
+	}
 }
 
 void con_free_unimap(struct vc_data *vc)
@@ -511,6 +562,7 @@ int con_set_unimap(struct vc_data *vc, u
 
 	for (i = 0; i <= 3; i++)
 		set_inverse_transl(vc, p, i); /* Update all inverse translations */
+	set_inverse_trans_unicode(vc, p);
   
 	return err;
 }
@@ -561,6 +613,7 @@ int con_set_default_unimap(struct vc_dat
 
 	for (i = 0; i <= 3; i++)
 		set_inverse_transl(vc, p, i);	/* Update all inverse translations */
+	set_inverse_trans_unicode(vc, p);
 	dflt = p;
 	return err;
 }
@@ -617,6 +670,19 @@ void con_protect_unimap(struct vc_data *
 		p->readonly = rdonly;
 }
 
+/* may be called during an interrupt */
+u32 conv_8bit_to_uni(unsigned char c)
+{
+	/*
+	 * Always use USER_MAP. This function is used by the keyboard,
+	 * which shouldn't be affected by G0/G1 switching, etc.
+	 * If the user map still contains default values, i.e. the
+	 * direct-to-font mapping, then assume user is using Latin1.
+	 */
+	unsigned short uni = translations[USER_MAP][c];
+	return uni == (0xf000 | c) ? c : uni;
+}
+
 int
 conv_uni_to_pc(struct vc_data *conp, long ucs) 
 {
drivers/char/keyboard.c +18 -8
Index: linux-2.6.22-rc3-git6/drivers/char/keyboard.c
===================================================================
@@ -24,6 +24,7 @@
  * 21-08-02: Converted to input API, major cleanup. (Vojtech Pavlik)
  */
 
+#include <linux/consolemap.h>
 #include <linux/module.h>
 #include <linux/sched.h>
 #include <linux/tty.h>
@@ -308,10 +309,9 @@ static void applkey(struct vc_data *vc, 
  * Many other routines do put_queue, but I think either
  * they produce ASCII, or they produce some user-assigned
  * string, and in both cases we might assume that it is
- * in utf-8 already. UTF-8 is defined for words of up to 31 bits,
- * but we need only 16 bits here
+ * in utf-8 already.
  */
-static void to_utf8(struct vc_data *vc, ushort c)
+static void to_utf8(struct vc_data *vc, uint c)
 {
 	if (c < 0x80)
 		/*  0******* */
@@ -320,11 +320,21 @@ static void to_utf8(struct vc_data *vc, 
 		/* 110***** 10****** */
 		put_queue(vc, 0xc0 | (c >> 6));
 		put_queue(vc, 0x80 | (c & 0x3f));
-	} else {
+    	} else if (c < 0x10000) {
+	       	if (c >= 0xD800 && c < 0xE000)
+			return;
+		if (c == 0xFFFF)
+			return;
 		/* 1110**** 10****** 10****** */
 		put_queue(vc, 0xe0 | (c >> 12));
 		put_queue(vc, 0x80 | ((c >> 6) & 0x3f));
 		put_queue(vc, 0x80 | (c & 0x3f));
+    	} else if (c < 0x110000) {
+		/* 11110*** 10****** 10****** 10****** */
+		put_queue(vc, 0xf0 | (c >> 18));
+		put_queue(vc, 0x80 | ((c >> 12) & 0x3f));
+		put_queue(vc, 0x80 | ((c >> 6) & 0x3f));
+		put_queue(vc, 0x80 | (c & 0x3f));
 	}
 }
 
@@ -393,7 +403,7 @@ static unsigned int handle_diacr(struct 
 		return d;
 
 	if (kbd->kbdmode == VC_UNICODE)
-		to_utf8(vc, d);
+		to_utf8(vc, conv_8bit_to_uni(d));
 	else if (d < 0x100)
 		put_queue(vc, d);
 
@@ -407,7 +417,7 @@ static void fn_enter(struct vc_data *vc)
 {
 	if (diacr) {
 		if (kbd->kbdmode == VC_UNICODE)
-			to_utf8(vc, diacr);
+			to_utf8(vc, conv_8bit_to_uni(diacr));
 		else if (diacr < 0x100)
 			put_queue(vc, diacr);
 		diacr = 0;
@@ -617,7 +627,7 @@ static void k_unicode(struct vc_data *vc
 		return;
 	}
 	if (kbd->kbdmode == VC_UNICODE)
-		to_utf8(vc, value);
+		to_utf8(vc, conv_8bit_to_uni(value));
 	else if (value < 0x100)
 		put_queue(vc, value);
 }
@@ -775,7 +785,7 @@ static void k_shift(struct vc_data *vc, 
 	/* kludge */
 	if (up_flag && shift_state != old_state && npadch != -1) {
 		if (kbd->kbdmode == VC_UNICODE)
-			to_utf8(vc, npadch & 0xffff);
+			to_utf8(vc, npadch);
 		else
 			put_queue(vc, npadch & 0xff);
 		npadch = -1;
drivers/char/selection.c +40 -8

104 lines not on this page · raw message

include/linux/consolemap.h +4 -1
Index: linux-2.6.22-rc3-git6/include/linux/consolemap.h
===================================================================
@@ -8,9 +8,12 @@
 #define IBMPC_MAP 2
 #define USER_MAP 3
 
+#include <linux/types.h>
+
 struct vc_data;
 
-extern unsigned char inverse_translate(struct vc_data *conp, int glyph);
+extern u16 inverse_translate(struct vc_data *conp, int glyph, int use_unicode);
 extern unsigned short *set_translate(int m, struct vc_data *vc);
 extern int conv_uni_to_pc(struct vc_data *conp, long ucs);
+extern u32 conv_8bit_to_uni(unsigned char c);
 void console_map_init(void);
#17 lore raw
In reply to DervishD
Quoted 8 lines · DervishD

Hi all :)

I have a do-it-yourself Linux box, and I'm planning to move to UTF8
(currently I'm using es_ES locale, with latin1 encoding). One of my main concerns (apart from programs with little or no utf8 support, which I will have to suffer) is kernel handling, because I only use the console; I only use X and a terminal emulator if I can't avoid it.

[...]

Quoted 6 lines

Will the console work as it works now if I can live with latin1
accented characters only? Is there any terminal emulator *for the console*, not for X, that handles utf8? Will I be sentenced to X to be able to use my computer with utf8?

Sure, the console will work (don't know about a console terminal
emulator). I'm not very keen on compose keys - I find dead diacriticals (like in X) are usually easier to enter, and I've got all the dead latin1 accents working on my uk keymap. Other diacriticals for normally-latin1 keymaps are a different matter (e.g. caron, ogonek, dot above) - they could be mapped for a specific letter on a specific key (e.g. AltGr z for ż ; z with dot above) but the diacritical modifiers can't be mapped for non latin1, at least in kbd-1.12. You can also alter the keymap to allow you do ISO 14755 input (ctrl+shift+hex_digits) - useful for occasional characters, if they are in your font and you can remember their value.

Ken

Signature · 1 line
das eine Mal als Tragödie, das andere Mal als Farce
author
#18 lore raw

Hi Ken :)

* Ken Moffat <[email protected]> dixit:

Quoted 10 lines · DervishD
Quoted 5 lines

Will the console work as it works now if I can live with latin1
accented characters only? Is there any terminal emulator *for the console*, not for X, that handles utf8? Will I be sentenced to X to be able to use my computer with utf8?

Sure, the console will work (don't know about a console terminal
emulator). I'm not very keen on compose keys - I find dead diacriticals (like in X) are usually easier to enter, and I've got all the dead latin1 accents working on my uk keymap.

I only use compose keys for a bunch of characters like © (copyright
mark) for example. For any other I use dead diacriticals (for example, to write the accented chars for my mother tongue).

Quoted 3 lines

You can also alter the keymap to allow you do ISO 14755 input (ctrl+shift+hex_digits) - useful for occasional characters, if they are in your font and you can remember their value.

I have my keymap modified for that, and I use it as a last resort.
If I find myself using AltGr+hex (that's the key combo I use), I usually put that character in a compose.

Thanks for the info, I'm very happy to hear that it works :)))

Raúl Núñez de Arenas Coronado

Signature · 2 lines
Linux Registered User 88736 | http://www.dervishd.net
It's my PC and I'll cry if I want to... RAmen!