Skip to contentSkip to conversation

Improved console UTF-8 support for the Linux kernel?

DISCUSSION Simos Xenitellis 6 participants 14 messages Dec 11, 2004 – Dec 12, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hi All,
The current UTF-8 keyboard input (for the console) of the Linux kernel does not support "composing" or writing characters with accents. This affects quite a few languages that require accents (French, German, Danish, Swedish?, Greek, cyrillic-based?, others?.).

In general, UTF-8 console support is good to display text in different character sets, enabling to configure a distribution to use UTF-8 locales for both console/Xorg. However, while it was possible to write in German, Spanish, French, etc, now it is not possible anymore.

While looking into the problem, I noticed that there is work to make Linux console handle Unicode better.

Two links are of interest
A. Improved UTF-8 support for the Linux kernel, by Chris Heath
http://chris.heathens.co.nz/linux/utf8.html
B. Notes on the Linux console, by Innocenti Maresin
http://www.comtv.ru/~av95/linux/console/

Discussion on these issues take place at the linux-utf8 mailing list, archived at
http://groups-beta.google.com/group/nlo.lists.linux-utf8

Chris Heath has a set of incremental patches
(http://chris.heathens.co.nz/linux/utf8.html) to enhance Unicode for the console.
I noticed that he contacted this list in May 2003
(http://seclists.org/lists/linux-kernel/2003/May/7956.html) but unfortunatelly the discussion was diverted to coding styles.

Is there an interest for re-submission of mentioned patches for inclusion in the kernel (yeah, provided coding style is "normalised")?

Simos

p.s.
I am not sending this e-mail on behalf of any of the authors, just myself.

#1 lore raw

Hi Simon ;),

Quoted 2 lines

The current UTF-8 keyboard input (for the console) of the Linux kernel does not support "composing" or writing characters with accents.

Yes, i recently find it out when trying to switch all my system to UTF-8. But the patch from Chris you mention below works very well for me (and for anybody that needs to type compose characters for languages based in the latin1 encoding i guess).

Quoted 2 lines

affects quite a few languages that require accents (French, German, Danish, Swedish?, Greek, cyrillic-based?, others?.).

Spanish ;))

Quoted 6 lines

Chris Heath has a set of incremental patches
(http://chris.heathens.co.nz/linux/utf8.html) to enhance Unicode for the console.
I noticed that he contacted this list in May 2003
(http://seclists.org/lists/linux-kernel/2003/May/7956.html) but unfortunatelly the discussion was diverted to coding styles.

Chris told me in the utf-8 mailing list that he doesn't think his patch to make the kernel generate UTF-8 characters in the compose tables will be included in the main kernel. Basically because is not a full solution that cover all the cases... But there is nothing better, so maybe it will be a good idea to include it. Current state is, for 2.6 kernel, text console is broken in UTF-8 mode because it cannot generate UTF-8 composed characters.

Quoted 2 lines

Is there an interest for re-submission of mentioned patches for inclusion in the kernel (yeah, provided coding style is "normalised")?

At least, I am _really_ interested :)

regards,

Signature · 1 line
David Gómez                                      Jabber ID: [email protected]
#2 lore raw
Quoted 2 lines
Quoted 2 lines

The current UTF-8 keyboard input (for the console) of the Linux kernel does not support "composing" or writing characters with accents.

That's weird, because "ö" (LATIN O WITH DIAERESIS) -- which clearly lies outside the 7-bit range, is working on my system without myself poking the kernel. Both hitting the key or using compose mode. This also applies to A-with-DIAERESIS, U-with-DIAERESIS, sharp german S, but does not for anything else, e.g. compose-'-e to generate E with accent aigu.

Quoted 9 lines

Yes, i recently find it out when trying to switch all my system to UTF-8. But the patch from Chris you mention below works very well for me (and for anybody that needs to type compose characters for languages based in the latin1 encoding i guess).

Quoted 2 lines

Is there an interest for re-submission of mentioned patches for inclusion in the kernel (yeah, provided coding style is "normalised")?

At least, I am _really_ interested :)

So am I. I have to use xterm for anything fancy now... (especially for the even-more fancy stuff that begins at three-byte UTF8 sequences, such as Japanese :-)

Jan Engelhardt

Signature · 1 line
ENOSPC
#3 lore raw

Hi Jan ;),

Quoted 6 lines · Jan Engelhardt
Quoted 2 lines
Quoted 2 lines

The current UTF-8 keyboard input (for the console) of the Linux kernel does not support "composing" or writing characters with accents.

That's weird, because "ö" (LATIN O WITH DIAERESIS) -- which clearly lies outside the 7-bit range, is working on my system without myself poking the kernel.

Indeed is weird. Are you sure you keyboard is generating an UTF-8 enconded "ö"? Just check it with echo:

$ echo -n ö | od -t x1

0000000 c3 b6
0000002

I'm using kernel 2.6.9 + Chris patch

Quoted 3 lines

So am I. I have to use xterm for anything fancy now... (especially for the even-more fancy stuff that begins at three-byte UTF8 sequences, such as Japanese :-)

I know :)). By the way, and this is offtopic, have you checked uim? I was testing it the other day with good results, and like it a lot as a japanese (or another script, although i only use this japanese) input method. I've used it with anthy, just have to check it with skk.

regards,

Signature · 1 line
David Gómez                                      Jabber ID: [email protected]
#4 lore raw
Quoted 7 lines

Indeed is weird. Are you sure you keyboard is generating an UTF-8 enconded "ö"? Just check it with echo:

$ echo -n ö | od -t x1

0000000 c3 b6
0000002

Yes it does generate 0xC3B6 (otherwise it would show up as garbage, because it would not be utf8-compliant if it only output 0xF6)

Quoted 1 line

I'm using kernel 2.6.9 + Chris patch

I am using SUSE's KOTD 20041202 (2.6.8 + 2.6.9-rc2)

Quoted 4 lines

I know :)). By the way, and this is offtopic, have you checked uim? I was testing it the other day with good results, and like it a lot as a japanese (or another script, although i only use this japanese) input method. I've used it with anthy, just have to check it with skk.

Have not seen it. What is it? Some sort of xterm?

Jan Engelhardt

Signature · 1 line
ENOSPC
#5 lore raw

Hi Jan ;),

Quoted 6 lines · Jan Engelhardt

Yes it does generate 0xC3B6 (otherwise it would show up as garbage, because it would not be utf8-compliant if it only output 0xF6)

Quoted 1 line

I'm using kernel 2.6.9 + Chris patch

I am using SUSE's KOTD 20041202 (2.6.8 + 2.6.9-rc2)

Maybe the patch or a fix has already been included in rc2/rc3, or in SUSE's version :??

Quoted 3 lines
Quoted 1 line

method. I've used it with anthy, just have to check it with skk.

Have not seen it. What is it? Some sort of xterm?

Just an input system. To be able to write Japanese all over the place ;))

regards,

Signature · 1 line
David Gómez                                      Jabber ID: [email protected]
#6 lore raw
In reply to Jan Engelhardt
Quoted 10 lines · Jan Engelhardt
Quoted 7 lines

Indeed is weird. Are you sure you keyboard is generating an UTF-8 enconded "ö"? Just check it with echo:

$ echo -n ö | od -t x1

0000000 c3 b6
0000002

Yes it does generate 0xC3B6 (otherwise it would show up as garbage, because it would not be utf8-compliant if it only output 0xF6)

Which is exactly (0xF6) what I'm getting. Kernel version 2.6.10-rc2-mm3-V0.7.32-18

As an american, I've often wondered how to go about getting those accented characters out of a std american keyboard. I used to be able to get all those accented and other stuffs out of my amiga's keyboard, stuff like the Beta sign and so on. No can do now, and I miss it.

[...]

Signature · 8 lines
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
99.30% setiathome rank, not too shabby for a WV hillbilly
Yahoo.com attorneys please note, additions to this message
by Gene Heskett are:
Copyright 2004 by Maurice Eugene Heskett, all rights reserved.
#8 lore raw
In reply to a message not in the archive
Quoted 2 lines

I am a bit confused. Could you please comment on the following, as a common test steps?

I do it a bit faster (i.e. without "od"): if after a Compose operation, I see something, it must have been UTF8. If not (like there has been only 8 bits output), the current line screws up a little. <- My test strategy; does not need od to confirm.

Quoted 4 lines

I am not sure how you wrote the above characters. According to UTF-8, characters with codepoints above 0x79 require two bytes so that to be valid. When you compose "ö" (you press something like ";", then "o") in the console?

ö is a "native key" on my keyboard, i.e. i do not need to play with compose to generate ö.

Quoted 1 line

For simplicity, let's assume you do something like

% loadkeys --unicode
compose '/' 'e' to U+00F6 # ('ö')
^D
%

I did that (a shortened form of yours), and saw by dumpkeys that there was now only one compose table entry, so I think loadkeys --unicode overwrites the table? Rightly so.
Still and despite there are now no compose table entries, with the exception of that one, I can still generate ö. <compose><"><o> rightly gives two 7-bit characters (rightly so at this point).

Quoted 1 line

Good. I hope more people raise their hands for this.

...want kanji-on-console, but I guess that will not come true with VGA, which only supports 256 (512) chars. OTOH, [free]BSD's mouse support uses a graphical mouse pointer rather than a "block" one like gpm does, and as I think of it, such a graphical mouse (most Norton apps for DOS also had such) needs some VGA magic or so to set single "pixels"/bits. If single bits within the 8x{16,etc.} char cell can be set, we could have kanji.
Can anyone elaborate on this graphical mouse stuff?

Jan Engelhardt

Signature · 1 line
ENOSPC
#9 lore raw

Hi Jan ;),

Quoted 7 lines
Quoted 4 lines

I am not sure how you wrote the above characters. According to UTF-8, characters with codepoints above 0x79 require two bytes so that to be valid. When you compose "ö" (you press something like ";", then "o") in the console?

ö is a "native key" on my keyboard, i.e. i do not need to play with compose to generate ö.

Aaahh ;), you've should said that before. The whole problem with the kernel is with the compose tables. If you have a native key for "ö" in your keyboard you'll not have problems. I can type for example a 'n with tilde' in my keyboard because is too is a native key, but for accentuated characters, for utf-8 output is neccesary to apply the patch :-/

regards,

Signature · 1 line
David Gómez                                      Jabber ID: [email protected]
#14 lore raw
In reply to Jan Engelhardt
Quoted 1 line · Jan Engelhardt

Can anyone elaborate on this graphical mouse stuff?

What norton does is simply use a few characters that happen to look like a mouse cursor on characters (or norton forces to look, more correctly). You can do that for a single object (like the mouse cursor), and a few more, but of course you can display much less characters that way than with a standard method, as it eats 4 characters/object.

Signature · 7 lines
                The choice of a                              |
      -----==-     _GNU_                                     |
      ----==-- _       generation     Marc Lehmann         +--
      ---==---(_)__  __ ____  __      [email protected]         |e|
      --==---/ / _ \/ // /\ \/ /      http://schmorp.de/   --+
      -=====/_/_//_/\_,_/ /_/\_\      XX11-RIPE            |
                                                           |
#10 lore raw
In reply to David Gómez
Quoted 7 lines · David Gómez

Hi Jan ;),

Quoted 5 lines
Quoted 4 lines

I am not sure how you wrote the above characters. According to UTF-8, characters with codepoints above 0x79 require two bytes so that to be valid. When you compose "ö" (you press something like ";", then "o") in the console?

Quoted 8 lines
Quoted 2 lines

ö is a "native key" on my keyboard, i.e. i do not need to play with compose to generate ö.

Aaahh ;), you've should said that before. The whole problem with the kernel is with the compose tables. If you have a native key for "ö" in your keyboard you'll not have problems. I can type for example a 'n with tilde' in my keyboard because is too is a native key, but for accentuated characters, for utf-8 output is neccesary to apply the patch :-/

And that's the whole issue.

As soon as the kernel is in Unicode mode for the console, currently there is no way to input accented characters through a dead key (composed).
Some years back when 8-bit encodings where used there was no problem, however now all distros are broken with regards to this.

I do not know what is the next step to consider adding the patch. Do we get a kernel maintainer related to console I/O speak up and say "Hmm, I *might* consider a patch, if I see it and people say they are happy"?

simos

#11 lore raw
Quoted 11 lines
Quoted 5 lines

Aaahh ;), you've should said that before. The whole problem with the kernel is with the compose tables. If you have a native key for "ö" in your keyboard you'll not have problems. I can type for example a 'n with tilde' in my keyboard because is too is a native key, but for accentuated characters, for utf-8 output is neccesary to apply the patch :-/

As soon as the kernel is in Unicode mode for the console, currently there is no way to input accented characters through a dead key (composed).
Some years back when 8-bit encodings where used there was no problem, however now all distros are broken with regards to this.

Take it; AFAIK, the DOS box in Windows XP does not support UTF-8 either.

Quoted 3 lines

I do not know what is the next step to consider adding the patch. Do we get a kernel maintainer related to console I/O speak up and say "Hmm, I *might* consider a patch, if I see it and people say they are happy"?

The proposed patch is working and that's ok. I am happy ÷) (first composed smiley hehe <compose><:><-><)> )

Jan Engelhardt

Signature · 1 line
ENOSPC
#12 lore raw
In reply to Simos Xenitellis

Hi Simon ;),

Quoted 11 lines · Simos Xenitellis
Quoted 5 lines

Aaahh ;), you've should said that before. The whole problem with the kernel is with the compose tables. If you have a native key for "ö" in your keyboard you'll not have problems. I can type for example a 'n with tilde' in my keyboard because is too is a native key, but for accentuated characters, for utf-8 output is neccesary to apply the patch :-/

And that's the whole issue.

As soon as the kernel is in Unicode mode for the console, currently there is no way to input accented characters through a dead key (composed).

True.

Quoted 2 lines

Some years back when 8-bit encodings where used there was no problem, however now all distros are broken with regards to this.

I guess that some distros use their own patches, like it seems with SuSE, but it's something that it's broken in the linux console and should be fixed.

Quoted 1 line

I do not know what is the next step to consider adding the patch.

Submitting the patch to lkml to discuss about its possible inclusion would be a good start. I don't know who's the console maintainer, Vojtech Pavlik perhaps?

Regards,

Signature · 1 line
David Gómez                                      Jabber ID: [email protected]
#13 lore raw
In reply to Simos Xenitellis
Quoted 3 lines · Simos Xenitellis

I do not know what is the next step to consider adding the patch. Do we get a kernel maintainer related to console I/O speak up and say "Hmm, I *might* consider a patch, if I see it and people say they are happy"?

You can send me patches if you want.
If I like them I'll submit them.

Very long ago I used to take care of console stuff.

Andries
[email protected]