Improved console UTF-8 support for the Linux kernel?
Subsystems (2)
Here is the patch that makes compose keys work. I consider it a hack... but it's simple and effective enough that it may be worth including. It's a hack because:
* Compose keys are still stored as 8-bit chars internally, but get converted to UTF-8 as they are output. Therefore you are still restricted to a single 8-bit character set.
* Logically, keyboard input and console display are separated in the kernel. When you switch in and out of Unicode mode you have to switch both the keyboard and the display separately. However, this patch intertwines the two because it does its 8-bit to Unicode conversion using a table that is designed for use by the display module. (When the display is in Unicode mode, this conversion table is unused, so why not use it for the keyboard module?)
As you have already figured out, Suse is using this patch in their distribution, so I figure it has had pretty wide testing already.
I have a couple of other patches on my website, which I am happy to submit (or you are welcome to take), but this is the simplest and the most popular.
Chris
include/linux/consolemap.h +0 -0
@@ -13,3 +13,4 @@ extern unsigned char inverse_translate(struct vc_data *conp, int
glyph);
extern unsigned short *set_translate(int m,int currcons); extern int conv_uni_to_pc(struct vc_data *conp, long ucs);
+extern u32 conv_8bit_to_uni(unsigned char c);
drivers/char/consolemap.c +13 -0
@@ -633,6 +633,19 @@if (p) p->readonly = rdonly; } +/* may be called during an interrupt */ +u32 conv_8bit_to_uni (unsigned char c) +{ +/* +* Always use USER_MAP. This function is used by the keyboard, +* which shouldn't be affected by G0/G1 switching, etc. +* If the user map still contains default values, i.e. the +* direct-to-font mapping, then assume user is using Latin1. +*/ +unsigned short uni = translations[USER_MAP ][c]; +return uni == (0xf000 | c) ? c : uni; +} +int conv_uni_to_pc (struct vc_data *conp,long ucs) {
drivers/char/keyboard.c +10 -0
@@ -35,6 +35,7 @@#include <linux/init.h> #include <linux/slab.h> +#include <linux/consolemap.h> #include <linux/kbd_kern.h> #include <linux/kbd_diacr.h> #include <linux/vt_kern.h>
@@ -347,6 +348,15 @@ } } +static void put_8bit (struct vc_data *vc,u8 c) +{ +if (kbd->kbdmode !=VC_UNICODE || c <32 || c ==127 ) +/* Don't translate control chars */ +put_queue (vc, c); +else +to_utf8 (vc,conv_8bit_to_uni (c)); +} +/* * Called after returning from RAW mode or when changing consoles -
recompute
* shift_down[] and shift_state from key_down[] maybe called when
keymap is
@@ -407,7 +417,7 @@
if (ch == ' ' || ch == d)
return d;
- put_queue(vc, d);
+ put_8bit(vc, d);
return ch;
}
@@ -417,7 +427,7 @@
static void fn_enter(struct vc_data *vc, struct pt_regs *regs) {
if (diacr) {
- put_queue(vc, diacr);
+ put_8bit(vc, diacr);
diacr = 0;
}
put_queue(vc, 13);
@@ -626,7 +636,7 @@
diacr = value;
return;
}
- put_queue(vc, value);
+ put_8bit(vc, value);
}
/*
Quoted 2 lines
* Logically, keyboard input and console display are separated in the kernel. When you switch in and out of Unicode mode you have to switch
But they get agglumerated when it comes to ttys.
Quoted 2 lines
As you have already figured out, Suse is using this patch in their distribution, so I figure it has had pretty wide testing already.
They actually don't, as of KOTD 20041202. Patching the tree with the three -CDH1 patches gave no fatal rejects (except a few linenoise), so the patches are not in there ATM.
Quoted 4 lines
I have a couple of other patches on my website, which I am happy to submit (or you are welcome to take), but this is the simplest and the most popular.
Jan Engelhardt
Signature · 1 line
ENOSPC
Quoted 9 lines · Chris Heath
Logically, keyboard input and console display are separated in the kernel. When you switch in and out of Unicode mode you have to switch both the keyboard and the display separately. However, this patch intertwines the two because it does its 8-bit to Unicode conversion using a table that is designed for use by the display module.
I have a couple of other patches on my website, which I am happy to submit (or you are welcome to take), but this is the simplest and the most popular.
Wouldnt mind looking at your other patches.
Will not submit this one - perhaps someone else likes it. I consider the below completely unacceptable.
You cannot use knowledge about the setup of the output side in the keyboard handler. These are independent in principle.
Andries
Quoted 11 lines
drivers/char/consolemap.c +11
-0+u32 conv_8bit_to_uni (unsigned char c) +{ +/* +* Always use USER_MAP. This function is used by the keyboard, +* which shouldn't be affected by G0/G1 switching, etc. +* If the user map still contains default values, i.e. the +* direct-to-font mapping, then assume user is using Latin1. +*/ +unsigned short uni = translations[USER_MAP ][c]; +return uni == (0xf000 | c) ? c : uni; +}
Hi All,
I tried the patch of Chris on Fedora Core 2 and here is my take, from the user's point of view.
The patch works, surprisingly very well.
It can be easily intergrated to the various distributions simply by modifying the /etc/sysconfig/i18n and /etc/sysconfig/keyboard configuration files.
The system scripts essentially call the following two commands (assuming we are already in Unicode mode):
% setfont <font-name> -m <console-screen-map>
% loadkeys <keymap>
For example:
For Spanish:
% setfont latarcyrheb-sun16 -m 8859-1
% loadkeys es
For Finish:
% setfont latarcyrheb-sun16 -m 8859-1
% loadkeys fi
For Greek:
% setfont iso07u-16 -m 8859-7
% loadkeys gr
The character and key maps used are the "old" 8-bit versions. setfonts loads a Unicode map with the "-u" options instead of "-m". Also, the key maps for a few languages have been updates (for example, "gr-utf" for Greek). The new files (very few) cannot be used here. No need to update them ;-).
I tried a few languages and what follows shows characters produced from the console with composing. I used "vim" as my editor.
gr: Greek ά έ ί ό ύ ώ ϊ ϋ ϊ Ά Έ Ί Ή Ύ Ϋ
es: Spanish ñ á é í ý ú ü ï ÿ ä ë
nl: Dutch á é í ó ú ý à è ì ò ù
cz: Czech ä ë ö
us-ascentos: á é í ó ú ý ä ë ï ö ü ÿ
cf: french-canadian à è ì ò ù
fi: Finish ä ë ï ö ü â ê î ô û
fr French â ê î ô û ä ë ï ö ü ÿ
Therefore, from the user's point of view the patch works.
Cheers,
Simos Xenitellis