kconfig: fix multi-byte UTF handling in nconfig
cover kconfig: fix multi-byte UTF handling in nconfig
Hi,
The first patch is trivial.
The second is inspired by a long-standing bugzilla entry:
https://bugzilla.kernel.org/show_bug.cgi?id=43067
The MTD_NAND_CAFE Kconfig symbol (drivers/mtd/nand/Kconfig) has description text which uses a multi-byte UTF-8 character: the 'É' in 'CAFÉ'. This character (and other similar >8bit UTF-8 characters) is not handled correctly by many of the kernel configuration tools (notably 'make nconfig' and 'make xconfig'). nconfig was especially broken, as it would completely drop any menu entry which had non-ASCII characters, as well as ALL subsequent entries in the same window (!!).
The fix for nconfig is to allow linking against the "wide" ncurses library. I did not bother learning QT well enough to fix 'make xconfig'; it still appears broken w.r.t. wide characters, and makes liberal use of the QString::latin1() conversion for potentially non-Latin strings.
Notably, this issue is not very obvious for the common user. For instance, on Ubuntu one might install libncurses5-dev, which is sufficient for getting 'menuconfig' to compile/link/run just fine. It is easy to miss the fact that unicode handling is incorrect, because the behavior is undefined (usually just chunk characters, but nconfig just silently drops data), and nothing informs them that they should have installed libncursesw5-dev instead.
Ideally, we could drop support for linking against legacy ncurses, and instead require ncursesw, but that might be painful to enforce for all users (i.e., nearly everyone who configures kernels). I welcome any thoughts on improving this state for others (like me, for a long time) who don't realize that they should install the ncursesw development package in order to get 21st century support for unicode help text.
Brian
Brian Norris (2):
kconfig: lxdialog: fix spelling
kconfig: nconfig: fix multi-byte UTF handling
scripts/kconfig/Makefile | 3 ++- scripts/kconfig/lxdialog/dialog.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
Signature · 1 line
1.7.9.5
PATCH 1/2 kconfig: lxdialog: fix spelling
--- scripts/kconfig/lxdialog/dialog.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/kconfig/lxdialog/dialog.h +1 -1
@@ -170,7 +170,7 @@ char item_tag(void);/* item list manipulation for lxdialog use */ #define MAXITEMSTR 200struct dialog_item { -char str[MAXITEMSTR ];/* promtp displayed */ +char str[MAXITEMSTR ];/* prompt displayed */ char tag;void *data;/* pointer to menu item - used by menubox+checklist */ int selected;/* Set to 1 by dialog_*() function if selected. */
Signature · 1 line
1.7.9.5
PATCH 2/2 kconfig: nconfig: fix multi-byte UTF handling
In reply to Brian NorrisCurrently, Kconfig descriptions that use multi-byte UTF-8 characters (such as MTD_NAND_CAFE) will have their menu entries dropped from the 'make nconfig' ncurses menu, and all subsequent entries in the same window will be omitted. This seems to be due to the ncurses 'menu' library, which does not traditionally handle UTF-8 >8-bit characters properly.
The ncursesw library ('w' is for "wide") is written to handle these UTF-8 characters, and is practically a drop-in replacement at the source level. Use it by default, if available.
--- scripts/kconfig/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/kconfig/Makefile +2 -1
@@ -220,7 +220,8 @@ HOSTCFLAGS_gconf.o = `pkg-config --cflags gtk+-2.0 gmodule-2.0 libglade-2.0` \ HOSTLOADLIBES_mconf = $(shell $(CONFIG_SHELL) $(check-lxdialog) -ldflags $(HOSTCC)) HOSTLOADLIBES_nconf = $(shell \ - pkg-config --libs menu panel ncurses 2>/dev/null \ + pkg-config --libs menuw panelw ncursesw 2>/dev/null \ + || pkg-config --libs menu panel ncurses 2>/dev/null \ || echo "-lmenu -lpanel -lncurses" ) $(obj)/qconf.o: $(obj)/.tmp_qtcheck
Signature · 1 line
1.7.9.5
Re: [PATCH 2/2] kconfig: nconfig: fix multi-byte UTF handling
Quoted 11 lines · Brian Norris
Currently,
Kconfigdescriptions that use multi-byte UTF-8 characters (such asMTD_NAND_CAFE) will have their menu entries dropped from the 'make nconfig' ncurses menu, and all subsequent entries in the same window will be omitted. This seems to be due to the ncurses 'menu' library, which does not traditionally handle UTF-8 >8-bit characters properly.The ncursesw library ('w' is for "wide") is written to handle these UTF-8 characters, and is practically a drop-in replacement at the source level. Use it by default, if available.
Quoted 5 lines
Signed-off-by Brian Norris <[email protected]>Cc "Yann E. MORIN" <[email protected]>Cc Martin Walch <[email protected]>
Quoted 11 lines · On Wednesday 04 June 2014 00:52:29 Brian Norris wrote:
The second is inspired by a long-standing bugzilla entry:
https://bugzilla.kernel.org/show_bug.cgi?idC067
The
MTD_NAND_CAFEKconfigsymbol (drivers/mtd/nand/Kconfig) has description text which uses a multi-byte UTF-8 character: the 'É' in 'CAFÉ'. This character (and other similar >8bit UTF-8 characters) is not handled correctly by many of the kernel configuration tools (notably 'make nconfig' and 'make xconfig'). nconfig was especially broken, as it would completely drop any menu entry which had non-ASCII characters, as well as ALL subsequent entries in the same window (!!).
Hi,
so far I have not seen any solid hint that the configuration system was designed with support for anything beyond 7 bit ASCII characters in mind. Except some "of course we use UTF-8 for everything in the 21st century" ranting, I have also not seen any commonly accepted decision that it should use any other character set.
Currently there are 14145 symbols in the mainline kernel and I know of only two that do not use exclusively 7 bit ASCII characters. One is MTD_NAND_CAFE which prompts with "NAND support for OLPC CAFÉ chip" and reads in the help text "Use NAND flash attached to the CAFÉ chip designed for the OLPC laptop.", the other one is HID_XINMO, which has a UTF-8 "no brake space" in the help text "[..]Say Y here[..]" (after the Y). I guess the latter one is only accidentally there.
One reason for this is probably that there is currently no reliable UTF-8 support in the configuration system. Of course, this does not answer the question whether Kconfig files should accept UTF-8 characters or not.
IMO such a change (use UTF-8) should be consented by a wide audience, because it affects every user of the configuration system, and in particular every kernel developer.
As I am no expert for character encoding, please correct me if I am wrong with anything of the following.
While I think that using UTF-8 is often a good idea, I also think that it is a bad idea to just hack UTF-8 support into the configuration system without careful consideration and code review: ASCII is a least common denominator that is compatible with most character sets in regular use. Currently it hardly matters what character encoding the terminal uses and what the font supports as long as it is 7 bit ASCII compatible.
As far as I see, deciding for UTF-8 is an "all-in" thing. It is not feasible to then allow anything beside UTF-8. This will force any user to use a terminal and a font that support UTF-8.
For UTF-8 support, the whole code base of the configuration system should be revisited, because as far as I know it currently makes in some places the assumption that the size of one character equals sizeof(char), although most of the time this will not hurt.
Furthermore, consistent UTF-8 support is hard with flex as it does not really support wide characters. Of course you can make flex accept them, but a 16 bit character will be treated as two 8 bit characters. In flex, this is probably not too much of a drawback, but it is ugly.
Assumed that UTF-8 is the preferred character encoding, where should this apply? Only in help texts? Also in comments and in menu prompts? How about expansion variables? Default values? Symbol names? (the latter would force the C preprocessor to use that character set, which will probably not happen)
Anyway, I think it would help to have a clear specification (i.e. a documented decision), no matter if with or without UTF-8.
Regards,
Martin Walch
Quoted 3 lines
Assumed that UTF-8 is the preferred character encoding, where should this apply? Only in help texts? Also in comments and in menu prompts?
UTF-8 should be supported in all display texts.
So help + prompts.
The Kconfig type "string" is not UTF-8 enabled.
Any help in audit Kconfig for this - and testing too - would be appreciated.
Sam
Quoted 8 lines · Brian Norris
Hi,
The first patch is trivial.
The second is inspired by a long-standing bugzilla entry:
Quoted 7 lines
The
MTD_NAND_CAFEKconfigsymbol (drivers/mtd/nand/Kconfig) has description text which uses a multi-byte UTF-8 character: the 'É' in 'CAFÉ'. This character (and other similar >8bit UTF-8 characters) is not handled correctly by many of the kernel configuration tools (notably 'make nconfig' and 'make xconfig'). nconfig was especially broken, as it would completely drop any menu entry which had non-ASCII characters, as well as ALL subsequent entries in the same window (!!).
Thanks for fixing this.
Sam
Hi Yan,
Ping? The patches are simple.
Quoted 48 lines · Brian Norris
Hi,
The first patch is trivial.
The second is inspired by a long-standing bugzilla entry:
https://bugzilla.kernel.org/show_bug.cgi?id=43067
The
MTD_NAND_CAFEKconfigsymbol (drivers/mtd/nand/Kconfig) has description text which uses a multi-byte UTF-8 character: the 'É' in 'CAFÉ'. This character (and other similar >8bit UTF-8 characters) is not handled correctly by many of the kernel configuration tools (notably 'make nconfig' and 'make xconfig'). nconfig was especially broken, as it would completely drop any menu entry which had non-ASCII characters, as well as ALL subsequent entries in the same window (!!).The fix for nconfig is to allow linking against the "wide" ncurses library. I did not bother learning QT well enough to fix 'make xconfig'; it still appears broken w.r.t. wide characters, and makes liberal use of the
QString::latin1()conversion for potentially non-Latin strings.Notably, this issue is not very obvious for the common user. For instance, on Ubuntu one might install libncurses5-dev, which is sufficient for getting 'menuconfig' to compile/link/run just fine. It is easy to miss the fact that unicode handling is incorrect, because the behavior is undefined (usually just chunk characters, but nconfig just silently drops data), and nothing informs them that they should have installed libncursesw5-dev instead.
Ideally, we could drop support for linking against legacy ncurses, and instead require ncursesw, but that might be painful to enforce for all users (i.e., nearly everyone who configures kernels). I welcome any thoughts on improving this state for others (like me, for a long time) who don't realize that they should install the ncursesw development package in order to get 21st century support for unicode help text.
Brian
Brian Norris (2):
kconfig: lxdialog: fix spelling
kconfig: nconfig: fix multi-byte UTF handlingscripts/kconfig/Makefile | 3 ++- scripts/kconfig/lxdialog/dialog.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)--
1.7.9.5
Hi Yan,
Quoted 1 line · Brian Norris
Ping? The patches are simple.
Ping.
Quoted 50 lines
Quoted 48 lines · Brian Norris
Hi,
The first patch is trivial.
The second is inspired by a long-standing bugzilla entry:
https://bugzilla.kernel.org/show_bug.cgi?id=43067
The
MTD_NAND_CAFEKconfigsymbol (drivers/mtd/nand/Kconfig) has description text which uses a multi-byte UTF-8 character: the 'É' in 'CAFÉ'. This character (and other similar >8bit UTF-8 characters) is not handled correctly by many of the kernel configuration tools (notably 'make nconfig' and 'make xconfig'). nconfig was especially broken, as it would completely drop any menu entry which had non-ASCII characters, as well as ALL subsequent entries in the same window (!!).The fix for nconfig is to allow linking against the "wide" ncurses library. I did not bother learning QT well enough to fix 'make xconfig'; it still appears broken w.r.t. wide characters, and makes liberal use of the
QString::latin1()conversion for potentially non-Latin strings.Notably, this issue is not very obvious for the common user. For instance, on Ubuntu one might install libncurses5-dev, which is sufficient for getting 'menuconfig' to compile/link/run just fine. It is easy to miss the fact that unicode handling is incorrect, because the behavior is undefined (usually just chunk characters, but nconfig just silently drops data), and nothing informs them that they should have installed libncursesw5-dev instead.
Ideally, we could drop support for linking against legacy ncurses, and instead require ncursesw, but that might be painful to enforce for all users (i.e., nearly everyone who configures kernels). I welcome any thoughts on improving this state for others (like me, for a long time) who don't realize that they should install the ncursesw development package in order to get 21st century support for unicode help text.
Brian
Brian Norris (2):
kconfig: lxdialog: fix spelling
kconfig: nconfig: fix multi-byte UTF handlingscripts/kconfig/Makefile | 3 ++- scripts/kconfig/lxdialog/dialog.h | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)--
1.7.9.5
Dne 20.8.2014 18:40, Brian Norris napsal(a):
Quoted 6 lines
Hi Yan,
Quoted 1 line · Brian Norris
Ping? The patches are simple.
Ping.
I applied the patches to my kbuild tree.
Michal
Quoted 9 lines · Michal Marek
Dne 20.8.2014 18:40, Brian Norris napsal(a):
Quoted 6 lines
Hi Yan,
Quoted 1 line · Brian Norris
Ping? The patches are simple.
Ping.
I applied the patches to my kbuild tree.
Great, thanks!
Brian