Skip to contentSkip to conversation

UTF-8ifying the kernel source

PATCH In review David Eger 11 participants 21 messages Mar 4, 2004 – Mar 14, 2004

lore
Share & tools
lore

David Eger

author
#0 lore raw

PATCH 1/1 UTF-8ifying the kernel source

http://www.yak.net/random/linux-2.6.3-utf8-cleanup-auto.diff.bz2

Here you find the first of several patches to convert the kernel source from ISO Latin-1 to UTF-8. I'm working on the files that didn't auto-convert easily; comments welcome ;-)

First, some statistics!

In Linux 2.6.3, there are:
15860 clean 7-bit ASCII files
274 text files are not 7-bit clean

38 of these 274 files are not auto-convertible -- either they are not ISO Latin-1 or the high octets appear within the actual code (not comments).

This first patch applies to help files, documentation, and comments which are trivially correct ISO Latin-1 => UTF-8 conversions. The work I have left to do is summarized below.

--dte

Un-needed/wrong non-ASCII characters (these fixes will form patch 2)
====================================================================
drivers/video/amifb.c - +- sign?
Documentation/i2c/i2c-protocol - NBSP, but why?
arch/i386/kernel/cpu/cyrix.c - NBSP, but why?
arch/v850/kernel/as85ep1.ld - WTF? comments in some random charset...
drivers/char/ftape/lowlevel/fdc-isr.c - WTF? shit in the comments
include/asm-m68k/atarihw.h - 0x94 - "cancel character"?
include/asm-m68k/atariints.h - 0x94 - "cancel character"?
include/linux/802_11.h - why the non-standard dash?
scripts/docproc.c - why the bizarre spelling for specific?
fs/ext2/xattr.c - bad ASCII art
fs/ext3/xattr.c - bad ASCII art
fs/afs/vlclient.h - a degrees sign, but why?

Box-drawing ASCII art (these fixes will form patch 3)
=====================================================
Documentation/networking/tms380tr.txt - DOS-style ASCII art
arch/arm/nwfpe/fpopcode.h - line-drawing characters

C strings - (what to do?)
=========================
arch/ppc/platforms/proc_rtas.c - a C string containing "degrees"
arch/ppc64/kernel/rtas-proc.c - a C string containing "degrees"
drivers/macintosh/therm_adt7467.c - degrees, MODULE_PARAM_DESC(),
and a C string
drivers/mtd/chips/cfi_probe.c - C strings
drivers/net/wireless/netwave_cs.c - C strings
drivers/scsi/dc395x.c - C strings

Other - (i'd convert it, but...)
================================
drivers/pci/pci.ids - I don't know what program processes this...
drivers/ieee1394/oui.db - I don't know what program processes this...

Machine / charset specific shite - (does anything need to be done?)
===================================================================
arch/m68k/hp300/hp300map.map - maps to "char"s.. grr
drivers/char/defkeymap.map - a map file... maps to "char"s.. grr
drivers/char/qtronixmap.c_shipped - maps to "char"s.. grr
drivers/char/qtronixmap.map - maps to "char"s.. grr
drivers/tc/lk201-map.c_shipped - maps to "char"s.. grr
drivers/tc/lk201-map.map - maps to "char"s.. grr
drivers/acorn/char/defkeymap-l7200.c - maps to "char"s.. grr
arch/s390/kernel/ebcdic.c - comments on a keymap table
drivers/video/console/font_8x16.c - comments on a keymap table
drivers/video/console/font_8x8.c - comments on a keymap table
drivers/video/console/font_pearl_8x8.c - comments on a keymap table
drivers/s390/ebcdic.c - comments on a keymap table

Noise from userland (this I won't be touching)
==============================================
Documentation/networking/ethertap.txt - random crap cat'd from /dev/tap0
Documentation/s390/Debugging390.txt - weird gdb output

#1 lore raw

DE> Here you find the first of several patches to convert the kernel DE> source from ISO Latin-1 to UTF-8. I'm working on the files that didn't DE> auto-convert easily; comments welcome ;-)

Why? It's just easier to use plain 8-bit text files today (with editors, code tools etc) and accept the limitations of it that to overcome the limitations by forcing people to UTF-8 editors & other tools.

I am not a kernel developer but this seems a bad idea to me.

Signature · 1 line
Meelis Roos
#2 lore raw
Quoted 7 lines · Meelis Roos

DE> Here you find the first of several patches to convert the kernel DE> source from ISO Latin-1 to UTF-8. I'm working on the files that didn't DE> auto-convert easily; comments welcome ;-)

Why? It's just easier to use plain 8-bit text files today (with editors, code tools etc) and accept the limitations of it that to overcome the limitations by forcing people to UTF-8 editors & other tools.

How do you propose that editors should know which encoding a file uses? The trend seems to be moving towards UTF-8 for everything, so the kernel might as well do it too.

Signature · 2 lines
Måns Rullgård
[email protected]
#3 lore raw
In reply to David Eger
Quoted 12 lines · David Eger

http://www.yak.net/random/linux-2.6.3-utf8-cleanup-auto.diff.bz2

Here you find the first of several patches to convert the kernel source from ISO Latin-1 to UTF-8. I'm working on the files that didn't auto-convert easily; comments welcome ;-)

First, some statistics!

In Linux 2.6.3, there are:
15860 clean 7-bit ASCII files
274 text files are not 7-bit clean

Quoted 6 lines

38 of these 274 files are not auto-convertible -- either they are not ISO Latin-1 or the high octets appear within the actual code (not comments).

This first patch applies to help files, documentation, and comments which are trivially correct ISO Latin-1 => UTF-8 conversions. The work I have left to do is summarized below.

That will be of a great help for the future developers that will edit
kernel sources in Microsoft Word.

[a large collection of expletives in multiple languages and charsets is skipped here]

Signature · 1 line
Alex
#4 lore raw
In reply to David Eger
Quoted 1 line · David Eger

arch/v850/kernel/as85ep1.ld - WTF? comments in some random charset...

FWIW, the charset is EUC-JP.

Even other files in that same directory aren't consistent, e.g., as85ep1.c uses ISO-2022-JP.

[My fault, but it never really registered on my important-enough-to fix radar (emacs autodetects them all so I never really noticed the discrepancy).]

-Miles

Signature · 2 lines
We are all lying in the gutter, but some of us are looking at the stars.
-Oscar Wilde
#5 lore raw
Followup to:  <[email protected]>
By author:    Miles Bader <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 4 lines
Quoted 1 line · David Eger

arch/v850/kernel/as85ep1.ld - WTF? comments in some random charset...

Quoted 9 lines

FWIW, the charset is EUC-JP.

Even other files in that same directory aren't consistent, e.g., as85ep1.c uses ISO-2022-JP.

[My fault, but it never really registered on my important-enough-to fix radar (emacs autodetects them all so I never really noticed the discrepancy).]

OK, this is definitely a good reason to go to UTF-8 across the board.

-hpa

#6 lore raw
Quoted 13 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    Miles Bader <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 9 lines
Quoted 1 line · David Eger

arch/v850/kernel/as85ep1.ld - WTF? comments in some random charset...

FWIW, the charset is EUC-JP.

Even other files in that same directory aren't consistent, e.g., as85ep1.c uses ISO-2022-JP.

Quoted 7 lines
Quoted 4 lines

[My fault, but it never really registered on my important-enough-to fix radar (emacs autodetects them all so I never really noticed the discrepancy).]

OK, this is definitely a good reason to go to UTF-8 across the board.

So when is "less" going to support utf8? Right now, it just shows escape codes... :(

#7 lore raw
Quoted 6 lines · Mike Fedyk
Quoted 2 lines

OK, this is definitely a good reason to go to UTF-8 across the board.

So when is "less" going to support utf8? Right now, it just shows escape codes... :(

Why don't you ask the "less" maintainer about that?

Right now, "less" seems to insist on showing ampersands for *any* non-ASCII character for me...

-hpa

#8 lore raw
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines · Mike Fedyk
Quoted 2 lines

OK, this is definitely a good reason to go to UTF-8 across the board.

Quoted 8 lines
Quoted 3 lines

So when is "less" going to support utf8? Right now, it just shows escape codes... :(

Why don't you ask the "less" maintainer about that?

Right now, "less" seems to insist on showing ampersands for *any* non-ASCII character for me...

Less version 381 is working fine here with UTF-8. I have LANG and LC_CTYPE set to en_US.UTF-8.

Signature · 2 lines
Måns Rullgård
[email protected]
#9 lore raw
Quoted 6 lines · Måns Rullgård
Quoted 3 lines

Right now, "less" seems to insist on showing ampersands for *any* non-ASCII character for me...

Less version 381 is working fine here with UTF-8. I have LANG and LC_CTYPE set to en_US.UTF-8.

less 340 works fine here with the same settings.

Charles

Signature · 4 lines
-----------------------------------------------------------------------
Charles Cazabon                            <[email protected]>
GPL'ed software available at:     http://www.qcc.ca/~charlesc/software/
-----------------------------------------------------------------------
author
#10 lore raw
In reply to Mike Fedyk
Quoted 3 lines · Mike Fedyk

So when is "less" going to support utf8? Right now, it just shows escape codes... :(

bash user? try:

$ export LESSCHARSET="utf-8"
$ less myfavoritefile.c

-dte ;-)

author
#11 lore raw
In reply to David Eger

There are now three patches available, and some work left to go.

The first patch hasn't changed, still the trivial ISO Latin-1 => UTF-8.

The second patch takes care of a lot of wrong and/or unneeded non-ASCII.

The third patch concerns 8-bit characters embedded in C strings. These are almost always output to devfs or proc. The characters used are the degrees symbol (for ppc temp. sensors) and mu (for micro-seconds). I do not want to make a value judgement on what the kernel outputs to userspace, so I leave the strings the same. However, C99 makes it implementation defined how the source character set is translated to the character set in the compiled binary... Therefore, I've taken the raw octets and converted them in the source file to octal constants in the strings, just to make sure cc doesn't mangle things if you set your locale differently...

http://www.yak.net/random/linux-2.6.3-utf8-cleanup-auto.diff.bz2
http://www.yak.net/random/linux-2.6.3-utf8-cleanup-wrong.diff
http://www.yak.net/random/linux-2.6.3-utf8-cleanup-cstrings.diff

-dte

Un-needed/wrong non-ASCII characters (patch 2)
==============================================
drivers/video/amifb.c - +- sign (NOTE: X's .ttf files just don't have it)
Documentation/i2c/i2c-protocol - NBSP, but why? (made regular space)
arch/i386/kernel/cpu/cyrix.c - NBSP, but why? (made regular space)
include/linux/802_11.h - why the non-standard dash? (made regular dash)
scripts/docproc.c - why the bizarre spelling for specific? (fixed)
fs/ext2/xattr.c - bad ASCII art (made regular pipe - fixed)
fs/ext3/xattr.c - bad ASCII art (made regular pipe - fixed)
arch/arm/nwfpe/fpopcode.h - line-drawing characters (fixed)
include/asm-m68k/atarihw.h - 0x94? no, it's an ö, for Björn
include/asm-m68k/atariints.h - 0x94? no, it's an ö, for Björn

C strings - (patch 3)
=====================
arch/ppc/platforms/proc_rtas.c - a C string w/"degrees": exports to proc
arch/ppc64/kernel/rtas-proc.c - a C string w/"degrees": exports to proc
drivers/macintosh/therm_adt7467.c - temperature reporting (degrees sign)
- several printk's, output to a devfs interface, MODULE_PARAM_DESC(),
drivers/mtd/chips/cfi_probe.c - time reporting (micro sign)
- printk's in the DEBUG code
drivers/net/wireless/netwave_cs.c - module version string
(author's name - but it doesn't seem to be *used* for anything...)

BELOW HERE not fixed...

(was going to be fixed w/ patch, but, umm, huh?)
==================================================
arch/v850/kernel/as85ep1.ld - according to Miles Bader,

	it's EUC-JP in the comments, and e.g. as85ep1.c uses ISO-2022-JP...

drivers/char/ftape/lowlevel/fdc-isr.c - WTF? shit in the comments
fs/afs/vlclient.h - a degrees sign, but why? (author says he'll get it)
drivers/scsi/dc395x.c - C debug strings... is this chinese traditional?
Documentation/networking/tms380tr.txt - DOS-style ASCII art

Other - (i'd convert it, but...)
================================
drivers/pci/pci.ids - I don't know what program processes this...
drivers/ieee1394/oui.db - I don't know what program processes this...

Machine / charset specific shite - (does anything need to be done?)
===================================================================
arch/m68k/hp300/hp300map.map - maps to "char"s.. grr
drivers/char/defkeymap.map - a map file... maps to "char"s.. grr
drivers/char/qtronixmap.c_shipped - maps to "char"s.. grr
drivers/char/qtronixmap.map - maps to "char"s.. grr
drivers/tc/lk201-map.c_shipped - maps to "char"s.. grr
drivers/tc/lk201-map.map - maps to "char"s.. grr
drivers/acorn/char/defkeymap-l7200.c - maps to "char"s.. grr
arch/s390/kernel/ebcdic.c - comments on a keymap table
drivers/video/console/font_8x16.c - comments on a keymap table
drivers/video/console/font_8x8.c - comments on a keymap table
drivers/video/console/font_pearl_8x8.c - comments on a keymap table
drivers/s390/ebcdic.c - comments on a keymap table

Noise from userland (this I won't be touching)
==============================================
Documentation/networking/ethertap.txt - random crap cat'd from /dev/tap0
Documentation/s390/Debugging390.txt - weird gdb output

#12 lore raw
Followup to:  <[email protected]>
By author:    David Eger <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 11 lines

The third patch concerns 8-bit characters embedded in C strings. These are almost always output to devfs or proc. The characters used are the degrees symbol (for ppc temp. sensors) and mu (for micro-seconds). I do not want to make a value judgement on what the kernel outputs to userspace, so I leave the strings the same. However, C99 makes it implementation defined how the source character set is translated to the character set in the compiled binary... Therefore, I've taken the raw octets and converted them in the source file to octal constants in the strings, just to make sure cc doesn't mangle things if you set your locale differently...

I would highly vote for making those UTF-8 unless it breaks protocol.

Plain ASCII would be better, though.

-hpa

#13 lore raw
Quoted 9 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    David Eger <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 3 lines

The third patch concerns 8-bit characters embedded in C strings. These are almost always output to devfs or proc. The characters used are the degrees symbol (for ppc temp. sensors) and mu (for micro-seconds).

I would highly vote for making those UTF-8 unless it breaks protocol.

ISO-8859-1 characters are mostly the same in UTF-8.

Xav

#14 lore raw
Quoted 5 lines · Xavier Bestel
Quoted 4 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    David Eger <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

The third patch concerns 8-bit characters embedded in C strings. These are almost always output to devfs or proc. The characters used are the degrees symbol (for ppc temp. sensors) and mu (for micro-seconds).

I would highly vote for making those UTF-8 unless it breaks protocol.

ISO-8859-1 characters are mostly the same in UTF-8.

The 7-bit ones are the same. The 8-bit ones are all different.

Signature · 2 lines
Måns Rullgård
[email protected]
#15 lore raw
In reply to Xavier Bestel
Followup to:  <[email protected]>
By author:    Xavier Bestel <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 6 lines · H. Peter Anvin
Quoted 4 lines
Followup to:  <[email protected]>
By author:    David Eger <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The third patch concerns 8-bit characters embedded in C strings. These are almost always output to devfs or proc. The characters used are the degrees symbol (for ppc temp. sensors) and mu (for micro-seconds).

I would highly vote for making those UTF-8 unless it breaks protocol.

ISO-8859-1 characters are mostly the same in UTF-8.

Unicode, yes. UTF-8, no. The ISO-8859-1 character "Å" (0xC5) does, indeed correspond to Unicode character U+00C5, but it's encoded 0xC3 0x85 in UTF-8.

-hpa

#16 lore raw
Quoted 8 lines · H. Peter Anvin
Followup to:  <[email protected]>
By author:    Xavier Bestel <[email protected]>
Quoted 2 lines

ISO-8859-1 characters are mostly the same in UTF-8.

Unicode, yes. UTF-8, no. The ISO-8859-1 character "Å" (0xC5) does, indeed correspond to Unicode character U+00C5, but it's encoded 0xC3 0x85 in UTF-8.

Yeah, that's what I realized, after posting of course. While utf-8ying the sources is certainly a good thing, I have mixed feelings about kernel strings. It will render poorly in some environments.
Maybe the all-ascii route is better for strings ?

Xav

author
#17 lore raw

Other bizarre thing... backspaces?

In reply to David Eger

There are five files with embedded backspace octets in them.... ;-)

fs/hfs/FAQ.txt
fs/hfs/HFS.txt
fs/hfs/INSTALL.txt
Documentation/filesystems/coda.txt
Documentation/uml/UserModeLinux-HOWTO.txt

-dte

#18 lore raw

Re: Other bizarre thing... backspaces?

Quoted 1 line · David Eger

There are five files with embedded backspace octets in them.... ;-)

That's an old way to do underlining and bold face and it seems like at least coda.txt is doing that. If I could choose I'd probably just remove them.

Signature · 2 lines
Måns Rullgård
[email protected]
#19 lore raw

Re: Re: Other bizarre thing... backspaces?

Dear diary, on Sat, Mar 06, 2004 at 03:04:35PM CET, I got a letter, where M?ns Rullg?rd <[email protected]> told me, that...

Quoted 7 lines
Quoted 1 line · David Eger

There are five files with embedded backspace octets in them.... ;-)

That's an old way to do underlining and bold face and it seems like at least coda.txt is doing that. If I could choose I'd probably just remove them.

Well, what's the "new way" for ASCII documents? At least less produces a desired result.

Kind regards,

Signature · 5 lines
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw
#20 lore raw
In reply to David Eger
Quoted 3 lines · David Eger

Un-needed/wrong non-ASCII characters (patch 2)
==============================================
drivers/video/amifb.c - +- sign (NOTE: X's .ttf files just don't have it)

do_blank is either 0 (do nothing), -1 (unblank), or +1 (blank).

You can replace it by `+/-1' if you want.

Quoted 2 lines

include/asm-m68k/atarihw.h - 0x94? no, it's an ö, for Björn
include/asm-m68k/atariints.h - 0x94? no, it's an ö, for Björn

Yep.

Quoted 9 lines

Machine / charset specific shite - (does anything need to be done?)
===================================================================
arch/m68k/hp300/hp300map.map - maps to "char"s.. grr
drivers/char/defkeymap.map - a map file... maps to "char"s.. grr
drivers/char/qtronixmap.c_shipped - maps to "char"s.. grr
drivers/char/qtronixmap.map - maps to "char"s.. grr
drivers/tc/lk201-map.c_shipped - maps to "char"s.. grr
drivers/tc/lk201-map.map - maps to "char"s.. grr
drivers/acorn/char/defkeymap-l7200.c - maps to "char"s.. grr

If you want the keyboard to generate UTF-8, I think you should change these (not sure, please test).

Quoted 3 lines

drivers/video/console/font_8x16.c - comments on a keymap table
drivers/video/console/font_8x8.c - comments on a keymap table
drivers/video/console/font_pearl_8x8.c - comments on a keymap table

These fonts have the box-drawing ASCII art.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds