Alphabet of kernel source
Guys,
I have a silly question, for which I am unable to google out the answer so far. Do we have a Linus' decree on the charset and encoding of the kernel source?
I had a funny situation recently... I prefer non-MIME attachements for two reasons: a) I grab parts of the header and fold them into patch and b) it is easier to quote fragments of the patch with clients I tried (mutt and sylpheed). Admittendly, a different MUA software may change these habits, but please bear with me here. So, someone sent me a patch which included a context line with MODULE_AUTHOR() with an accented name, which the author entered in ISO-8859-1 (he was German). I replied, but my mail agent recoded the reply as UTF-8. The author agreed to my patch, and copied my reply, sent to me. Everything was perfectly readable at this point, but the patch rejected. Because I use Russian and Japanese simultaneously, all utilities run with UTF-8 my boxes, so it took me a moment to do "LANG=C vi" and find the problem.
Anyhow, long story short, this got me thinking... What is the charset and the encoding of the actual source? I saw quite a discussion about the filenames, but this is different. I am sorry if this was discussed previously.
-- Pete
I started a thread a while ago (2.6.3/2.6.4) where I submitted some patches to UTF-8ifying the kernel sources. Basically, most of the kernel is ASCII (98.4% of the files). The rest are mostly ISO-Latin-1, with the rare bit of Japanese (in a couple of charsets) and some just random bytes in some of the Documentation/...
http://www.yak.net/random/linux-2.6.4-utf8-cleanup-auto.diff
http://www.yak.net/random/linux-2.6.4-utf8-cleanup-cstrings2utf8.diff
http://www.yak.net/random/linux-2.6.4-utf8-cleanup-jp.diff
http://www.yak.net/random/linux-2.6.4-utf8-cleanup-wrong.diff
It's sorta difficult to do non-ASCII patches over email because the kernel developers like reading their mail in mutt, and don't like attachments (the only sane ways to send non 7-bit clean data:
8-bit MIME: tagged and bagged or uuencoded)
Further, you confuse the hell out of vi if you have any trash (8bit data in another charset) in a file that's supposed to be UTF-8. i.e. don't think you're going to be able to look at a charset changing patch in anything.
-dte
Quoted 5 lines · David Eger
I started a thread a while ago (2.6.3/2.6.4) where I submitted some patches to UTF-8ifying the kernel sources. Basically, most of the kernel is ASCII (98.4% of the files). The rest are mostly ISO-Latin-1, with the rare bit of Japanese (in a couple of charsets) and some just random bytes in some of the
Documentation/...
The "problem" is contributor names, although having everything in plain ASCII is resonable, I guess.
A lot of names and some art supposed to be ASCII.
Some degree symbols and microseconds... and names.
I remember having problems with lm-sensors trying to print degrees, how did they fight the problem?
Ok, this Japanese is only in the comments.
I can translate that in no time and fix this diff.
WTF is arch/v850/ ?
I guess you had some kind of script, can you try it on vanilla 2.6.7, plesae, and post results.
There are a few microseconds written properly, but may commonly by typed as us, or just don't use abbr.
Quoted 8 lines
It's sorta difficult to do non-ASCII patches over email because
the kernel developers like reading their mail in mutt, and don't like attachments (the only sane ways to send non 7-bit clean data:
8-bit MIME: tagged and bagged or uuencoded)Further, you confuse the hell out of vi if you have any trash (8bit data in another charset) in a file that's supposed to be UTF-8. i.e. don't think you're going to be able to look at a charset changing patch in anything.
Totally agree, although I use Mozilla Mail (and sometimes mutt).
Kalin.
Signature · 3 lines
||///_ o ***************************** ||//'_/> WWW: http://ThinRope.net/ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 4 lines · Kalin KOZHUHAROV
Some degree symbols and microseconds... and names.
I remember having problems with lm-sensors trying to print degrees, how did they fight the problem?
I assume the local charset on the machines where they cat /proc/blah are running in ISO Latin 1 ;-)
Quoted 3 lines
Ok, this Japanese is only in the comments. I can translate that in no time and fix this diff.
actually, I'm pretty sure the diff is correct against 2.6.4 - the bytes should all be correct, as I checked it with someone who works with said files...
Quoted 2 lines
I guess you had some kind of script, can you try it on vanilla 2.6.7, plesae, and post results.
I will regenerate the patches if someone in charge (Linus or Andrew) actually wants them.
-dte
PATCH 1/1 Translate Japanese comments in arch/v850 ( was: Alphabet of kernel source)
Quoted 8 lines · David Eger
Quoted 4 lines
Ok, this Japanese is only in the comments. I can translate that in no time and fix this diff.
actually, I'm pretty sure the diff is correct against 2.6.4 - the bytes should all be correct, as I checked it with someone who works with said files...
OK, I had a few idle minutes, so I did patch the Japanese comments in arch/v850.
I am not exactly 100% sure I translated it correctly since I have no idea what exactly was that NEC v850 evaluation board, but should be OK (say 95% sure).
Patches just the comments, so code is untouched.
The other thing is that one of the files was encoded (i.e. readable) in iso-2022-jp, the other in euc-jp... No idea how patch will handle this, I hope it doesn't bother with locale settings, etc.
Attaching as application/octet-stream in a hope for better handling of i18n issues, sorry for the inconvenience.
Here goes the patch:
arch/v850/kernel/as85ep1.c +6 -6
@@ -57,10 +57,10 @@AS85EP1_ASC =0 ;AS85EP1_LBS =0x00A9 ; -AS85EP1_PORT_PMC (6 ) =0xFF ;/* A20-25, A0,A1 $BM-8z(B */ -AS85EP1_PORT_PMC (7 ) =0x0E ;/* CS1,2,3 $BM-8z(B */ -AS85EP1_PORT_PMC (9 ) =0xFF ;/* D16-23 $BM-8z(B */ -AS85EP1_PORT_PMC (10 ) =0xFF ;/* D24-31 $BM-8z(B */ +AS85EP1_PORT_PMC (6 ) =0xFF ;/* valid A0,A1,A20-A25 */ +AS85EP1_PORT_PMC (7 ) =0x0E ;/* valid CS1-CS3 */ +AS85EP1_PORT_PMC (9 ) =0xFF ;/* valid D16-D23 */ +AS85EP1_PORT_PMC (10 ) =0xFF ;/* valid D24-D31 */ AS85EP1_RFS (1 ) =0x800c ;AS85EP1_RFS (3 ) =0x800c ;
@@ -76,7 +76,7 @@write to address (N - 0x10). We avoid this (effectively) by writing in 16-byte chunks backwards from the end. */ -AS85EP1_IRAMM =0x3 ;/* $BFbB"L?Na(BRAM$B$O!V(Bwrite-mode$B!W$K$J$j$^$9(B */ +AS85EP1_IRAMM =0x3 ;/* "write-mode" for the internal instruction memory */ src = (u32 *)(((u32 )&_intv_copy_src_end -1 ) & ~0xF ); dst = (u32 *)&_intv_copy_dst_start
@@ -88,7 +88,7 @@ src -= 4; }while (src > (u32 *)&_intv_copy_src_start); -AS85EP1_IRAMM =0x0 ;/* $BFbB"L?Na(BRAM$B$O!V(Bread-mode$B!W$K$J$j$^$9(B */ +AS85EP1_IRAMM =0x0 ;/* "read-mode" for the internal instruction memory */ #endif /* !CONFIG_ROM_KERNEL */ v850e_intc_disable_irqs ();
arch/v850/kernel/as85ep1.ld +1 -1
@@ -2,7 +2,7 @@ (CONFIG_V850E_AS85EP1). */ MEMORY { - /* 1MB of internal memory (��¢̿��RAM). */ + /* 1MB of internal instruction memory. */ iMEM0 : ORIGIN = 0, LENGTH = 0x00100000 /* 1MB of static RAM. */
Quoted 4 lines · Pete Zaitcev
Anyhow, long story short, this got me thinking... What is the charset and the encoding of the actual source? I saw quite a discussion about the filenames, but this is different. I am sorry if this was discussed previously.
This has come up repeatedly. As far as I recall, Linus has never said anything. The de facto situation can be seen by just inspecting the MAINTAINERS file. Kai Makisara has a diaeresis on the first vowel of his last name. Today (2.6.6) that is still coded in ISO 8859-1.
In old discussions people who disliked 8859-1 expressed strong preference for plain ASCII (possibly with TeX-like escape sequences for non-ASCII). These days it seems that, if anything is changed, the only reasonable action would be to switch to UTF-8.
Andries
Quoted 6 lines · Pete Zaitcev
Guys,
I have a silly question, for which I am unable to google out the answer so far. Do we have a Linus' decree on the charset and encoding of the kernel source?
[SNIPPED...]
Good question! It was supposed to be ASCII which, I guess is UTF-8 or something like that. However, I find that tabs, which were decreed to be at 8-collumn intervals end up being used instead of spaces i.e., one-column, etc. So, if you look at some well-patched source you sometimes see a mess.
The names of contributors often have non-ASCII characters in them. This may not be a problem, but when using pine without the 'latest-and-greatest' version, they sometimes are unreadable.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.