Support UTF-8 scripts
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Bernd Petrovitsch
Quoted 9 lines · Martin v. Löwis
Quoted 7 lines · Bernd Petrovitsch
On Fri, 2005-09-16 at 22:41 +0200, "Martin v. Löwis" wrote:
[ Language-specific examples ]
And that's the only working way - the programming languages can actually do it because it defines the syntax and semantics of the contents anyways.
Quoted 6 lines
Quoted 2 lines
It works from the programming language point of view, but it is a mess from the text editor point of view.
Most of the text editors have ways to markup the source files. Not even the various editors are able to agreen on one method for all, so why could the (Linux) world agree on one for all text files?
You don't need a marker for all text files, but it's legal to have a marker for utf-8 text files (see the uniocode standard 4.0.0 section 15.9), and it's handy to use it until you made everybody in the world convert everything to utf-8 (but not utf-{16,32}{le,be}).
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
With this marker you are interferign with (at least) *all* text files.
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
If utf-8 encoded text files are text files, and text files are scripts, and all of them shall have the same features, utf-8 encoded text files with BOM MUST be recognized as legal scripts, too. Therefore this patch fixes a kernel bug.
BTW: Implementing the other utf signatures from Table 15.3 is left to the reader as an exercise.-)
Quoted 2 lines
If you think "script" and "text file" are different, define both of them, please, otherwise a discussion is pointless.
If all text files are script files, execute this mail.
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
And there are always tools out there which simply do not understand the generic marker and can not ignore it since these bytes are part of the file.
This conclusion is false. Many tools that don't understand the file structure still can do their job on the files. So the fact that a tool does not understand the structure does not necessarily imply that the tool breaks when the structure changes.
It *may* break just because of some to-be-ignored inline marking due to some questionable feature.
How exactly does it break, and what is it? And why must *it* be prevented from breaking by ignoring script signatures in valid text files?
Quoted 2 lines
And *when* (not if) it breaks, it is probably cumbersome to find since you have pretty unprintable characters.
If your tools can't print utf-8 encoded characters, they are broken for ISO-8859-*, too. Besides that, it's not a kernel problem.
Quoted 3 lines
Let alone the confusion why the size of a file with
ls -lis different from the size in the editor or a marker-awarewc -c.
So IMHO either you have a clear and visible marker or you none at all.
Like e.g. the "From "-line starting each message in a mbox file? Virtually no email client will display it. The size of email messages does differ from it's unencoded content size, too! Off cause nobody can handle this, and all users contantly try to kill themselfes because of that - NOT.
Signature · 2 lines
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF verbreiteten Lügen zu sabotieren.
Quoted 1 line · Bodo Eggert
Bernd Petrovitsch <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 3 lines
Most of the text editors have ways to markup the source files. Not even the various editors are able to agreen on one method for all, so why could the (Linux) world agree on one for all text files?
You don't need a marker for all text files, but it's legal to have a marker for utf-8 text files (see the uniocode standard 4.0.0 section 15.9), and it's handy to use it until you made everybody in the world convert everything to utf-8 (but not utf-{16,32}{le,be}).
Have fun patching almost every text processing tool and concept out there.
Apart from that the way of that marker is wrong it seems to me that the UTF-8 body has no other choice than such a insane "rule" or recommendation).
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
With this marker you are interferign with (at least) *all* text files.
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
If utf-8 encoded text files are text files, and text files are scripts,
No one said all text files are scripts, instead it is the other way 'round.
[ snipped because of ex falso quod libet ]
Quoted 4 lines
Quoted 2 lines
If you think "script" and "text file" are different, define both of them, please, otherwise a discussion is pointless.
If all text files are script files, execute this mail.
See above. Obviously you misunderstand some thing.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
And there are always tools out there which simply do not understand the generic marker and can not ignore it since these bytes are part of the file.
This conclusion is false. Many tools that don't understand the file structure still can do their job on the files. So the fact that a tool does not understand the structure does not necessarily imply that the tool breaks when the structure changes.
Quoted 5 lines
Quoted 2 lines
It *may* break just because of some to-be-ignored inline marking due to some questionable feature.
How exactly does it break, and what is it? And why must *it* be prevented from breaking by ignoring script signatures in valid text files?
The question was: What is if this marker in encountered within a file? To be ignored (by UTF-8 aware tools)? Some other interpretation? Illegal/Forbidden?
Quoted 5 lines
Quoted 2 lines
And *when* (not if) it breaks, it is probably cumbersome to find since you have pretty unprintable characters.
If your tools can't print utf-8 encoded characters, they are broken for ISO-8859-*, too. Besides that, it's not a kernel problem.
Which is again not true since lots of tools out there printed ISO-8859-* correctly before UTF-8 was deployed.
[...]
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesA message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Bernd Petrovitsch
On Sun, 2005-09-18 at 09:23 +0200, "Martin v. Löwis" wrote:
[...]Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
Quoted 7 lines
Quoted 4 lines
The specific feature I get is that when I pass a file starting with <utf8sig>#! to execve, Linux will execute the file following the #!. In what way do I get this feature for text in general? And if I do, why is that a problem?
After applying this patch it seems that "Linux" is supporting this marker officially in general - especially if the kernel supports it.
It will be the first POSIX kernel to correctly support utf-8 scripts. It's 2005, and according to other(?) posters, this should be standard.
Quoted 3 lines
I
suppose the next kernel patch is to support Win-like CR-LF sequences (which is not the case AFAIK).
Maybe it should, maybe it shouldn't. If I used MAC or DOS, I'd be sure it should.-)
Quoted 1 line
BTW even some standards body thinks that this is the way to go,
Not surprisingly the Unicode Consortium is one of them.
Quoted 2 lines
it
raises more problems and questions than resolves anything.
The problem of ow to handle BOM is solved by reading the standard.
Quoted 6 lines
And though scripts are usually edited/changed/"parsed"/... with an text editor, it is not always the case. Therefore the automatic extension to *all text files* (especially as the marker basically applies to all text files, not only scripts).
You want to focus just on your patch and ignore the directly implied potential problems arising ...
There is no problem arising from the patch, it solves one. To solve the rest, use recode.
[...]
Quoted 4 lines
Apparently I have to repeat: If you do
cat a.txt b.txt >c.txtwhere a.txt and b.txt have this marker, then c.txt have the marker of b.txt somewhere in the middle. Does this make sense in anyway? How do I get rid of the marker in the middle transparently?
The unicode standard defines how to handle them.
Quoted 9 lines
Quoted 7 lines
Quoted 2 lines
Let alone the confusion why the size of a file with
ls -lis different from the size in the editor or a marker-awarewc -c.This is true for any UTF-8 file, or any multibyte encoding. For any multibyte encoding, the number of bytes in the file is different from the number of characters. That doesn't (and shouldn't) stop people from using multi-byte encodings.
It is different even if a pure ASCII file is marked as UTF-8.
No pure ASCII file will be marked, since a marked file will be no ASCII file.
Quoted 1 line
And sure, the problem exists in general with multi-byte encodings.
ACK, but that's not a kernel problem nor a specific unicode problem. Fix it by making China, Greece an Japan convert to ASCII and by making all mathematicans stop using strange characters. All other users will follow.
Quoted 11 lines
Quoted 8 lines
What the editor displays as the number of "things" is up to its own. The output of wc -c will always be the same as the one of ls -l, as wc -c does *not* give you characters:
-c, --bytes
print the byte countsYou might have been thinking of 'wc -m'.
It depends on the definition of "character". There are other standards which define "character" as "byte".
There are architectures defining a byte to be 32 bit. They are irrelevant, too.
[...]
Quoted 5 lines
Quoted 2 lines
Not sure what this has
to do with the specific patch, though.It is not supported by the kernel. So either you remove it or you make some compatibility hack (like an appropriate sym-link
-EDOESNOTWORK
#!/usr/bin/perl -T -s -w
Quoted 3 lines
, etc.). Since the
kernel can start java classes directly, you can probably make a similar thing for the UTF-8 stuff.
If MSDOS text files are text files are legal scripts, the kernel should recognize [\x0D\x0A] as valid line breaks.
(The real reason would be unicode allowing NEL to be encoded as 0x0D
or 0x0A.)
This compile-tested patch adds 32 bytes to binfmt_script:
fs/binfmt_script.c 2005-09-18 20:29:44.000000000 +0200 renamed from fs/binfmt_script.c.old 2005-09-18 20:28:32.000000000 +0200 +4 -1
@@ -18,7 +18,7 @@ static int load_script(struct linux_binprm *bprm,struct pt_regs *regs) { - char *cp, *i_name, *i_arg; + char *cp, *cp2, *i_name, *i_arg; struct file *file; char interp[BINPRM_BUF_SIZE]; int retval;
@@ -47,6 +47,9 @@ static int load_script(struct linux_binp bprm->buf[BINPRM_BUF_SIZE - 1] = '\0'; if ((cp = strchr(bprm->buf, '\n')) == NULL) cp = bprm->buf+BINPRM_BUF_SIZE-1; + if ((cp2 = strchr(bprm->buf, '\x0D')) != NULL + && cp2 < cp) + cp = cp2; *cp = '\0'; while (cp > bprm->buf) { cp--;
Signature · 2 lines
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF verbreiteten Lügen zu sabotieren.
On Sun, 2005-09-18 at 21:23 +0200, Bodo Eggert wrote:
[...]
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
Not sure what this has
to do with the specific patch, though.It is not supported by the kernel. So either you remove it or you make some compatibility hack (like an appropriate sym-link
-EDOESNOTWORK#!/usr/bin/perl -T -s -w
It depends on /usr/bin/perl how it handles a white-space character directly after "-w".
Quoted 6 lines
Quoted 3 lines
, etc.). Since the
kernel can start java classes directly, you can probably make a similar thing for the UTF-8 stuff.If MSDOS text files are text files are legal scripts, the kernel should recognize [\x0D\x0A] as valid line breaks.
The Unix worls does recognize the line breaks. It's up to the tool how to handle the white-space character before it. Especially for C and similar languages with continuation lines this leads to interesting (or now more boring) problems.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 10 lines · Bernd Petrovitsch
Quoted 6 lines · Bodo Eggert
Quoted 3 lines
, etc.). Since the
kernel can start java classes directly, you can probably make a similar thing for the UTF-8 stuff.If MSDOS text files are text files are legal scripts, the kernel should recognize [\x0D\x0A] as valid line breaks.
The Unix worls does recognize the line breaks.
Create a valid text file with macintosh line breaks (as allowed in unicode files) and try it.
Signature · 1 line
If enough data is collected, a board of inquiry can prove ANYTHING.
Quoted 7 lines · Bodo Eggert
Quoted 4 lines · Bernd Petrovitsch
Apparently I have to repeat: If you do
cat a.txt b.txt >c.txtwhere a.txt and b.txt have this marker, then c.txt have the marker of b.txt somewhere in the middle. Does this make sense in anyway? How do I get rid of the marker in the middle transparently?The unicode standard defines how to handle them.
For the benefit of those of us who are interested in the problem, but aren't in the mood to wade through a long standard looking for the answer to a specific question, can you elaborate?
It isn't as obvious as all that, because of all the nasty corner cases...
Quoted 4 lines
Quoted 1 line
It is different even if a pure ASCII file is marked as UTF-8.
No pure ASCII file will be marked, since a marked file will be no ASCII file.
Given a file "a.txt" that's pure ASCII, and a file "b.txt" that has the BOM marker on it, what happens when you do "cat a.txt b.txt > c.txt"?
'cat' doesn't know, and has no way of knowing, that c.txt needs a BOM at the *front* of the file until it's already written past the point in c.txt where the BOM has to go.
What does the Unicode standard say to do in this case?
Quoted 4 lines · Bodo Eggert
It will be the first POSIX kernel to correctly support utf-8 scripts. It's 2005, and according to other(?) posters, this should be standard.
UTF-8, yes. BOM bullshit, no.
-hpa
Quoted 8 lines · Bernd Petrovitsch
Quoted 4 lines
The specific feature I get is that when I pass a file starting with <utf8sig>#! to execve, Linux will execute the file following the #!. In what way do I get this feature for text in general? And if I do, why is that a problem?
After applying this patch it seems that "Linux" is supporting this marker officially in general - especially if the kernel supports it.
What makes it seem so? That binfmt_script supports a certain convention doesn't mean that all other programs also somehow need to support that convention - and certainly not in the same way.
Quoted 2 lines
I suppose the next kernel patch is to support Win-like CR-LF sequences (which is not the case AFAIK).
What makes you suppose that? I have no plans to submit such a patch.
Quoted 6 lines
And though scripts are usually edited/changed/"parsed"/... with an text editor, it is not always the case. Therefore the automatic extension to *all text files* (especially as the marker basically applies to all text files, not only scripts).
You want to focus just on your patch and ignore the directly implied potential problems arising ...
Because there are no problems arising. The next time somebody submits a patch to cat(1) to strip off UTF-8 signatures, you *then* complain that this is the wrong thing to do, because it violates the specification of cat.
This reasoning is just flawed: it is like saying to a web browser
developer: "don't _support_ XHTML, because there are so many tools which use HTML 4".
Quoted 3 lines
Apparently I have to repeat: If you do
cat a.txt b.txt >c.txtwhere a.txt and b.txt have this marker, then c.txt have the marker of b.txt somewhere in the middle. Does this make sense in anyway?
Indeed, it does. There is nothing inherently wrong with having the marker in the middle.
Quoted 1 line
How do I get rid of the marker in the middle transparently?
http://www.unicode.org/faq/utf_bom.html#38
Quoted 4 lines
Quoted 4 lines
What the editor displays as the number of "things" is up to its own. The output of wc -c will always be the same as the one of ls -l, as wc -c does *not* give you characters:
Quoted 8 lines
Quoted 4 lines
-c, --bytes
print the byte countsYou might have been thinking of 'wc -m'.
It depends on the definition of "character". There are other standards which define "character" as "byte".
Certainly. However, you specifically talked about 'wc -c', and, in wc(1), atleast in the implementation commonly used on Linux, characters and bytes are not the same.
Quoted 4 lines
Quoted 1 line
It depends on the editor I use, of course
No, more on the OS the editor runs on.
You talked about Windows specifically. On Windows, most editors give you the choice of chosing the line ending, and will preserve whatever line ending they find when adding new lines to a file.
Regards,
Martin
Quoted 6 lines · Martin v. Löwis
Quoted 5 lines · Bernd Petrovitsch
Quoted 4 lines
The specific feature I get is that when I pass a file starting with <utf8sig>#! to execve, Linux will execute the file following the #!. In what way do I get this feature for text in general? And if I do, why is that a problem?
Quoted 6 lines
Quoted 2 lines
After applying this patch it seems that "Linux" is supporting this marker officially in general - especially if the kernel supports it.
What makes it seem so? That
binfmt_scriptsupports a certain convention doesn't mean that all other programs also somehow need to support that convention - and certainly not in the same way.
We will see how it develops. Actually the marker could be used to detect endianness of the file if I read below URL correctly ....
Quoted 4 lines
Quoted 2 lines
I suppose the next kernel patch is to support Win-like CR-LF sequences (which is not the case AFAIK).
What makes you suppose that? I have no plans to submit such a patch.
No need to. Other people tried already.
Quoted 3 lines
This reasoning is just flawed: it is like saying to a web browser
developer: "don't _support_ XHTML, because there are so many tools which use HTML 4".
No, the saying was more: "don't support XHTML since it may break HTML compliant browsers."
For XHTML/HTML we all know that this is not the case, so the comparison is flawed.
Quoted 10 lines
Quoted 3 lines
Apparently I have to repeat: If you do
cat a.txt b.txt >c.txtwhere a.txt and b.txt have this marker, then c.txt have the marker of b.txt somewhere in the middle. Does this make sense in anyway?Indeed, it does. There is nothing inherently wrong with having the marker in the middle.
Quoted 1 line
How do I get rid of the marker in the middle transparently?
Thanks.
---- snip ----
In that case, any U+FEFF occurring in the middle of the file can be ignored, or treated as an error.
---- snip ----
Well, this doesn't sound like an clear rule stating that it *must* be ignored.
BTW:
---- snip ----
Q: How I should deal with BOMs?
[...]
3. Some byte oriented protocols expect ASCII characters at the beginning of a file. If UTF-8 is used with these protocols, use of the BOM as encoding form signature should be avoided.
---- snip ----
Voila. Avoid the BOM in your scripts and be done.
Quoted 6 lines
Quoted 2 lines
It depends on the definition of "character". There are other standards which define "character" as "byte".
Certainly. However, you specifically talked about 'wc -c', and, in wc(1), atleast in the implementation commonly used on Linux, characters and bytes are not the same.
Yes, now since multi-byte character sets gets more commonly used. However, I don't think you get this into the C standard. But we are now far off the discussion ....
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
It depends on the editor I use, of course
No, more on the OS the editor runs on.
You talked about Windows specifically. On Windows, most editors give you the choice of chosing the line ending, and will preserve whatever line ending they find when adding new lines to a file.
I belive this vor vim, emacs, etc. but I don't believe ir for the native ones ...
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 2 lines · Bernd Petrovitsch
We will see how it develops. Actually the marker could be used to detect endianness of the file if I read below URL correctly ....
Text files have endianness????
Quoted 8 lines
---- snip ----
Q: How I should deal with BOMs?
[...]
3. Some byte oriented protocols expect ASCII characters at the beginning of a file. If UTF-8 is used with these protocols, use of the BOM as encoding form signature should be avoided.
---- snip ----
Voila. Avoid the BOM in your scripts and be done.
At which point the proposed kernel patch becomes pointless.. ;)
Quoted 6 lines · On Mon, 2005-09-19 at 05:00 -0400, [email protected] wrote:
Quoted 2 lines · Bernd Petrovitsch
We will see how it develops. Actually the marker could be used to detect endianness of the file if I read below URL correctly ....
Text files have endianness????
Unicode-16 ones with 16 bit per character (as in Win NT), yes. UTF-8 ones not AFAIK.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 11 lines · Bernd Petrovitsch
Quoted 6 lines
Quoted 2 lines
It depends on the definition of "character". There are other standards which define "character" as "byte".
Certainly. However, you specifically talked about 'wc -c', and, in wc(1), atleast in the implementation commonly used on Linux, characters and bytes are not the same.
Yes, now since multi-byte character sets gets more commonly used. However, I don't think you get this into the C standard. But we are now far off the discussion ....
It does indeed, so just one final clarification. wc(1) is not part of the C standard - ISO 9899 does not talk about command line utilities at all. The relevant standard is POSIX; IEEE Std 1003.1, 2004 Edition says, in
http://www.opengroup.org/onlinepubs/009695399/utilities/wc.html
-c
Write to the standard output the number of bytes in each input file.
[...]
-m
Write to the standard output the number of characters in each input
file.
[...]
RATIONALE
[...]
The -c option stands for "character" count, even though it counts bytes. This stems from the sometimes erroneous historical view that bytes and characters are the same size. Due to international requirements, the -m option (reminiscent of "multi-byte") was added to obtain actual character counts.
Regards,
Martin