Support UTF-8 scripts
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Martin Mares
I doubt that. For ages people were using several different encodings on a single system (at least here in .cz) without any markers and although there were some rough edges, almost everything worked. Now we do the same with ISO-8859-2 and UTF-8, again with no need for a marker.
This is true for text files, where a human reader can interpret the data correctly even in absence of a declaration. For programming languages, this is typically not the case. Instead, in order to correctly interpret the source code, you need to declare the encoding. For a script, this should be done inside the file itself, as there is no explicit invocation of a compiler or some such where the script encoding could be specified externally.
Regards,
Martin
Hello!
Quoted 4 lines
This is true for text files, where a human reader can interpret the data correctly even in absence of a declaration. For programming languages, this is typically not the case. Instead, in order to correctly interpret the source code, you need to declare the encoding. For a script,
[...]
This makes no sense. For a script, the shell does not care about the encoding at all.
Also, currently, people use zillions of encodings, most of which have no signature, so introducing a signature for UTF-8 does not win anything.
In the future, most people will probably use only UTF-8, so the signature carries no information.
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth Q: Who invented the first airplane that did not fly? A: The Wrong Brothers.
Quoted 2 lines · Martin Mares
This makes no sense. For a script, the shell does not care about the encoding at all.
I'm not (only) talking about /bin/sh. I'm primarily talking about /usr/bin/python, /usr/bin/perl, and /usr/bin/wish. In all these languages, the interpreter *does* care about the encoding.
1. In Python, the syntax
u"some data"
denotes a Unicode literal (stored internally either in UCS-2 or UCS-4); the literals are converted from the source encoding to the internal representation. This requires knowledge of the source encoding.
2. In Tcl, all strings are internally represented in UTF-8, and converted from the source encoding (which currently is inferred from the locale of the process executing the script).
3. In Perl, 'use utf8' declares that the encoding of the script is UTF-8, meaning that non-ASCII can be used in string literals, identifiers, and regular expressions.
Quoted 2 lines
Also, currently, people use zillions of encodings, most of which have no signature, so introducing a signature for UTF-8 does not win anything.
This specific patch does win something: it allows to executed scripts which start with <utf8 signature>#!
This is useful e.g. for Python, which recognizes the UTF-8 signature as declaring the source encoding of the Python module to be UTF-8.
Quoted 2 lines
In the future, most people will probably use only UTF-8, so the signature carries no information.
In the future, the signature *will* carry no information. But the future is, well, in the future.
I just can't understand why (some) people are so opposed to this patch. It is a really trivial, straight-forward change. It introduces no policy, just a feature: you can put the UTF-8 signature in your script file, if you want to (and your scripting language supports it). By no means it forces you to put the UTF-8 signature in your all script files, let alone all your text files.
Regards,
Martin
Hello!
Quoted 3 lines
I'm not (only) talking about
/bin/sh. I'm primarily talking about/usr/bin/python,/usr/bin/perl, and/usr/bin/wish. In all these languages, the interpreter *does* care about the encoding.
Agreed. On the other hand, in all these languages you can pass the encoding as a parameter to the interpreter, cannot you?
Quoted 4 lines
In the future, the signature *will* carry no information. But the future is, well, in the future.
I just can't understand why (some) people are so opposed to this patch.
Occam's razor?
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth "In accord to UNIX philosophy, PERL gives you enough rope to hang yourself." -- Larry Wall
Quoted 2 lines · Martin Mares
Agreed. On the other hand, in all these languages you can pass the encoding as a parameter to the interpreter, cannot you?
Not in general, no. If you have a library of multiple modules, different modules may have different encodings. In particular, if UTF-8 in source code becomes more common (because it is better supported than now), people will start using it for libraries. At the same time, a lot of code is around that still uses other encodings (typically Latin-1). So you may have two encodings in the same program (different modules); that's why you need the encoding declared *in* the file.
Now, there are different ways to do that: you can find language-specific ways (such as 'use utf8;'), and this is what most languages currently do. However, this is a nightmare for editor developers, and a severe inconvenience for script authors - which now have to put the encoding declaration into the files.
With the UTF-8 signature, things become much simpler: editors can automatically detect presence of the signature, and need no language-specific parsing. The language interpreters have a guarantee that the signature is at the beginning of the file, so they don't need to switch encodings in the middle of parsing. Users can configure their editors to always write the signature for certain types of files, and don't need to worry about putting correct encoding declarations into the files.
Quoted 7 lines
Quoted 4 lines
In the future, the signature *will* carry no information. But the future is, well, in the future.
I just can't understand why (some) people are so opposed to this patch.
Occam's razor?
Probably not literally, as we are not searching for an explanation of some phenomenon. You are probably suggesting that people dislike the feature because they see no need for it (as one poster stated it:
I don't use UTF-8, so I don't want that feature).
However, I do believe there is a need for the feature, and that the gains by far outweigh the costs.
Regards,
Martin
Hello!
Quoted 3 lines
With the UTF-8 signature, things become much simpler: editors can automatically detect presence of the signature, and need no language-specific parsing.
I still think that this does solve only a completely insignificant part of the problem. Given the zillion existing encodings, you are able to identify UTF-8, leaving you with zillion-1 other encodings you are unable to deal with.
Quoted 2 lines
Probably not literally, as we are not searching for an explanation of some phenomenon.
ACK, not literally.
Quoted 3 lines
You are probably suggesting that people dislike the feature because they see no need for it (as one poster stated it:
I don't use UTF-8, so I don't want that feature).
I see a need for a feature which would help identify the charset of the script, but the patch in question obviously doesn't offer that -- it solves only a single special case of the problem in a completely non-systematic way. This does not sound right.
Have a nice fortnight
Signature · 3 lines
Martin `MJ' Mares <[email protected]> http://atrey.karlin.mff.cuni.cz/~mj/ Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth "How I need a drink, alcoholic in nature, after the tough chapters involving quantum mechanics!" = \pi
Quoted 3 lines · Martin Mares
I still think that this does solve only a completely insignificant part of the problem. Given the zillion existing encodings, you are able to identify UTF-8, leaving you with zillion-1 other encodings you are unable to deal with.
Correct. This is a special case only. The more general problem is already solved: both Python and Perl support source encodings in the entire zillion encodings. As I explained, this general solution, while being general, is also not very user-friendly.
Now, why does UTF-8 deserve to be a special case? One reason is that it has the potential to replace the entire zillion of encodings over time. However, this can only happen if tool support for this encoding is really good. The patch contributes a (minor) fragment to the support - it is a small patch only.
The other reason is that UTF-8 defines its own encoding declaration, unlike most of the other zillion-1 encodings. So naturally, an implementation that supports UTF-8 in this way cannot extend to other encodings. hpa suggested that ISO-2022 would be a more general mechanism, but pointed out that it hasn't implemented widely in the last 30 years, so it is unlikely that it will get much better support in the next thirty years.
Quoted 4 lines
I see a need for a feature which would help identify the charset of the script, but the patch in question obviously doesn't offer that -- it solves only a single special case of the problem in a completely non-systematic way. This does not sound right.
It's not a complete solution, but it *is* part of a general solution. People have tried in the past to solve the general problem of "identify the encoding of a text file", both in really general ways (iso-2022) and in format-specific ways (perl, python). All these solutions are tedious to use.
There is another general solution: gradually replace the zillion encodings with a single one, namely Unicode (or, specifically, UTF-8). This solution will only work when done gradually. Clearly, this patch doesn't implement this solution entirely, but it contributes to it, by making usage of UTF-8 in script files more simple. Many more changes to other software (i.e. non-kernel changes) will be necessary to implement this solution, as well as (obviously) changes to existing files.
Regards,
Martin
Hi!
Quoted 6 lines
I just can't understand why (some) people are so opposed to this patch. It is a really trivial, straight-forward change. It introduces no policy, just a feature: you can put the UTF-8 signature in your script file, if you want to (and your scripting language supports it). By no means it forces you to put the UTF-8 signature in your all script files, let alone all your text files.
Why is binfmt_misc not enough for you?
Pavel
Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
Quoted 1 line · Pavel Machek
Why is
binfmt_miscnot enough for you?
For two reasons: for one, it has the overhead of yet another exec call. This is different from usages for, say, Java byte code or Python byte code, where the registered interpreter already is the eventual binary which has to be invoked anyway; for a binfmt_misc application, you need an additional wrapper which reinterprets the first line, and then invokes the eventual interpreter.
The other reason is availability: as an author of an UTF-8 script, you would have to communicate to your users that they need the right binfmt_misc wrapper installed (which they may have to build first). While installing additional stuff to run a single program is acceptable for large applications, it is likely not for script files. To make the feature useful in practice, it must be builtin.
Regards,
Martin
Quoted 10 lines · Martin v. Löwis
Quoted 1 line · Pavel Machek
Why is
binfmt_miscnot enough for you?For two reasons: for one, it has the overhead of yet another exec call. This is different from usages for, say, Java byte code or Python byte code, where the registered interpreter already is the eventual binary which has to be invoked anyway; for a
binfmt_miscapplication, you need an additional wrapper which reinterprets the first line, and then invokes the eventual interpreter.
Who cares? exec is fast.
Quoted 7 lines
The other reason is availability: as an author of an UTF-8 script, you would have to communicate to your users that they need the right
binfmt_miscwrapper installed (which they may have to build first). While installing additional stuff to run a single program is acceptable for large applications, it is likely not for script files. To make the feature useful in practice, it must be builtin.
This is distribution problem, not kernel problem. "/bin/ls should be built into kernel, because otherwise you can't call /bin/ls from script" is not an argument.
If UTF-8 compatibility is important, distros will get it right. If it is not, you loose, but at least kernel is not messed up.
Pavel
Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
Quoted 2 lines · Pavel Machek
If UTF-8 compatibility is important, distros will get it right. If it is not, you loose, but at least kernel is not messed up.
The patch doesn't mess up the kernel.
Regards,
Martin
Quoted 5 lines · Martin v. Löwis
Quoted 2 lines · Pavel Machek
If UTF-8 compatibility is important, distros will get it right. If it is not, you loose, but at least kernel is not messed up.
The patch doesn't mess up the kernel.
Every patch does.
Except that yours one does not because it is not going in :-).
Pavel
Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
Quoted 4 lines · Pavel Machek
Quoted 2 lines
which reinterprets the first line, and then invokes the eventual interpreter.
Who cares? exec is fast.
It would be nice if it was but exec + user space overhead of startup is merely "faster than many equivalent systems". It's still slow
Quoted 5 lines · Martin v. Löwis
Quoted 1 line · Pavel Machek
Why is
binfmt_miscnot enough for you?For two reasons: for one, it has the overhead of yet another exec call.
For an interpreted language this is surely irrelevant.
[...]
Quoted 7 lines
The other reason is availability: as an author of an UTF-8 script, you would have to communicate to your users that they need the right
binfmt_miscwrapper installed (which they may have to build first). While installing additional stuff to run a single program is acceptable for large applications, it is likely not for script files. To make the feature useful in practice, it must be builtin.
That is a distribution problem.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · H. Peter Anvin
You don't have markers (although they're defined, see ISO 2022) for your 8-bit encodings, and *THEY'RE THE ONES THAT NEED TO BE DISTINGUISHED.* Flagging UTF-8, especially with the BOM (as opposed to the ISO 2022 signature, <ESC>%G) is pointless in the context, since you still can't distinguish your arbitrary number of legacy encodings.
In programming languages that support the notion of source encodings, you do have markers for 8-bit encodings. For example, in Python, you can specify
# -*- coding: iso-8859-1 -*-
to denote the source encoding. In Perl, you write
use encoding "latin-1";
(with 'use utf8;' being a special-case shortcut).
In Java, you can specify the encoding through the -encoding argument to javac. In gcc, you use -finput-charset (with the special case of -fexec-charset and -fwide-exec-charset potentially being different).
So you *must* use encoding declarations in some languages; the UTF-8 signature is a particularly convenient way of doing so, since it allows for uniformity across languages, with no need for the text editors to parse all the different programming languages.
Regards,
Martin
Quoted 12 lines · Martin v. Löwis
In programming languages that support the notion of source encodings, you do have markers for 8-bit encodings. For example, in Python, you can specify
# -*- coding: iso-8859-1 -*-
to denote the source encoding. In Perl, you write
use encoding "latin-1";
(with 'use utf8;' being a special-case shortcut).
Quoted 8 lines
In Java, you can specify the encoding through the -encoding argument to javac. In gcc, you use -finput-charset (with the special case of -fexec-charset and -fwide-exec-charset potentially being different).
So you *must* use encoding declarations in some languages; the UTF-8 signature is a particularly convenient way of doing so, since it allows for uniformity across languages, with no need for the text editors to parse all the different programming languages.
Did you miss the point? There has been a standard for marking for *30 years*, and virtually NOONE (outside Japan) uses it.
-hpa
Quoted 2 lines · H. Peter Anvin
Did you miss the point? There has been a standard for marking for *30 years*, and virtually NOONE (outside Japan) uses it.
I understood that fact - but I fail to see the point. If you mean to imply "people did not use ISO-2022, therefore, they will never use encoding declarations", I think this implication is false. People do use encoding declarations.
If you mean to imply "people did not use ISO-2022, therefore, they will never use the UTF-8 signature", I think this implications is also false. People do use the UTF-8 signature, even outside Japan. The primary reason is that the UTF-8 signature is much easier to implement than ISO-2022: if you support UTF-8 in your tool (say, a text editor), anyway, adding support for the UTF-8 signature is almost trivial. Therefore, many more editors support the UTF-8 signature today than ever supported ISO-2022.
Regards,
Martin
On Fri, 2005-09-16 at 22:41 +0200, "Martin v. Löwis" wrote:
[ Language-specific examples ]
And that's the only working way - the programming languages can actually do it because it defines the syntax and semantics of the contents anyways.
With this marker you are interferign with (at least) *all* text files. And thus with *all* tools which "handle" those text files.
Quoted 1 line
So you *must* use encoding declarations in some languages; the UTF-8
... if you absolutely want to use Non-ASCII characters in the source code. In most (if not all) of them exist a native gettext() interface ...
Quoted 3 lines
signature is a particularly convenient way of doing so, since it allows for uniformity across languages, with no need for the text editors to parse all the different programming languages.
And there are always tools out there which simply do not understand the generic marker and can not ignore it since these bytes are part of the file. And thus tools (and people) will kill those markers (for whatever reason and if it's simple ignorance) anyway.
Or another example: (Try to) start a perl/shell/... script (without paranmeter on the first line) which was edited on Win* and binary copied to a Unix system. Or at least guess what will happen ....
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 6 lines · Bernd Petrovitsch
On Fri, 2005-09-16 at 22:41 +0200, "Martin v. Löwis" wrote:
[ Language-specific examples ]And that's the only working way - the programming languages can actually do it because it defines the syntax and semantics of the contents anyways.
It works from the programming language point of view, but it is a mess from the text editor point of view.
Even for the programming language, it is a pain to implement: what if you have non-ASCII characters before the pragma that declares the encoding? and so on.
Quoted 1 line
With this marker you are interferign with (at least) *all* text files.
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
Quoted 1 line
And thus with *all* tools which "handle" those text files.
This is simply not true. My patch does not interfere with any such tools. They continue to work just fine.
Quoted 6 lines
Quoted 1 line
So you *must* use encoding declarations in some languages; the UTF-8
... if you absolutely want to use Non-ASCII characters in the source code. In most (if not all) of them exist a native
gettext()interface ...
True. However, this is more tedious to use. Also, it doesn't apply to all cases: e.g. if you have comments, documentation etc. in the source code, gettext is no option.
Likewise, people often want to use non-ASCII in identifiers (e.g. class Lösung); this can also only work if you know what the source encoding is. You may argue that people just shouldn't do that, because it does not work well, but this is not convincing: it doesn't work well because language developers are to lazy to implement it. In fact, some languages (C, C++, Java, C#) do support non-ASCII identifiers (atleast in their specifications); there really isn't a good reason not to support it in scripting languages as well.
Quoted 3 lines
And there are always tools out there which simply do not understand the generic marker and can not ignore it since these bytes are part of the file.
This conclusion is false. Many tools that don't understand the file structure still can do their job on the files. So the fact that a tool does not understand the structure does not necessarily imply that the tool breaks when the structure changes.
Quoted 3 lines
Or another example: (Try to) start a perl/shell/... script (without paranmeter on the first line) which was edited on Win* and binary copied to a Unix system. Or at least guess what will happen ....
For a Python script, I don't need to guess: It will just work.
Regards,
Martin
Quoted 10 lines · Martin v. Löwis
Quoted 6 lines · Bernd Petrovitsch
On Fri, 2005-09-16 at 22:41 +0200, "Martin v. Löwis" wrote:
[ Language-specific examples ]And that's the only working way - the programming languages can actually do it because it defines the syntax and semantics of the contents anyways.
It works from the programming language point of view, but it is a mess from the text editor point of view.
Most of the text editors have ways to markup the source files. Not even the various editors are able to agreen on one method for all, so why could the (Linux) world agree on one for all text files?
Quoted 3 lines
Even for the programming language, it is a pain to implement: what if you have non-ASCII characters before the pragma that declares the encoding? and so on.
That's the problem of the language definers who absolutely want such (IMHO absolutely superflous) features.
Quoted 5 lines
Quoted 1 line
With this marker you are interferign with (at least) *all* text files.
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
If you think "script" and "text file" are different, define both of them, please, otherwise a discussion is pointless.
Quoted 8 lines
Quoted 3 lines
And there are always tools out there which simply do not understand the generic marker and can not ignore it since these bytes are part of the file.
This conclusion is false. Many tools that don't understand the file structure still can do their job on the files. So the fact that a tool does not understand the structure does not necessarily imply that the tool breaks when the structure changes.
It *may* break just because of some to-be-ignored inline marking due to some questionable feature.
And *when* (not if) it breaks, it is probably cumbersome to find since you have pretty unprintable characters.
Let alone the confusion why the size of a file with ls -l is different from the size in the editor or a marker-aware wc -c.
So IMHO either you have a clear and visible marker or you none at all.
Quoted 5 lines
Quoted 3 lines
Or another example: (Try to) start a perl/shell/... script (without paranmeter on the first line) which was edited on Win* and binary copied to a Unix system. Or at least guess what will happen ....
For a Python script, I don't need to guess: It will just work.
Then write a short python script (with a "#!/usr/bin/python" line at the start [without parameters]) natively on a Win*-system, copy it binary over to an arbitrary Linux system and see what's happening.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 3 lines · Bernd Petrovitsch
Most of the text editors have ways to markup the source files. Not even the various editors are able to agreen on one method for all, so why could the (Linux) world agree on one for all text files?
You are ignoring the role of standardization. People invent their own mechanism if a standard is missing (or virtually unimplementable). For declaring encodings, there is no standard (except of iso-2022, which is really hard to implement correctly). Therefore, editor authors create their own standards.
Atleast Python abstained from creating yet another standard, and instead supports both the declarations from Emacs and vim. To some degree, it also supports notepad (namely through the UTF-8 signature).
However, people are much more likely to agree on a technology when it is defined by a recognized standards body. This is the case for the UTF-8 signature, which is defined by the Unicode consortium, for precisely this purpose. Therefore, editors *will* agree on that mechanism, while keeping their own mechanism for the more general problem.
Quoted 7 lines
Quoted 3 lines
Even for the programming language, it is a pain to implement: what if you have non-ASCII characters before the pragma that declares the encoding? and so on.
That's the problem of the language definers who absolutely want such (IMHO absolutely superflous) features.
It's not the language designers who absolutely want this feature. It's the language users. Of course, you'ld have to be a language designer to know that fact - language users go to the language designers asking for the feature, not to the kernel developers.
Quoted 8 lines
Quoted 3 lines
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
The specific feature I get is that when I pass a file starting with <utf8sig>#! to execve, Linux will execute the file following the #!. In what way do I get this feature for text in general? And if I do, why is that a problem?
Quoted 2 lines
If you think "script" and "text file" are different, define both of them, please, otherwise a discussion is pointless.
A script file (in the context of this discussion) is a text file that is executable (i.e. has the appropriate subset of S_IXUSR|S_IXGRP|S_IXOTH set), starts with #!, and has the path name of an executable file after the #!.
More generally, a script file is a text file written in a scripting language. A scripting language is a programming language which supports "direct" execution of source code. So in the more general definition, a script file does not need to start with #!; for the context of this discussion, we should restrict attention to files actually affected by the patch.
Quoted 8 lines
Quoted 4 lines
This conclusion is false. Many tools that don't understand the file structure still can do their job on the files. So the fact that a tool does not understand the structure does not necessarily imply that the tool breaks when the structure changes.
It *may* break just because of some to-be-ignored inline marking due to some questionable feature.
Be more specific. For what specific kind of file will cat(1) break? Unless cat(1) has a 2GB limitation, I very much doubt it will break (i.e. fail to do its job, "concatenate files and print on the standard output") for any kind of input - whether this is text files, binary files, images, sound files, HTML files. cat always does what it is designed to do.
Quoted 2 lines
Let alone the confusion why the size of a file with
ls -lis different from the size in the editor or a marker-awarewc -c.
This is true for any UTF-8 file, or any multibyte encoding. For any multibyte encoding, the number of bytes in the file is different from the number of characters. That doesn't (and shouldn't) stop people from using multi-byte encodings.
What the editor displays as the number of "things" is up to its own. The output of wc -c will always be the same as the one of ls -l, as wc -c does *not* give you characters:
-c, --bytes
print the byte counts
You might have been thinking of 'wc -m'.
Quoted 6 lines
Quoted 1 line
For a Python script, I don't need to guess: It will just work.
Then write a short python script (with a "#!/usr/bin/python" line at the start [without parameters]) natively on a Win*-system, copy it binary over to an arbitrary Linux system and see what's happening.
It depends on the editor I use, of course: the kernel will consider any CR after the n as part of the interpreter name. Not sure what this has to do with the specific patch, though.
Regards,
Martin
On Sun, 2005-09-18 at 09:23 +0200, "Martin v. Löwis" wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Hmm. What does that have to do with the patch I'm proposing? This patch does *not* interfere with all text files. It is only relevant for executable files starting with the #! magic.
Quoted 8 lines
Quoted 3 lines
It *does* interfere since scripts are also text files in every aspect. So every feature you want for "scripts" you also get for text files (and vice versa BTW).
The specific feature I get is that when I pass a file starting with <utf8sig>#! to execve, Linux will execute the file following the #!. In what way do I get this feature for text in general? And if I do, why is that a problem?
After applying this patch it seems that "Linux" is supporting this marker officially in general - especially if the kernel supports it. I suppose the next kernel patch is to support Win-like CR-LF sequences (which is not the case AFAIK).
BTW even some standards body thinks that this is the way to go, it raises more problems and questions than resolves anything.
Quoted 8 lines
Quoted 2 lines
If you think "script" and "text file" are different, define both of them, please, otherwise a discussion is pointless.
A script file (in the context of this discussion) is a text file that is executable (i.e. has the appropriate subset of S_IXUSR|S_IXGRP|S_IXOTH set), starts with #!, and has the path name of an executable file after the #!.
Quoted 6 lines
More generally, a script file is a text file written in a scripting language. A scripting language is a programming language which supports "direct" execution of source code. So in the more general definition, a script file does not need to start with #!; for the context of this discussion, we should restrict attention to files actually affected by the patch.
And though scripts are usually edited/changed/"parsed"/... with an text editor, it is not always the case. Therefore the automatic extension to *all text files* (especially as the marker basically applies to all text files, not only scripts).
You want to focus just on your patch and ignore the directly implied potential problems arising ...
[...]
Quoted 4 lines
Quoted 2 lines
It *may* break just because of some to-be-ignored inline marking due to some questionable feature.
Be more specific. For what specific kind of file will cat(1) break?
cat as such will not break (as such).
Quoted 5 lines
Unless cat(1) has a 2GB limitation, I very much doubt it will break (i.e. fail to do its job, "concatenate files and print on the standard output") for any kind of input - whether this is text files, binary files, images, sound files, HTML files. cat always does what it is designed to do.
Apparently I have to repeat: If you do cat a.txt b.txt >c.txt where a.txt and b.txt have this marker, then c.txt have the marker of b.txt somewhere in the middle. Does this make sense in anyway? How do I get rid of the marker in the middle transparently?
Quoted 7 lines
Quoted 2 lines
Let alone the confusion why the size of a file with
ls -lis different from the size in the editor or a marker-awarewc -c.This is true for any UTF-8 file, or any multibyte encoding. For any multibyte encoding, the number of bytes in the file is different from the number of characters. That doesn't (and shouldn't) stop people from using multi-byte encodings.
It is different even if a pure ASCII file is marked as UTF-8. And sure, the problem exists in general with multi-byte encodings.
Quoted 8 lines
What the editor displays as the number of "things" is up to its own. The output of wc -c will always be the same as the one of ls -l, as wc -c does *not* give you characters:
-c, --bytes
print the byte countsYou might have been thinking of 'wc -m'.
It depends on the definition of "character". There are other standards which define "character" as "byte".
[...]
Quoted 5 lines
Quoted 3 lines
Then write a short python script (with a "#!/usr/bin/python" line at the start [without parameters]) natively on a Win*-system, copy it binary over to an arbitrary Linux system and see what's happening.
It depends on the editor I use, of course: the kernel will consider any
No, more on the OS the editor runs on.
Quoted 1 line
CR after the n as part of the interpreter name. Not sure what this has
ACK.
Quoted 1 line
to do with the specific patch, though.
It is not supported by the kernel. So either you remove it or you make some compatibility hack (like an appropriate sym-link, etc.). Since the kernel can start java classes directly, you can probably make a similar thing for the UTF-8 stuff.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 6 lines · Bodo Eggert
BTW2: However, I don't like the patch.
I'd first check for a utf-8 signature, and if it's found, adjust the buffer offset by 3. Then I'd run the old code checking for the
sh_bang. OTOH, I just read the patch and not the .c file, maybe (unlikely) my idea wouldn't work correctly.
I believe this wouldn't work. binfmt_script currently has the code
for (cp = bprm->buf+2; (*cp == ' ') || (*cp == '\t'); cp++);
to get out the (start of the) interpreter file name. This knows implicitly that you need to skip two bytes #!; for UTF-8 signatures, it would be 5 bytes.
Now, if you meant to suggest that bprm->buf should be adjusted (e.g. through 'brpm->buf += 3'): this cannot work, either. It would break subsequent binfmt modules which assume that bprm->buf is the first 1KiB (or so) of the file to be executed.
If you suggest that the patch should merely check for the signature, and then skip it: this is what the patch does.
Regards,
Martin
P.S. I just noticed there is a
bprm->buf[BINPRM_BUF_SIZE - 1] = '\0';
which seems incorrect: it puts a null-byte into the buffer data, thus (slightly) corrupting the data for subsequent binfmt modules (although it already knows the file starts with #!, so the
subsequent modules will fail, anyway)
Also, I think the above loop should also terminate for '
*cp == '\0'
if there is neither a space nor a tab in the file.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · D. Hazelton
Quoted 3 lines
I would need to write a compiled C program to do all sorts of fragile hackish things like calling a script
/sbin/init.sh.Problem is, the program
would not be fragile or hackish - it'd be almost as simple as a "hello world" program.
Quoted 6 lines
#include <
unistd.h>int
main(){
/* if this fails the system is busted anyway */
returnexecve( "/bin/sh", "/sbin/init.sh", 0 );
};
This attempt nicely illustrates Kyle's point. This program *is* fragile and hackish. It is fragile because, even though it is only five lines, contains two major bugs:
1. execve takes an argv array, not a null-terminated list of strings. So this compiles with a warning about incompatible pointer types; you meant to use execl(3).
2. In the exec family, the path to the program is different from argv[0]. So the correct line would be
return execl("/bin/sh", "sh", /sbin/init.sh", 0);It is hackisch, because it also lacks a feature commonly found in such wrappers:
3. arguments passed to the wrapper are not forwarded to the executable. In particular, init takes several arguments (e.g. the runlevel), which should be forwarded to the final executable.
Just try completing the wrapper on your own.
Regards,
Martin