Skip to contentSkip to conversation

Support UTF-8 scripts

PATCH Bodo Eggert 8 participants 13 messages Sep 16, 2005 – Sep 19, 2005 7 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 7 lines · Bernd Petrovitsch
Quoted 6 lines · Martin v. Löwis
Quoted 4 lines · H. Peter Anvin

In Unix, it's a hideously bad idea.  The reason is that Unix inherently assumes that text streams can be merged, split, and modified.  In other words, unless you can guarantee that EVERY program can handle BOM EVERYWHERE, it's broken.

You can't sort /bin/ls into /tmp/ls and expect /tmp/ls to be meaningfull, but /bin/ls works as expected. You can't usurally concat perl scripts and shell scripts either, but both kinds of script run quite well.

And if you do "cat /bin/cat /bin/cp > /bin/catcp", what's "catcp foo bar" supposed to do? First output foo and bar to stdout, then copy foo to bar? Is execve() broken if it doesn't do what I described? Is the ELF header broken because it's not recogmized EVERYWHERE? I don't think so.

Quoted 8 lines
Quoted 2 lines

This argument is bogus. We are talking about scripts here, which cannot be merged, split, and modified. You don't cat(1) or sort(1) them - it's

Sure they can since they are plain text files.
How do you think one merges scripts?
Just cating them all into one new file and edit that new file is much faster and simpler than to open an empty new file with your editor, then you open all the other scripts in your editor and copy them by hand.

What's supposed to happen if you concatenate a script from your french user and from your russian user, both using localized text, into one file? Unless you can guarantee every editor to correctly handle this case, all usage of 8-bit-characters should be disabled - NOT!

If you concatenate two plain text files, you will use cat. If you concatenate two pnm image files, you will use pnmcat. If you concatenate two utf-8 files, you will use utf8cat. If you concatenate two binaries, you will shoot your feet. That's easy, isn't it?

BTW: I think decent utf-8 capable programs SHOULD ignore extra BOM markers.

Quoted 1 line

And you (or at least I) do grep/egrep/fgrep, wc them.

You can *grep utf-8 scripts, but you can't *grep binaries. Shouldn't this be fixed by implementing an in-kernel ASCII assembler and convert all binaries to assembler text?

Quoted 3 lines

And
probably with several other tools too - think of find <dir> -type f -print0 | xargs -0r <cmd>.

utf-8 filenames will work correctly (unless used as an extended BASIC script with non-ASCII variable names, but that would be insane).

Quoted 5 lines
Quoted 2 lines

just pointless to do that. You create them with text editors, and those can handle the UTF-8 signature.

It is not uncommon to create scripts and the like with other programs, other scripts, what-else.

It's not uncommon to create binaries using other programs. So what?

Quoted 2 lines

Apart from the fact the a "script" is merely a plain text file with the eXecutable bit set.

And an utf-8 script is a utf-8 encoded text file with it's executable bit set.

Quoted 2 lines

And that is the only difference, so you have to at least (all instances of) chmod to insert and remove the BOM.

[...]

In order to make it harder for the interpreter to correctly detect utf-8? You can have DOS executables run in dosboxes, windows applications run in windows, java archives run in java, but utf-8 scripts should be mangled in order to work "correctly", and mangled back in order to be editable? *That*'s insane!

Just make execve ignore the BOM marker before "#!" as the patch does, and you're done. The rest is somebody else's not-a-problem.

BTW2: However, I don't like the patch.

I'd first check for a utf-8 signature, and if it's found, adjust the buffer offset by 3. Then I'd run the old code checking for the sh_bang. OTOH, I just read the patch and not the .c file, maybe (unlikely) my idea wouldn't work correctly.

Signature · 2 lines
Ich danke GMX dafür, die Verwendung meiner Adressen mittels per SPF
verbreiteten Lügen zu sabotieren.
#7 lore raw
Quoted 6 lines · Bodo Eggert

What's supposed to happen if you concatenate a script from your french user and from your russian user, both using localized text, into one file? Unless you can guarantee every editor to correctly handle this case, all usage of 8-bit-characters should be disabled - NOT!

Actually, it's quite easy to avoid problems by using UTF-8 consistently.
The 8-bit characters are oddballs and need to be treated specially,
but look, guys, it's 2005 - UTF-8 should be the norm, not the exception.

-hpa

#8 lore raw
Quoted 10 lines · H. Peter Anvin
Quoted 4 lines · Bodo Eggert

What's supposed to happen if you concatenate a script from your french user and from your russian user, both using localized text, into one file? Unless you can guarantee every editor to correctly handle this case, all usage of 8-bit-characters should be disabled - NOT!

Actually, it's quite easy to avoid problems by using UTF-8 consistently.
The 8-bit characters are oddballs and need to be treated specially,
but look, guys, it's 2005 - UTF-8 should be the norm, not the exception.

It should, but as long as old programs are still around, we'll have both and need a marker to distinguish them. Otherwise we'll be stuck with legacy scripts for a long time.

Signature · 2 lines
I'm a member of DNA (National Assocciation of Dyslexics).
	-- Storm in <[email protected]>
#9 lore raw

Hello!

Quoted 2 lines

It should, but as long as old programs are still around, we'll have both and need a marker to distinguish them.

I doubt that. For ages people were using several different encodings on a single system (at least here in .cz) without any markers and although there were some rough edges, almost everything worked. Now we do the same with ISO-8859-2 and UTF-8, again with no need for a marker.

Have a nice fortnight

Signature · 3 lines
Martin `MJ' Mares   <[email protected]>   http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
Linux vs. Windows is a no-WIN situation.
#10 lore raw
In reply to Bodo Eggert
Quoted 4 lines · Bodo Eggert

It should, but as long as old programs are still around, we'll have both and need a marker to distinguish them. Otherwise we'll be stuck with legacy scripts for a long time.

You don't have markers (although they're defined, see ISO 2022) for your 8-bit encodings, and *THEY'RE THE ONES THAT NEED TO BE DISTINGUISHED.* Flagging UTF-8, especially with the BOM (as opposed to the ISO 2022 signature, <ESC>%G) is pointless in the context, since you still can't distinguish your arbitrary number of legacy encodings.

Oh, yes, and try to stick ISO 2022 signatures in scripts or whatnot, and you can see what current software does with a signature standard that dates back to the 1970's.

-hpa

#11 lore raw
In reply to Bodo Eggert
Quoted 2 lines · Bodo Eggert
Quoted 1 line · H. Peter Anvin

Bodo Eggert wrote:

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Unless you can guarantee every editor to correctly handle this case, all usage of 8-bit-characters should be disabled - NOT!

Actually, it's quite easy to avoid problems by using UTF-8 consistently.
The 8-bit characters are oddballs and need to be treated specially,
but look, guys, it's 2005 - UTF-8 should be the norm, not the exception.

Right.

Quoted 3 lines

It should, but as long as old programs are still around, we'll have both and need a marker to distinguish them. Otherwise we'll be stuck with legacy scripts for a long time.

Please. Let people who mess with legacy stuff suffer, don't make everybody else (and forevermore!) pay the price.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw
Quoted 5 lines · D. Hazelton

This is a bogus argument. You're comparing the way a _binary_ executable works to the way an interpreted _text_ script works. execve(), at least on my system, isn't capable of running a script - if I want to do that from a program I have to tell execve() that it's running /bin/sh and the script file is in the parameter list.

This being the linux-kernel list, I assume your system is Linux, no? Well, on Linux, execve *does* support script files. This is the whole point of my patch - I would not propose a kernel patch to improve this support if it weren't there in the first place.

Quoted 4 lines

While I appreciate that the kernel is capable of performing complex actions when execve runs into a file that is not an a.out or elf binary I have yet to see a "binfmt script" option in the kernel config files ever.

It's not a config option because it is always enabled. See fs/binfmt_script.c for details. It wasn't integrated into the binfmt system until I made it so some ten years ago, though.

Quoted 4 lines

On the other hand, there is the "binfmt_misc" option, which does the work that you seem to be looking for and can, AFAIK, be set to handle both ASCII and UTF-8 scripts. Why add the complexity to the kernel when it's not needed?

One shouldn't add complexity if its not needed. However, this patch does not add complexity. It is fairly trivial.

Regards,
Martin

#14 lore raw
Quoted 8 lines · Martin v. Löwis
Quoted 6 lines · D. Hazelton

This is a bogus argument. You're comparing the way a _binary_ executable works to the way an interpreted _text_ script works. execve(), at least on my system, isn't capable of running a script - if I want to do that from a program I have to tell execve() that it's running /bin/sh and the script file is in the parameter list.

Quoted 4 lines

This being the linux-kernel list, I assume your system is Linux, no? Well, on Linux, execve *does* support script files. This is the whole point of my patch - I would not propose a kernel patch to improve this support if it weren't there in the first place.

This is news to me. The last time I handed execve() a script as a paramter I had errors returned from execve() -- I must admit that this was not on my current system and I had assumed that the behavior would be consistent.

Quoted 8 lines
Quoted 4 lines

While I appreciate that the kernel is capable of performing complex actions when execve runs into a file that is not an a.out or elf binary I have yet to see a "binfmt script" option in the kernel config files ever.

It's not a config option because it is always enabled. See fs/binfmt_script.c for details. It wasn't integrated into the binfmt system until I made it so some ten years ago, though.

I haven't gotten into that section of the code yet. I've been slowly working my way through the code from the drivers that seem to cause strange behavior on my system and then up the tree from there.

Quoted 7 lines
Quoted 4 lines

On the other hand, there is the "binfmt_misc" option, which does the work that you seem to be looking for and can, AFAIK, be set to handle both ASCII and UTF-8 scripts. Why add the complexity to the kernel when it's not needed?

One shouldn't add complexity if its not needed. However, this patch does not add complexity. It is fairly trivial.

You are correct. It is fairly trivial. However my point still is valid that the Kernel has the whole binfmt_misc system -- I will admit that I have recently been shown numbers that show a noticeable difference in the speed of a binary executed using the binfmt_misc system and the binfmt_script system, but the fact remains that offering handling for UTF8 and ASCII scripts directly in the kernel will likely lead to at least one more patch in which the the full Unicode standard is implemented.

That, and my point remains that the kernel should know absolutely nothing about how to execute a text file - the kernel should return an error to the extent of "I don't know what to do with this file" to the shell that tries to execute it, and the shell can then check for the sh_bang. I do admit that this change would break a lot of existing code, so I'll leave the argument to the experts.

Quoted 2 lines

Regards,
Martin

DRH

#15 lore raw
Quoted 1 line · D. Hazelton

That, and my point remains that the kernel should know absolutely nothing about how to execute a text file - the kernel should return an error to the extent of "I don't know what to do with this file" to the shell that tries to execute it, and the shell can then check for the sh_bang. I do admit that this change would break a lot of existing code, so I'll leave the argument to the experts.

No, that would not work at all. We have a very nice system to allow set-uid scripts (Specifically, I like my nice secure taint-mode set-uid perl scripts). If you did this, they would break completely, not to mention _add_ all sorts of unsolvable race conditions to the few ways of working around such a lack of SUID scripts. Also, it means that I can't just "mv /sbin/init /sbin/init.real ; vim /sbin/init" to do a simple wrapper around the init program, I would need to write a compiled C program to do all sorts of fragile hackish things like calling a script /sbin/init.sh.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.
-- C.A.R. Hoare

#16 lore raw
Quoted 9 lines · Kyle Moffett
Quoted 7 lines · D. Hazelton

That, and my point remains that the kernel should know absolutely nothing about how to execute a text file - the kernel should return an error to the extent of "I don't know what to do with this file" to the shell that tries to execute it, and the shell can then check for the sh_bang. I do admit that this change would break a lot of existing code, so I'll leave the argument to the experts.

Quoted 10 lines

No, that would not work at all. We have a very nice system to allow set-uid scripts (Specifically, I like my nice secure taint-mode set- uid perl scripts). If you did this, they would break completely, not to mention _add_ all sorts of unsolvable race conditions to the few ways of working around such a lack of SUID scripts. Also, it means that I can't just "mv /sbin/init /sbin/init.real ; vim /sbin/init" to do a simple wrapper around the init program, I would need to write a compiled C program to do all sorts of fragile hackish things like calling a script /sbin/init.sh.

This makes a lot more sense than I expected to hear. This argument alone is enough for me to understand the reasoning behind the kernel knowing how to interpret a shell script. Problem is, the program would not be fragile or hackish - it'd be almost as simple as a "hello world" program.

#include <unistd.h>

int main() {
/* if this fails the system is busted anyway */
return execve( "/bin/sh", "/sbin/init.sh", 0 );
};

-- This program would do the trick nicely, and since init is run as root, there is no need to worry about the program having to grab privs.

However, the real problem is that this would break the initrd systems used by most distributions for installation, and it would probably break most of the "early userspace" systems just coming into use. As I said originally - my comment about having the shell itself interpret the sh_bang would break a lot of stuff and I've been shown that I have to spend more time in the kernel code (as I haven't finished going through the various drivers to see how those have been made to work) before I can make a good suggestion in a discussion like this.

DRH

#17 lore raw
In reply to D. Hazelton
Quoted 4 lines · D. Hazelton

This is news to me. The last time I handed execve() a script as a paramter I had errors returned from execve() -- I must admit that this was not on my current system and I had assumed that the behavior would be consistent.

The kernel checks for #!<path>, and that <path> is an existing executable. If not, execve fails.

Quoted 8 lines

You are correct. It is fairly trivial. However my point still is valid that the Kernel has the whole binfmt_misc system -- I will admit that I have recently been shown numbers that show a noticeable difference in the speed of a binary executed using the binfmt_misc system and the binfmt_script system, but the fact remains that offering handling for UTF8 and ASCII scripts directly in the kernel will likely lead to at least one more patch in which the the full Unicode standard is implemented.

The problem with the binfmt_misc approach is that you need *another* execve call: with binfmt_misc, you register <utf8sig>#!, and a generic binary. Then, this generic binary will interpret the #! signature *again*, and invoke the proper interpreter. This will intepret the first line *yet again* (finding that it is a comment), and continue processing the file.

However, this is not the real problem. The real problem is that the specific binfmt_misc "backend" would not be universally available, and then the same script would start on some systems, and break on others. This may be acceptable for large or specific applications (e.g. you have to setup the ibcs2 module to run SCO applications); it is not for scripts.

Now, the "universally available" part would not apply right now, as only the most recent kernels would provide the feature. However, within a few years, the feature would be part of "Linux" - then people can start using it extensively.

Quoted 6 lines

That, and my point remains that the kernel should know absolutely nothing about how to execute a text file - the kernel should return an error to the extent of "I don't know what to do with this file" to the shell that tries to execute it, and the shell can then check for the sh_bang. I do admit that this change would break a lot of existing code, so I'll leave the argument to the experts.

The point is that it is not necessarily the shell which starts programs - the shell is but one creator of new processes. It is very common today that, say, httpd starts new programs - this mechanism is called CGI. Your approach was in use until 1985 or so, when Unix implementations started to support #! natively. This was done both for convenience and for performance: if programs would always use system(3) to start new processes, there would always be a shell that execs the eventual interpreter.

I'm not sure, but I believe that most current shells have "forgotten" how to do the #! magic, since, by now, "traditionally" this is a kernel responsibility.

Regards,
Martin

#18 lore raw
Quoted 18 lines · Martin v. Löwis
Quoted 4 lines · D. Hazelton

This is news to me. The last time I handed execve() a script as a paramter I had errors returned from execve() -- I must admit that this was not on my current system and I had assumed that the behavior would be consistent.

The kernel checks for #!<path>, and that <path> is an existing executable. If not, execve fails.

Quoted 8 lines

You are correct. It is fairly trivial. However my point still is valid that the Kernel has the whole binfmt_misc system -- I will admit that I have recently been shown numbers that show a noticeable difference in the speed of a binary executed using the binfmt_misc system and the binfmt_script system, but the fact remains that offering handling for UTF8 and ASCII scripts directly in the kernel will likely lead to at least one more patch in which the the full Unicode standard is implemented.

Quoted 6 lines

The problem with the binfmt_misc approach is that you need *another* execve call: with binfmt_misc, you register <utf8sig>#!, and a generic binary. Then, this generic binary will interpret the #! signature *again*, and invoke the proper interpreter. This will intepret the first line *yet again* (finding that it is a comment), and continue processing the file.

True. I had forgotten that for truly generic rules about handling the #! there would be double the overhead for the sh_bang.

Quoted 6 lines

However, this is not the real problem. The real problem is that the specific binfmt_misc "backend" would not be universally available, and then the same script would start on some systems, and break on others. This may be acceptable for large or specific applications (e.g. you have to setup the ibcs2 module to run SCO applications); it is not for scripts.

Again this is all too true. Doubly so with the problem of an initrd that has 'init' as a script.

Quoted 4 lines

Now, the "universally available" part would not apply right now, as only the most recent kernels would provide the feature. However, within a few years, the feature would be part of "Linux" - then people can start using it extensively.

This sounds to me like you're saying in a few years my suggestion of using binfmt_misc would be tenable. Unfortunately, unless forced into it, no distro would ever use it. As I now see it, binfmt_script is pretty much a hard-coded hack that gives the system a bit more speed for running scripts. And since I've thought about the consequences of ripping it out after the posts yesterday - there is no clean way to remove it and still have a large number of systems still function.

Quoted 8 lines
Quoted 7 lines

That, and my point remains that the kernel should know absolutely nothing about how to execute a text file - the kernel should return an error to the extent of "I don't know what to do with this file" to the shell that tries to execute it, and the shell can then check for the sh_bang. I do admit that this change would break a lot of existing code, so I'll leave the argument to the experts.

Quoted 9 lines

The point is that it is not necessarily the shell which starts programs - the shell is but one creator of new processes. It is very common today that, say, httpd starts new programs - this mechanism is called CGI. Your approach was in use until 1985 or so, when Unix implementations started to support #! natively. This was done both for convenience and for performance: if programs would always use system(3) to start new processes, there would always be a shell that execs the eventual interpreter.

True. In some cases, though, system(3) is really unusable - like you mentioned, httpd often starts new processes. Since daemons don't, technically, run on top of a shell, having one use system(3) to start a new process would add a lot of unnecessary overhead.

Quoted 3 lines

I'm not sure, but I believe that most current shells have "forgotten" how to do the #! magic, since, by now, "traditionally" this is a kernel responsibility.

Not true. Bash, at least, still handles the sh_bang. (Provable by using it to call a perl script that doesn't have the exec bit set. This worked for me just a week ago :)

DRH

#19 lore raw
In reply to a message not in the archive
Quoted 16 lines · D. Hazelton
Quoted 15 lines · Bodo Eggert
Quoted 8 lines · Bernd Petrovitsch
Quoted 7 lines · Martin v. Löwis
Quoted 5 lines · H. Peter Anvin

In Unix, it's a hideously bad
idea.  The reason is that Unix inherently assumes that text streams can be merged, split, and modified.  In other words, unless you can guarantee that EVERY program can handle BOM EVERYWHERE, it's broken.

You can't sort /bin/ls into /tmp/ls and expect /tmp/ls to be meaningfull, but /bin/ls works as expected. You can't usurally concat perl scripts and shell scripts either, but both kinds of script run quite well.

Quoted 8 lines
Quoted 5 lines

And if you do "cat /bin/cat /bin/cp > /bin/catcp", what's "catcp foo bar" supposed to do? First output foo and bar to stdout, then copy foo to bar? Is execve() broken if it doesn't do what I described? Is the ELF header broken because it's not recogmized EVERYWHERE? I don't think so.

This is a bogus argument. You're comparing the way a _binary_ executable works to the way an interpreted _text_ script works.

You can live with binaries, therefore the features not provided by binaries aren't vital for each and every executable.

Quoted 3 lines

execve(), at least on my system, isn't capable of running a script - if I want to do that from a program I have to tell execve() that it's running /bin/sh and the script file is in the parameter list.

Fix your system, it's broken.

Quoted 4 lines

While I appreciate that the kernel is capable of performing complex actions when execve runs into a file that is not an a.out or elf binary I have yet to see a "binfmt script" option in the kernel config files ever.

Your wish ... but you won't be happy.

fs/Makefile renamed from t/linux-2.6.12/fs/Makefile +1 -3
@@ -20,9 +20,7 @@ obj-y				+= $(nfsd-y) $(nfsd-m)
 obj-$(CONFIG_BINFMT_AOUT)	+= binfmt_aout.o
 obj-$(CONFIG_BINFMT_EM86)	+= binfmt_em86.o
 obj-$(CONFIG_BINFMT_MISC)	+= binfmt_misc.o
-
-# binfmt_script is always there
-obj-y				+= binfmt_script.o
+obj-$(CONFIG_BINFMT_SCRIPT)	+= binfmt_script.o
 
 obj-$(CONFIG_BINFMT_ELF)	+= binfmt_elf.o
 obj-$(CONFIG_BINFMT_ELF_FDPIC)	+= binfmt_elf_fdpic.o
fs/Kconfig.binfmt renamed from t/linux-2.6.12/fs/Kconfig.binfmt +6 -0
@@ -42,6 +42,12 @@ config BINFMT_FLAT
 	help
 	  Support uClinux FLAT format binaries.
 
+config BINFMT_SCRIPT
+       bool "Kernel support for script files"
+       default y
+       help
+         Support script files starting with a '#!' marker.
+
 config BINFMT_ZFLAT
 	bool "Enable ZFLAT support"
 	depends on BINFMT_FLAT
Quoted 4 lines

On the other hand, there is the "binfmt_misc" option, which does the work that you seem to be looking for and can, AFAIK, be set to handle both ASCII and UTF-8 scripts. Why add the complexity to the kernel when it's not needed?

Skipping 3 bytes vs. handling tons of binary formats? I bet the memory required to hold the utf8 binfmt_misc entry alone will be bigger than the code added by this patch.

Quoted 8 lines
Quoted 2 lines

BTW: I think decent utf-8 capable programs SHOULD ignore extra BOM markers.

All well and good if you use UTF-8. I, personally, am happy with ASCII and have found no need for the extensive UTF character set (in fact, I despise it when people insist on using UTF-8 in mediums in which the character set is defined in the standards to be ASCII or a subset of ASCII)

I'm not using it, because nobody else is using it, and evrybody else does the same for exactly the same reasons. That's why just using utf-8 does not work out.

However, if there were means of using both transparently, people could migrate. The editor part is simple, but if you can't use your favorite editor to generate shell scripts, it's a showstopper.

Quoted 4 lines

Since I am quite happy with the small subset of ASCII that I use on a regular basis, and since I am always seeking ways to optimize my code and my scripts I don't want the editor I'm using adding extra characters behind my back.

ACK. But you should be able to edit international text without tons of helper scripts, so a BOM will be usefull to mark utf-8.

Quoted 10 lines
Quoted 5 lines
Quoted 1 line

And you (or at least I) do grep/egrep/fgrep, wc them.

You can *grep utf-8 scripts, but you can't *grep binaries. Shouldn't this be fixed by implementing an in-kernel ASCII assembler and convert all binaries to assembler text?

Bogus argument. Every shell I've ever used has expected the command line to contain only ASCII characters. With that restriction in mind it's clear that it'd be hard to put a UTF8 string as the argument to grep. Although I doubt wc would be buggered by UTF8 input...

If your shell isn't 8-bit-clean, it should have been replaced in the last millenium. Handling combined characters will be the problem.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

And
probably with several other tools too - think of find <dir> -type f -print0 | xargs -0r <cmd>.

utf-8 filenames will work correctly (unless used as an extended BASIC script with non-ASCII variable names, but that would be insane).

Quoted 6 lines

This is the truth. As I previously mentioned I have yet to find a shell that accepted UTF8 on the command line without choking. And allowing UTF8 for filenames would, I believe, require any number of changes to the kernel, not the least of which would be changes to the various filesystems to allow for UTF8 and to any number of system calls that would be taking a filename for an argument.

It's not a task of allowing utf-8 filenames, but a task of disallowing non-canonialized and non-utf8 filenames if files might be created. Systems doing that won't be a strictly POSIX conformant, but as long as there is a mounted FAT partition, it can't be anyway.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

just pointless to do that. You create them with text editors, and those can handle the UTF-8 signature.

It is not uncommon to create scripts and the like with other programs, other scripts, what-else.

Quoted 7 lines
Quoted 1 line

It's not uncommon to create binaries using other programs. So what?

Bullsh*t. The case of one binary creating another doesn't apply - because you either enter the data for the binary by hand (tedious and difficult) or you use a binary that takes input and produces the binary you need. And if the binary is missing the proper headers, it's pretty much useless.

And you can live with binaries being non-editable, non-generatable without propper tools.

Quoted 4 lines

When a script creates another script it is
just creating a text file, putting the data in the file as it reaches those parts and has no way to know that it should be inserting the BOM.

If scripts are just text files, why doesn't sort<script|sh usurally do the right thing?

Scripts are _not_ random text, they have specific structures. They consist of well-formed data, and you should better know what kind of script you're creating, therefore you should also know wether to write sh_bang or BOM_sh_bang. If you don't, don't generate the script!

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Apart from the fact the a "script" is merely a plain text file with the eXecutable bit set.

And an utf-8 script is a utf-8 encoded text file with it's executable bit set.

Quoted 9 lines

And the kernel should have no more code in it to execute them than is already present in the binfmt_misc code. No need for special kernel code when you can simply hand a chunk of parameters regarding the various executable formats to the kernel using a clean, simple and proven interface. And even then I feel it should be limited to binaries - a script is, by definition, interpreted. As such, it belongs in the same place as the interpreter - in userland. (And I fail to see why this is even brought up other than some people being lazy and not wanting to do things _correctly_)

So the binfmt_sh code should be completely abandoned in favor of binfmt_misc?

Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

And that is the only difference, so you have to at least (all instances of) chmod to insert and remove the BOM.

[...]

In order to make it harder for the interpreter to correctly detect utf-8? You can have DOS executables run in dosboxes, windows applications run in windows, java archives run in java, but utf-8 scripts should be mangled in order to work "correctly", and mangled back in order to be editable? *That*'s insane!

Quoted 7 lines
Quoted 2 lines

Just make execve ignore the BOM marker before "#!" as the patch does, and you're done. The rest is somebody else's not-a-problem.

GCC allows for non-ascii input as a formality. The specifications of both C and C++ clearly define the input character set to be limited to an extremely limited subset of ASCII, as do the specifications of most other language.

This is a userspace problem.

Quoted 3 lines

(Perl 6 is the first language I've ever heard of
that directly includes non-ascii characters in the accepted character set)

The MS-DOS 3.3 shell accepted international characters in program names.

Quoted 3 lines

AFAIK, the most common shells don't accept UTF-8 in the command set - they instead see the non-ascii UTF-8 characters as a series of bytes, and if one of them happens to be NULL, you're pretty much screwed.

There is no '\0' in utf-8-encoded data.

Quoted 3 lines
Quoted 1 line

BTW2: However, I don't like the patch.

Neither do I. such a thing doesn't belong in the kernel.

It's better than

- using a legacy wrapper script for each script.

- mangeling each utf8 file before and after editing it

- forcing the world to convert to utf-8 within two weeks

- using a wrapper script around each and every utf-8 script which would unnescensarily throw out pages and wastes CPU cycles while requiring each user to add several KB of kernel code for binfmt_misc and to have the interpreter for the wrapper script installed

I actually created a wrapper script for binfmt_misc and called it a hundres times, here is the result:

$ time for((i=0;i<100;i++));do ./foo > /dev/null;done # with wrapper
real    0m2.350s
user    0m1.808s
sys     0m0.476s
$ time for((i=0;i<100;i++));do ./bar > /dev/null;done # without wrapper
real    0m0.461s
user    0m0.232s
sys     0m0.216s

And I'm sure this script has a bug to exploit.

(foo and bar will ust print "test\n" to stdout)

Signature · 3 lines
"Our parents, worse than our grandparents, gave birth to us who are worse than
they, and we shall in our turn bear offspring still more evil."
	-- Horace (BC 65-8)