Support UTF-8 scripts

This patch adds support for UTF-8 signatures (aka BOM, byte order mark) to binfmt_script. Files that start with EF BF FF # ! are now recognized as scripts (in addition to files starting with # !).

With such support, creating scripts that reliably carry non-ASCII characters is simplified. Editors and the script interpreter can easily agree on what the encoding of the script is, and the interpreter can then render strings appropriately. Currently, Python supports source files that start with the UTF-8 signature; the approach would naturally extend to Perl to enhance/replace the "use utf8" pragma. Likewise, Tcl could use the UTF-8 signature to reliably identify UTF-8 source code (instead of assuming [encoding system] for source code).

Please find the patch attached below.

Regards,
Martin

@@ -1,7 +1,7 @@
 /*
  *  linux/fs/binfmt_script.c
  *
- *  Copyright (C) 1996  Martin von Löwis
+ *  Copyright (C) 1996, 2005  Martin von Löwis
  *  original #!-checking implemented by tytso.
  */
 

@@ -23,7 +23,16 @@ static int load_script(struct linux_binp
        char interp[BINPRM_BUF_SIZE];
        int retval;
 
-       if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!') ||

(bprm->sh_bang))
+ /* It is a recursive invocation. */
+ if (bprm->sh_bang)
+ return -ENOEXEC;
+
+ /* It starts neither with #!, nor with #! preceded by
+ the UTF-8 signature. */
+ if (!(((bprm->buf[0] == '#') && (bprm->buf[1] == '!'))
+ || ((bprm->buf[0] == '\xef') && (bprm->buf[1] == '\xbb')
+ && (bprm->buf[2] == '\xbf') && (bprm->buf[3] == '#')
+ && (bprm->buf[4] == '!'))))

                return -ENOEXEC;
        /*
         * This section does the #! interpretation.
@@ -46,7 +55,8 @@ static int load_script(struct linux_binp
                else
                        break;
        }
-       for (cp = bprm->buf+2; (*cp == ' ') || (*cp == '\t'); cp++);
+       cp = (bprm->buf[0]=='\xef') ? bprm->buf+5 : bprm->buf+2;
+       while ((*cp == ' ') || (*cp == '\t')) cp++;
        if (*cp == '\0')
                return -ENOEXEC; /* No interpreter name found */
        i_name = cp;

Thats great for the perl6 people.
http://dev.perl.org/perl6/doc/design/syn/S03.html says they are going to be using « and » as operators... So I'd imagine that a lot of perl6 scripts would be utf8.

http://dmoz.org/profiles/pollei.html
http://sourceforge.net/users/stephen_pollei/
http://www.orkut.com/Profile.aspx?uid=2455954990164098214
http://stephen_pollei.home.comcast.net/

Is Larry smoking crack? That's one of the worst ideas I've heard in a long time. There's no easy way to enter those at the keyboard!

http://www.cl.cam.ac.uk/~mgk25/unicode.html#input

Lee

I have "setxkbmap -symbols 'en_US(pc102)+gb'" in my ~/.xsession,
and « and » are available as AltGr-z and AltGr-x respectively.

Hugo.

=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 1C335860 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Anyone who claims their cryptographic protocol is secure is ---   
         either a genius or a fool.  Given the genius/fool ratio         
                 for our species,  the odds aren't good.                 

Most keyboards don't have an AltGr key.

Lee

You must be an American. Most old the worlds keyboards have an AltGr key. You'll find that US keyboards have two alt keys to avoid confusing people (like one button mice ;)) but the right one is understood by the X bindings to be "AltGr". Even though the US keyboard is apparently lacking functionality its purely a text label issue

Alan

And those of us who are Mac OS X oriented have patched our console and X keycodes to match the mac way of generating symbols:

Alt-\        = «
Alt-Shift-\  = »
Alt-Shift-+  = ±

If only someone could come up with a good character palette like exists on that OS, something that could generate a wide variety of keysyms, preferably all of UTF-8, and send them to the topmost window.

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things, because that would also stop them from doing clever things.
-- Doug Gwyn

My point exactly, it's idiotic for Perl6 to use these as OPERATORS, the atoms of the language, when there's not even a platform independent way to type them in.

Lee

I anyone had bothered to read the URL in one of the earlier emails you would have seen that '<<' is an accepted alternative spelling.

Wichert.

Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Well, now it's obvious he's just trying to raise the bar for the obfuscated perl contest. If you thought these were fun before, you'll love them with ¥ and « and »!

Lee

.Xmodmap: keycode 117 = MultiKey

and then use [the Windows(R) Context Menu Key],[<],[<] to generate « Cheers :)

Jan Engelhardt

The command line console mappings may not include them by default (you can obviously add them if your keyboard lacks them). The X keyboard however does include compose functionality for » and « and many other symbols that might be useful eg ±

Alan

Alan> The command line console mappings may not include them by Alan> default (you can obviously add them if your keyboard lacks Alan> them). The X keyboard however does include compose functionality Alan> for » and « and many other symbols that might be useful eg ±

Not to mention that many editors, including emacs and vim, have their own support for entering such non-ascii characters no matter what the console or X11 keyboards look like.

-JimC

James H. Cloos, Jr. <[email protected]>

LR> Is Larry smoking crack? That's one of the worst ideas I've heard LR> in a long time. There's no easy way to enter those at the LR> keyboard!

I know folks enjoy trashing Perl these days, but it's not justified in this case. From the Perl6-Bible -
http://search.cpan.org/dist/Perl6-Bible/lib/Perl6/Bible/S03.pod:

For those still living without the blessings of Unicode, that can also be written: << ... >>.

- J<

I know the alternatives are available. That doesn't make it any less idiotic to use non ASCII characters as operators. I think it's a very slippery slope. We write code in ASCII, dammit.

Lee

Yes you and I might write 99.9% of our code in good'ol **American** Standard Code for Information Interchange -- however not all the world is USA. For instance notice the http://de.wikipedia.org/wiki/Umlaut/
in "Löwis"... Seems like lots of Europeans might want a bigger charset, not to mention Asians, Hindus, and whomever else.

http://dmoz.org/profiles/pollei.html
http://sourceforge.net/users/stephen_pollei/
http://www.orkut.com/Profile.aspx?uid=2455954990164098214
http://stephen_pollei.home.comcast.net/

For strings, of course. But there's no need for UTF-8 operators.

Lee

Indeed - this is the main rationale for the patch, of course. People want to write non-ASCII in script primarily in string literals, and (perhaps even more often) in comments. Now, for comments, it wouldn't really matter that the interpreter knows what the encoding is - but the editor would have to know, and the UTF-8 signature primarily helps the editor (*).

Then we are back to the rationale for this patch: if you need the UTF-8 signature to reliably identify the script as being UTF-8 encoded, you then currently cannot easily run it as a script through binfmt_script, as that code requires a script to start with #!.

Regards,
Martin

(*) As I said before: atleast for Python, the UTF-8 signature also has syntactic meaning. It is allowed at the beginning of a file as an addition to the language syntax, and it tells the interpreter that Unicode literals (usually represented internally as UCS-2) are represented as UTF-8 in the source code.

Its a trivial patch and there is a lot to be said for UTF-8 scripts. As to writing code in ascii, the kernel regularly has outbreaks of either UTF-8 or ISO-8859-* especially in the docs directory. Standardising these on UTF-8 would be helpful.

Yes the kernel code is C so ASCII except for the odd abuser of the © symbol.

Alan

<URL: http://www.madore.org/~david/weblog/2004-12.html#d.2004-12-03.0813 >

:-)

     David A. Madore
    ([email protected],
     http://www.madore.org/~david/ )

We write kernel code in ASCII because of patches in e-mail. When a patch is saved (often by a script), it is divorced of the encoding in which e-mail was done. Forwarding of patches then causes them to fail to apply. Everything else can be worked around.

In my experience, the most common case of such patch rejects has to do with a European using a non-UTF-8 encoding for his name, rather than with the copyright symbol.

-- Pete

I think this confirms that the answer is yes. See the following at the above URL:

Since when is the string "!.?|" an operator??? Or "?^", "+|", "~|", "?|", etc. I think Larry's gone off the deep end on this one. It may be an incredibly powerful and expressive language, but it seems _really_ strange, and probably will produce the best Obfuscated-code contest the world has ever seen. (Better even than the Perl5 one).

Cheers,
Kyle Moffett

--
Simple things should be simple and complex things should be possible
-- Alan Kay

I think that was supposed to read:

Note that ?^ is functionally identical to !.
?| differs from ?? in that ?| returns (and so on)

(two separate sentences lacking whitespace between them....

On your keyboard, that is. So what?

My keyboard happen to have no easy way of entering a dollar sign, even though it is in «ascii». That makes sense though, as it is one of those ascii characters that is almost never used in my part of the world.

Still, if I needed to use the «$» when programming, I sure could map it to some key combination. X is nice that way.

Helge Hafting

Followup to:  <[email protected]>
By author:    =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <[email protected]>
In newsgroup: linux.dev.kernel

BOM should not be used in UTF-8. In fact, it shouldn't be used at all.

-hpa