UTF-8 in file systems? xfs/extfs/etc.
Morning!
What Linux supported filesystems support UTF-8 filenames?
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
Is it somehow planned to enable it?
Or are you waiting for patches which do that job?
Greetings,
Nico
Signature · 3 lines
Keep it simple & stupid, use what's available. pgp: 8D0E E27A | Nico Schottelius http://nerd-hosting.net | http://linux.schottelius.org
Quoted 3 lines · Nico Schottelius
Morning!
What Linux supported filesystems support UTF-8 filenames?
AFAIK, the filesystems don't care what you put in the filenames. They just treat is as a sequence of bytes.
Quoted 3 lines
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
Wrong in what way? How did you create the filenames?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 5 lines · Nico Schottelius
Morning!
What Linux supported filesystems support UTF-8 filenames?
Looks like at least xfs and reiserfs are not able of handling them,
I'm using ReiserFS on my media drive, and I can tell you that the
UTF-8 works fine on the filesystem:
hrm@vlad:Les-Granges-Brulées $ ls
Descente-Au-Village.ogg Les-Granges-Brulées.ogg
Générique.ogg L'Hélicoptère.ogg
Hésitation.ogg Reconstitution.ogg
La-Chanson-Des-Grange-Brulées.ogg Rose.ogg
La-Perquisition-Et-Les-Paysans.ogg Théme-De-L'Argent.ogg
La-Vérité.ogg typescript
Le-Car+Le-Chasse-Neige.ogg Une-Morte-Dans-La-Neige.ogg
Le-Juge.ogg Zig-Zag.ogg
Le-Pays-de-Rose.ogg
Quoted 2 lines
As Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
I'd suspect a problem with Apache more than a problem with the
filesystem.
Hugo.
Signature · 5 lines
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 1C335860 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- There's an infinite number of monkeys outside who want to ---
talk to us about this new script for Hamlet
they've worked out! Quoted 7 lines · Nico Schottelius
Morning!
What Linux supported filesystems support UTF-8 filenames?
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
I have no problem creating and accessing files with Chinese names using ls, vi, cat, etc. in (u)xterm on my reiserfs /tmp partition.
Matt
Quoted 1 line · Nico Schottelius
What Linux supported filesystems support UTF-8 filenames?
Filenames, to the kernel, are a sequence of 8-bit things commonly called "bytes" or "octets", excluding '/' and '\0'.
=> Answer: "All of them". (Or at least ext2/reiser ;-)
Quoted 3 lines
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
That's an Apache bug, and/or a problem with your Apache configuration. Talk to Apache people.
Signature · 1 line
Matthias Urlichs
Quoted 7 lines · Matthias Urlichs
Quoted 1 line · Nico Schottelius
What Linux supported filesystems support UTF-8 filenames?
Filenames, to the kernel, are a sequence of 8-bit things commonly called "bytes" or "octets", excluding '/' and '\0'.
You can have "/" in the filename also, though that could be encoded somehow...
Hi,
Mike Fedyk:
Quoted 2 lines
You can have "/" in the filename also, though that could be encoded somehow...
Such encoding isn't valid UTF-8.
Of course you could use ⁄ instead (fractional slash, U+2044). Or perhaps ∕ (division slash, U+2215).
How to visually distinguish these from a / (U+002F) is left as an exercise to the reader. :-/
The fun part about this email is that I'm writing it with plain old vi (ummm.... I _do_ know that there's nothing "plain old" about vim ;-) and I don't see silly square boxes here.
Signature · 1 line
Matthias Urlichs
Quoted 1 line · Mike Fedyk
You can have "/" in the filename also, though that could be encoded somehow...
Maybe you are thinking of KDE's convention with %-encoding, e.g. if I save a web link in KDE it may look like "http://kernel.org/.desktop" in Konqueror, but "http:%2f%2fkernel.org%2f.desktop" with ls. That's on top of whatever character encoding is being used for regular characters.
-- robin
Quoted 10 lines · Mike Fedyk
Quoted 7 lines · Matthias Urlichs
Quoted 1 line · Nico Schottelius
What Linux supported filesystems support UTF-8 filenames?
Filenames, to the kernel, are a sequence of 8-bit things commonly called "bytes" or "octets", excluding '/' and '\0'.
You can have "/" in the filename also, though that could be encoded somehow...
You might be able to have a non-ASCII character that looks like / but not 0x2f.
I for one do not want open("/var/tpm/diddle", O_WRONLY | O_CREAT) to create a file "tpm/diddle" in /var just because /var/tpm doesn't exist. Fortunately what happens is it fails with ENOENT.
I expect UTF-8 to have no multi-byte sequences containing NUL but it might be awkward if a multi-byte sequence contained 0x2F (/). I would hope that the committees chose to avoid using symbol and punctuation byte-codes for alphanumeric sequences.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Quoted 5 lines · jw schultz
I expect UTF-8 to have no multi-byte sequences containing NUL but it might be awkward if a multi-byte sequence contained
0x2F(/). I would hope that the committees chose to avoid using symbol and punctuation byte-codes for alphanumeric sequences.
UTF-8 single-byte sequences are in range 0--127 with obvious mapping to ASCII. All bytes in UTF-8 multi-byte sequences are in range 128--255.
Quoted 13 lines · jw schultz
Quoted 11 lines · Mike Fedyk
Quoted 7 lines · Matthias Urlichs
Quoted 1 line · Nico Schottelius
What Linux supported filesystems support UTF-8 filenames?
Filenames, to the kernel, are a sequence of 8-bit things commonly called "bytes" or "octets", excluding '/' and '\0'.
You can have "/" in the filename also, though that could be encoded somehow...
Quoted 6 lines
You might be able to have a non-ASCII character that looks like / but not
0x2f.I for one do not want
open("/var/tpm/diddle", O_WRONLY | O_CREAT)to create a file "tpm/diddle" in /var just because/var/tpmdoesn't exist.
Just imagine all the possibilities for ambiguous file names it would open up.
Quoted 5 lines
I expect UTF-8 to have no multi-byte sequences containing NUL but it might be awkward if a multi-byte sequence contained
0x2F(/). I would hope that the committees chose to avoid using symbol and punctuation byte-codes for alphanumeric sequences.
IIRC, UTF-8 doesn't use bytes <128 in any multi-byte sequences.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 12 lines · jw schultz
Quoted 10 lines · Mike Fedyk
Quoted 7 lines · Matthias Urlichs
Quoted 1 line · Nico Schottelius
What Linux supported filesystems support UTF-8 filenames?
Filenames, to the kernel, are a sequence of 8-bit things commonly called "bytes" or "octets", excluding '/' and '\0'.
You can have "/" in the filename also, though that could be encoded somehow...
Quoted 7 lines
You might be able to have a non-ASCII character that looks like / but not
0x2f.I for one do not want
open("/var/tpm/diddle", O_WRONLY | O_CREAT)to create a file "tpm/diddle" in /var just because/var/tpmdoesn't exist. Fortunately what happens is it fails with ENOENT.
OK, I stand corrected.
Thanks.
Quoted 3 lines · Matthias Urlichs
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
Are you sure your filenames are in UTF-8 rather than ISO8859-1? If not, then they'll appear as an invalid UTF-8 string and code that expects UTF-8 will be unhappy.
Signature · 1 line
Matthew Garrett | [email protected]
Nico Schottelius <[email protected]> said on Mon, 9 Feb 2004 12:58:52 +0100:
Quoted 6 lines
--GRPZ8SYKNexpdSJ7
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Quoted 7 lines
Morning!
What Linux supported filesystems support UTF-8 filenames?
Looks like at least xfs and reiserfs are not able of handling them, as Apache with UTF-8 as default charset delievers wrong names, when accessing files with German umlauts.
I submitted a bug to the jfs people, because jfs incorrectly returns -EINVAL (this isn't even documented in man pages as a valid return from open()) from an open() on a filename with UTF-8 in it.
See http://www-124.ibm.com/developerworks/bugs/?func=detailbug&bug_id=3838&group_id=35
and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=229308
This was triggered just by upgrading the console-utils package in debian (the problem existed all along, except that when I first made the filesystem a jfs one, I reinstalled from backups, rather than reinstalling debian from scratch)
Signature · 3 lines
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Just don't create a file called -rf. :-)
-- Larry Wall in <[email protected]>JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 11 lines · Tim Connors
I submitted a bug to the jfs people, because jfs incorrectly returns
-EINVAL(this isn't even documented in man pages as a valid return fromopen()) from anopen()on a filename with UTF-8 in it.See http://www-124.ibm.com/developerworks/bugs/?func=detailbug&bug_id=3838&group_id=35
and http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=229308This was triggered just by upgrading the console-utils package in debian (the problem existed all along, except that when I first made the filesystem a jfs one, I reinstalled from backups, rather than reinstalling debian from scratch)
Yeah, JFS has poor default behavior based on CONFIG_NLS_DEFAULT. I attempted to explain why it works that way in the first bug listed above if anyone is curious.
I think the right thing for JFS to do is to change the default behavior to simply store the bytes as they are seen, and to only do charset conversion when the iocharset mount option is explicitly set. This may impact some current users, but they will be able to get the old behavior by setting iocharset to whatever CONFIG_NLS_DEFAULT is set to in the running kernel.
I intend to make this change soon if there are no objections.
Thanks,
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 3 lines · Dave Kleikamp
Yeah, JFS has poor default behavior based on
CONFIG_NLS_DEFAULT. I attempted to explain why it works that way in the first bug listed above if anyone is curious.
I think your suggested fix is good, but it begs the question:
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
I read your response about OS/2, and maybe I'm just slow, but I don't see what that has to do with anything.
Does JFS on AIX have the same buggy behavior?
What behavior was the code originally designed to implement, on OS/2? Why was that behavior chosen rather than "filenames are a string of bytes"?
Feel free to point to a "Design of the OS/2 JFS interface" document if such exists and answers my question. :)
-andy
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Andy Isaacson <[email protected]> said on Wed, 11 Feb 2004 18:45:32 -0600:
Quoted 10 lines · Dave Kleikamp
Quoted 3 lines
Yeah, JFS has poor default behavior based on
CONFIG_NLS_DEFAULT. I attempted to explain why it works that way in the first bug listed above if anyone is curious.I think your suggested fix is good, but it begs the question:
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
Thanks for wording my question better. That was *precisely* the question I was trying to ask :)
Signature · 2 lines
TimC -- http://astronomy.swin.edu.au/staff/tconnors/ Disclaimer: This post owned by the owner
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Andy IsaacsonQuoted 14 lines · Andy Isaacson
Quoted 3 lines · Dave Kleikamp
Yeah, JFS has poor default behavior based on
CONFIG_NLS_DEFAULT. I attempted to explain why it works that way in the first bug listed above if anyone is curious.I think your suggested fix is good, but it begs the question:
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
I read your response about OS/2, and maybe I'm just slow, but I don't see what that has to do with anything.
Quoted 8 lines
Does JFS on AIX have the same buggy behavior?
What behavior was the code originally designed to implement, on OS/2? Why was that behavior chosen rather than "filenames are a string of bytes"?
Feel free to point to a "Design of the OS/2 JFS interface" document if such exists and answers my question. :)
His first link almost explains it.
| In OS/2, the kernel had access to each process's locale | information, and converting the pathnames from the user's | charset to unicode made access to the filesystem very | transparent, even when users used different character sets | on the same computer. | | Unfortunately, in Linux the kernel has no per-process | information to go on, so it uses the charset specified by | CONFIG_NLS_DEFAULT when the kernel is built. Obviously, | this is neither intuitive or generally useful. | | I am considering changing the default behavior to | trivially convert the user-supplied pathnames to utf-16 | when stored in on-disk. This default behavior could be | overridden by specifying the iocharset= mount flag.
Apparently in OS2 they implemented a policy of utf-16 into the kernel so that applications would not have to be as locale aware. This could be called kernel pollution.
For Linux there is no policy except perhaps in userspace. It is up to userspace to determine what the policy will be regarding charset for filename storage. Common practice seems to be utf-8.
Signature · 5 lines
________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: [email protected] Remember Cernan and Schmitt
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 4 lines · jw schultz
For Linux there is no policy except perhaps in userspace. It is up to userspace to determine what the policy will be regarding charset for filename storage. Common practice seems to be utf-8.
Isn't it is the user's locale, whatever that is? I believe my file names use ISO-8859-1 (except ntfs, vfat). In northern/western europe ISO-8859-1 is common. (Sometimes ISO-8859-15 which for all practical purposes is backwards compatible with 8859-1). UTF-8 is gaining terrain though since it is now the default in some distributions even for Nordic languages (causing big problems for those not expecting it).
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Andy IsaacsonQuoted 3 lines · Andy Isaacson
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
Perhaps for the same reason that these other in-tree filesystems are sensitive to the character encoding:
Joliet (ISO-9660 extension), FAT/VFAT, NTFS, BeFS, SMBFS, CIFS.
Those filesystems will also fail, or give unexpected behaviour (such as bytes being changed to '?'), if you pass them names which are not in the appropriate encoding.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 5 lines · you
Quoted 3 lines · Andy Isaacson
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
Quoted 8 lines
Perhaps for the same reason that these other in-tree filesystems are sensitive to the character encoding:
Joliet (ISO-9660 extension), FAT/VFAT, NTFS, BeFS, SMBFS, CIFS.
Those filesystems will also fail, or give unexpected behaviour (such as bytes being changed to '?'), if you pass them names which are not in the appropriate encoding.
Definitely a good reason. It seem many assume file names are a local thing, but this is not so. Now consider the case with an external firewire disk or memory stick created on a machine with iso-8859-1 as the system character set and e.g xfs as the file system. What happens when I hook it up to a new redhat installation that thinks file names are best stored as utf8? Most non-ascii file names aren't even legal in utf8.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 6 lines
Definitely a good reason. It seem many assume file names are a local thing, but this is not so. Now consider the case with an external firewire disk or memory stick created on a machine with iso-8859-1 as the system character set and e.g xfs as the file system. What happens when I hook it up to a new redhat installation that thinks file names are best stored as utf8? Most non-ascii file names aren't even legal in utf8.
Another thing to consider is that you can encode the same character in several ways using utf8, so two filenames could have different byte strings, but evaluate to the same set of unicode characters.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 3 lines · you
Another thing to consider is that you can encode the same character in several ways using utf8, so two filenames could have different byte strings, but evaluate to the same set of unicode characters.
No. That's not UTF-8.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quote from Robin Rosenberg <[email protected]>:
Quoted 6 lines · you
Quoted 3 lines
Another thing to consider is that you can encode the same character in several ways using utf8, so two filenames could have different byte strings, but evaluate to the same set of unicode characters.
No. That's not UTF-8.
Please don't break the CC list on replies.
I'm not sure whether it's valid UTF-8 or not, but it's certainly possible to code, for example, an 'A', (decimal 65), via an escape to a 31-bit character representation. Presumably the majority of UTF-8 parsers would decode the sequence as 65, rather than emit an error.
Also, even ignoring that, how do you handle things like accented characters which can be represented as single characters, or as sequences containing combining characters? Some applications might convert the sequence containing combining characters in to the single character, and others might not.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 4 lines · John Bradford
I'm not sure whether it's valid UTF-8 or not, but it's certainly possible to code, for example, an 'A', (decimal 65), via an escape to a 31-bit character representation. Presumably the majority of UTF-8 parsers would decode the sequence as 65, rather than emit an error.
There are many ways of getting things wrong. The algorithm for encoding UTF-8 doesn't give you the option of encoding 65 as two bytes; any UCS-4 character with code 0-0x7F must result in a onand the same principle goes for every other character and the unicdeo standard forbids the use of anything but the shortest possible sequence.
Quoted 5 lines
Also, even ignoring that, how do you handle things like accented characters which can be represented as single characters, or as sequences containing combining characters? Some applications might convert the sequence containing combining characters in to the single character, and others might not.
In UTF-8 you cannot represent à as a. I can have both in a file name and they are different. An application that assumes a is the same a à (in UTF-8) is broken and should be fixed.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 10 lines
Quoted 4 lines
I'm not sure whether it's valid UTF-8 or not, but it's certainly possible to code, for example, an 'A', (decimal 65), via an escape to a 31-bit character representation. Presumably the majority of UTF-8 parsers would decode the sequence as 65, rather than emit an error.
There are many ways of getting things wrong. The algorithm for encoding UTF-8 doesn't give you the option of encoding 65 as two bytes; any UCS-4 character with code 0-0x7F must result in a onand the same principle goes for every other character and the unicdeo standard forbids the use of anything but the shortest possible sequence.
The recommended encoding algorithm forbids anything but the shortest sequence, yes, but what will the majority of decoders do? I suspect that at least some will follow the usual networking rule of be liberal in what you accept, which for filenames may well cause all sorts of security holes.
Quoted 9 lines
Quoted 5 lines
Also, even ignoring that, how do you handle things like accented characters which can be represented as single characters, or as sequences containing combining characters? Some applications might convert the sequence containing combining characters in to the single character, and others might not.
In UTF-8 you cannot represent � as
a. I can have both in a file name and they are different. An application that assumesa is the same a � (in UTF-8) is broken and should be fixed.
Well, as long as every userspace implementation gets it correct, we'll be OK. Personally, I doubt they all will, especially those that convert from legacy encodings to Unicode, although quite possibly the above scenario with combining characters is not likely to happen for filenames. Or is it? What about copying a file from a filesystem with a UTF-8 encoding to a filesystem with a legacy encoding, and then back again?
However, I am less concerned about this second scenario than the first.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 7 lines · you
Quoted 5 lines
There are many ways of getting things wrong. The algorithm for encoding UTF-8 doesn't give you the option of encoding 65 as two bytes; any UCS-4 character with code 0-0x7F must result in a onand the same principle goes for every other character and the unicdeo standard forbids the use of anything but the shortest possible sequence.
The recommended encoding algorithm forbids anything but the shortest
That algorithm is the /definition/ of UTF-8, not just an example. Sure you can actually do it another way, but the result is uniquely defined (or else it's not UTF-8).
Quoted 7 lines
Well, as long as every userspace implementation gets it correct, we'll be OK. Personally, I doubt they all will, especially those that convert from legacy encodings to Unicode, although quite possibly the above scenario with combining characters is not likely to happen for filenames. Or is it? What about copying a file from a filesystem with a UTF-8 encoding to a filesystem with a legacy encoding, and then back again?
Sounds like you think we want to invent a new problem. The problem is here and it's real (not in the U.S, but the the rest of the world). There are Network file systems (samba in particular), partitions belonging to other OS's (ntfs, fat or even other Linux installation on the same machine), removable devices etc etc.
Microsoft introduced a kludge for managing long file names in a short filename context. Since Linux doesn't have the length limit a nicer kludge could be used to represent unicode as non-unicode in userspace like a Uxxxxx. When there is a mismatch there has to be kludge, but it's still many times better than a bunch of character that look like garbage (and cause legacy application so choke).
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quote from Robin Rosenberg <[email protected]>:
Quoted 10 lines · you
Quoted 7 lines
Quoted 5 lines
There are many ways of getting things wrong. The algorithm for encoding UTF-8 doesn't give you the option of encoding 65 as two bytes; any UCS-4 character with code 0-0x7F must result in a onand the same principle goes for every other character and the unicdeo standard forbids the use of anything but the shortest possible sequence.
The recommended encoding algorithm forbids anything but the shortest
That algorithm is the /definition/ of UTF-8, not just an example. Sure you can actually do it another way, but the result is uniquely defined (or else it's not UTF-8).
I know what you're saying, there is only one way to encode the data correctly. I totally agree with that.
However, we both know that UTF-8 provides escapes from the 7-bit encoding, and although it goes against the standard to encode 7-bit characters using such sequences, in the real world don't you think that there will be a lot of decoders which decode the multi-byte sequence back, rather than report an error? This is not something that will be happening in the kernel - it will be up to userspace to do it, so there may well be many different implementations.
Imagine you have two files, with the following filename bytes:
11000001 10000001 00000000
01000001 00000000
..and a _real world_ application, which is not necessarily completely UTF-8 conformant, tries to open the file with filename 'A'. Which one is it going to open?
Quoted 9 lines
Quoted 7 lines
Well, as long as every userspace implementation gets it correct, we'll be OK. Personally, I doubt they all will, especially those that convert from legacy encodings to Unicode, although quite possibly the above scenario with combining characters is not likely to happen for filenames. Or is it? What about copying a file from a filesystem with a UTF-8 encoding to a filesystem with a legacy encoding, and then back again?
Sounds like you think we want to invent a new problem.
I am aware that similar problems already exist. However, most legacy encodings don't suffer from the first issue we discussed above, where multiple byte sequences could be decoded to the same character codes. I don't think that the issue with combining characters is likely to be an issue, I only mentioned it as an example. As you pointed out a single accented character, and a two character combination are distinct, and converting the combination to the corresponding single character in a filename would definitely be wrong, in my opinion. However, that doesn't mean that software won't do it.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 10 lines · you
I know what you're saying, there is only one way to encode the data correctly. I totally agree with that.
However, we both know that UTF-8 provides escapes from the 7-bit encoding, and although it goes against the standard to encode 7-bit characters using such sequences, in the real world don't you think that there will be a lot of decoders which decode the multi-byte sequence back, rather than report an error? This is not something that will be happening in the kernel - it will be up to userspace to do it, so there may well be many different implementations.
Oh, I wasn't thinking of fixing *every* application out there, but making the kernel api's convert between the user locale and the file system locale, thus restricting the problems to places that can be fixed.
An alternative would be glibc since it's used by most apps, but then there could be funny and inefficient interactions with filesystems that already do the job. The "future" common case would be utf-utf conversion for all native file systems, i.e. no work.
[... ]
Quoted 6 lines
I don't think that the issue with combining characters is likely to be an issue, I only mentioned it as an example. As you pointed out a single accented character, and a two character combination are distinct, and converting the combination to the corresponding single character in a filename would definitely be wrong, in my opinion. However, that doesn't mean that software won't do it.
Some applications break if I put any non-ascii characters, but they few enough that I can afford the loss. Most shell scripts break if I even have a space in a filename. This shouldn't be any worse than that. The space issue is really serious (but I don't think that can be fixed other than teaching people to program properly, and possibly improving bash's knowledge of the difference between a space and argument separator).
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 4 lines · Robin Rosenberg
a space in a filename. This shouldn't be any worse than that. The space issue is really serious (but I don't think that can be fixed other than teaching people to program properly, and possibly improving bash's knowledge of the difference between a space and argument separator).
Other than allocating a key and bytecode for non-breaking-white-space as a separator (Hmm.. allocate 'left-windows' purely for ironic value? ;), how do you propose to actually improve it's knowledge of the distinction? The basic problem is that we're overloading x'20' as both space and separator, and then end up disambiguating based on context and syntax. And quite frankly, I don't see much hope for improving things as long as x'20' is overloaded.
Could go the VMS command/this/that/the/other/thing route, I guess? :)
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Robin RosenbergQuoted 5 lines · Robin Rosenberg
Most shell scripts break if I even have a space in a filename. This shouldn't be any worse than that. The space issue is really serious (but I don't think that can be fixed other than teaching people to program properly, and possibly improving bash's knowledge of the difference between a space and argument separator).
Space works fine for me. Completion, wildcard expansion, variable substition etc. all fine. Bash doesn't need changing - your scripts do.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 9 lines · Jamie Lokier
Quoted 5 lines · Robin Rosenberg
Most shell scripts break if I even have a space in a filename. This shouldn't be any worse than that. The space issue is really serious (but I don't think that can be fixed other than teaching people to program properly, and possibly improving bash's knowledge of the difference between a space and argument separator).
Space works fine for me. Completion, wildcard expansion, variable substition etc. all fine. Bash doesn't need changing - your scripts do.
I'm thinking about many scripts in the wild, and my own scripts (usually) handle spaces well, but it's awkward sometimes although quoting usually resolves the issue (never mind what happens with filenames with quotes, newlines and other garabage, but even those work sometimes. Fortunately these are rare, very rare and usually the result of a programming mistake elsewhere :-)
On the command line there is no problem.
With other script languages I use this is rarely an issue.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to John BradfordQuoted 3 lines · John Bradford
in the real world don't you think that there will be a lot of decoders which decode the multi-byte sequence back, rather than report an error?
There will be decoders which convert ASCII "a" to "A" too. We can't fix broken code; at least we can make it clear to anyone writing a decoder what is acceptable, and that being "liberal" in what's decoded is not acceptable and considered a security flaw.
An app author only writes the UTF-8 decoder once; it isn't at all hard to convert non-minimal forms to the replacement char U+FFFD. (Although that could be a security hole in some cases, it's much better than allowing non-zero characters to decoder to NUL or "/" or "."). Rejecting a non-minimal form is often hard, because the UTF-8 decoder is often used in a place which cannot flag errors.
Quoted 8 lines
Imagine you have two files, with the following filename bytes:
11000001 10000001 00000000
01000001 00000000..and a
_realworld_application, which is not necessarily completely UTF-8 conformant, tries to open the file with filename 'A'. Which one is it going to open?
The one which "ls" and other programs show as "A".
The other one will typically show as "?" or a diamond or something.
Quoted 6 lines
I don't think that the issue with combining characters is likely to be an issue, I only mentioned it as an example. As you pointed out a single accented character, and a two character combination are distinct, and converting the combination to the corresponding single character in a filename would definitely be wrong, in my opinion. However, that doesn't mean that software won't do it.
Indeed some software will do it, and worse than that: they may look the same in an editor or file selector. (See recent problems with misleading URLs for why that sort of thing can be a security hole).
The combining char problem is similar to case folding: some filesystems and programs treat "a" and "A" as equivalent too. If the kernel had an encoding converter, and the filesystem stored iso-8859-1 while userspace was presented with utf-8, it is likely that several Unicode characters would be mapped to "a", causing similar problems to automatic case folding in filesystems.
In other words, there is no clear solution to this problem.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to John Bradford#include <hallo.h>
Quoted 7 lines · John Bradford
Well, as long as every userspace implementation gets it correct, we'll be OK. Personally, I doubt they all will, especially those that convert from legacy encodings to Unicode, although quite possibly the above scenario with combining characters is not likely to happen for filenames. Or is it? What about copying a file from a filesystem with a UTF-8 encoding to a filesystem with a legacy encoding, and then back again?
I always wondered why there is no "iocharset" option for unixoid filesystems. IMO there could be an easy migration path for existing installations to UTF-8:
- convert all filenames to UTF-8 (or any other Unicode encoding)
- mount the FS with "iocharset=UTF-8,charset=latin1" (for current Latin1 users). Users can continue to use their latin1 names while they are stored in Unicode on the disk (this is what currently happens with VFAT, a very nice solution IMHO)
- when enough applications are ready for multibyte encodings, remove the charset/iocharset workaround and make people use .UTF-8 locales
Though, the ultimate solution for the steps 2. and 3. would be the Microsoft-like way:
- convert the filenames in libc (from $locale to UTF-8), depending on which locale the user has set
This sounds like cheating but would allow to be most flexible and most compatible to encoding-ignoring applications.
Eduard.
Signature · 2 lines
Wir sind nichts; was wir suchen ist alles. -- Johann Christian Friedrich Hölderlin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to John BradfordQuoted 9 lines · John Bradford
Quoted 6 lines
Definitely a good reason. It seem many assume file names are a local thing, but this is not so. Now consider the case with an external firewire disk or memory stick created on a machine with iso-8859-1 as the system character set and e.g xfs as the file system. What happens when I hook it up to a new redhat installation that thinks file names are best stored as utf8? Most non-ascii file names aren't even legal in utf8.
Another thing to consider is that you can encode the same character in several ways using utf8,
No, you can't. Only the shortest encoding of a character is valid UTF-8, and any program which claims to comply with Unicode is _required_ to reject all other encodings, citing security as the main reason.
That means any code which transcodes UTF-8 to another encoding (such as iso-8859-1) must reject the non-minimal forms as invalid characters, in whatever way that is done.
If there's any transcoding code in Linux which doesn't do that, it's a potential security hole and should be fixed.
Quoted 2 lines
so two filenames could have different byte strings, but evaluate to the same set of unicode characters.
That's true in some other encodings I think (the iso-2022 ones), but not UTF-8.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Robin RosenbergRobin Rosenberg wrote:
Now consider the case with an external firewire
Quoted 4 lines
disk or memory stick created on a machine with iso-8859-1 as the system character set and e.g xfs as the file system. What happens when I hook it up to a new redhat installation that thinks file names are best stored as utf8? Most non-ascii file names aren't even legal in utf8.
It goes wrong. This happens both with filesystems that know nothing about encodings, e.g. ext3, and filesystems that need to be told what to transcode to/from utf-8, e.g. ntfs.
It is also a problem that some applications access the filesystem assuming utf-8 and some don't. Nothing in the filesystem can make the different applications cooperate regarding these. E.g. I have filenames that look fine in "ls" containg things like c-cedilla, but xmms displays them wrongly.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 10 lines · Jamie Lokier
Robin Rosenberg wrote:
Now consider the case with an external firewireQuoted 4 lines
disk or memory stick created on a machine with iso-8859-1 as the system character set and e.g xfs as the file system. What happens when I hook it up to a new redhat installation that thinks file names are best stored as utf8? Most non-ascii file names aren't even legal in utf8.
It goes wrong. This happens both with filesystems that know nothing about encodings, e.g. ext3, and filesystems that need to be told what to transcode to/from utf-8, e.g. ntfs.
Yes, so ext3&co. should be equipped with charset options just the other so it can be fixed by the user or in some cases the mount tools.
Is there a place to store character set information in these file systems?
Quoted 5 lines
It is also a problem that some applications access the filesystem assuming utf-8 and some don't. Nothing in the filesystem can make the different applications cooperate regarding these. E.g. I have filenames that look fine in "ls" containg things like c-cedilla, but xmms displays them wrongly.
Some apps simply don't think non-ascii is relevant. Xmms is one, although is doesn't crash at least. My guess was that it was a font problem since it looks like XMMS uses some special fonts. Even new apps (like gedit have character set problems. These apps have to be fixed since they don't work properly anywhere outside the US. But that is a pure userspace problem, not a kernel one.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 1 line · Robin Rosenberg
Is there a place to store character set information in these file systems?
Please don't confuse character set with character encoding. The problem we are talking about here is about character encoding.
Once upon a time the two were muddled; that's why MIME and HTTP use "charset" to mean character encoding.
And the answer is: yes, you can store it wherever you want :)
Quoted 3 lines
Some apps simply don't think non-ascii is relevant. Xmms is one, although is doesn't crash at least. My guess was that it was a font problem since it looks like XMMS uses some special fonts.
It's not a font problem. XMMS simply displays each byte as a separate character because that's what it assumes it should do. No font will fix that.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 7 lines · Jamie Lokier
Quoted 1 line · Robin Rosenberg
Is there a place to store character set information in these file systems?
Please don't confuse character set with character encoding. The problem we are talking about here is about character encoding. Once upon a time the two were muddled; that's why MIME and HTTP use "charset" to mean character encoding.
I shall try not to mix them in the future. The reason for the name in MIME is probably because a (mime)charset does specify a character set (+encoding), while the mime-encoding only specifies raw bytes.
Quoted 1 line
And the answer is: yes, you can store it wherever you want :)
I was thinking of the file system meta data so mount or the kernel or the fs could handle it.
Quoted 6 lines
Quoted 3 lines
Some apps simply don't think non-ascii is relevant. Xmms is one, although is doesn't crash at least. My guess was that it was a font problem since it looks like XMMS uses some special fonts.
It's not a font problem. XMMS simply displays each byte as a separate character because that's what it assumes it should do. No font will fix that.
I assumed a font problem because my machine is using ISO-8859-1 and XMMS doesn't display tose non-ascii characters I use; of course it could be both.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Robin RosenbergQuoted 4 lines · Robin Rosenberg
Yes, so ext3&co. should be equipped with charset options just the other so it can be fixed by the user or in some cases the mount tools.
Is there a place to store character set information in these file systems?
Bullshit. Just as there is no timezone common for all users, there is no charset common for all of them. Charset of _machine_ doesn't make any sense at all - toy operating systems nonwithstanding.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 9 lines · [email protected] wrote:
Quoted 4 lines · Robin Rosenberg
Yes, so ext3&co. should be equipped with charset options just the other so it can be fixed by the user or in some cases the mount tools.
Is there a place to store character set information in these file systems?
Bullshit. Just as there is no timezone common for all users, there is no charset common for all of them. Charset of _machine_ doesn't make any sense at all - toy operating systems nonwithstanding.
Charset of a filename does make sense, though. That's not per user, it's per filename.
A name which one user entered as "£10.txt" should ideally display as that sequence of characters to all users who want to display the name.
I already have this problem on my filesystems: some programs show the names assuming UTF-8, other programs show them assuming iso-8859-1.
But it's worse than that. On my filesystem, names are stored in UTF-8 as is recommended these days. "ls" on some terminals shows the names as I wrote them. But on other terminals it shows the wrong names.
If I create a file using a shell command, what I get depends on which terminal I used to create it. If I am using a terminal which displays UTF-8 but ssh to another machine, the other machine assumes the terminal is displaying iso-8859-1 even though the other machine's default locale is UTF-8. And so on.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
#include <hallo.h>
Quoted 5 lines · Jamie Lokier
If I create a file using a shell command, what I get depends on which terminal I used to create it. If I am using a terminal which displays UTF-8 but ssh to another machine, the other machine assumes the terminal is displaying iso-8859-1 even though the other machine's default locale is UTF-8. And so on.
Then you have something wrong in the shell configuration of the remote machine. I do not see any problems in having a ssh shell opened from a UTF-8 terminal to a machine where the shell environment is also configured to use UTF-8 environment.
The only problem that may appear if you deliberatedly configured the user environment on the other side for latin1, then you would have to fix it in some way. Eg. configuring LANG depending on SSH* variables in .bashrc.
Regards,
Eduard.
Signature · 3 lines
Das Merkmal eines kleinen Menschen ist, daß er hochmütig wird, wenn er merkt, daß man ihn braucht. -- Friedl Beutelrock
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 10 lines · Eduard Bloch
Quoted 5 lines
If I create a file using a shell command, what I get depends on which terminal I used to create it. If I am using a terminal which displays UTF-8 but ssh to another machine, the other machine assumes the terminal is displaying iso-8859-1 even though the other machine's default locale is UTF-8. And so on.
Then you have something wrong in the shell configuration of the remote machine. I do not see any problems in having a ssh shell opened from a UTF-8 terminal to a machine where the shell environment is also configured to use UTF-8 environment.
Of course that's fine. What goes wrong is when you connect to that same machine from another terminal which is not UTF-8.
There are in fact two different problems, and you have ignored them both :)
Firstly, "ls", editors, filenames:
The shell configuration is irrelevant. If I create a file name like "£100.txt" (that's POUND followed by "100.txt") when I'm connected from a UTF-8 terminal, it creates a filename encoded in UTF-8 and displays it fine.
If I then log in to the same machine from another terminal which displays latin1, then "ls" will _not_ display the name correctly _regardless_ of shell or locale configuration.
If I then create a file called "£100.txt" (same name) using the terminal which displays latin1, it creates a filename encoded in latin1.
When I log in using the UTF-8 terminal, "ls" won't display the second name as it was entered. Neither will GNOME or KDE.
Unfortunately, to be compatible with shell utilities, programs like Mutt and Emacs which _are_ aware of the display and input encodings will use the current terminal's encoding when accessing the filesystem. So even those programs create file names with the wrong encoding, if you log in from the wrong kind of terminal.
When I open a file in Emacs, and the file contains UTF-8, that displays just fine on either kind of terminal (provided the terminal can display the characters). But Emacs, and many other programs, will display the wrong file _names_ when logged in from the wrong kind of terminal.
Secondly, message locale and the shell:
There is no mechanism for SSH to convey which character encoding the remote machine must use for displaying and inputting text, yet client terminals come in different flavours. That is the problem.
(On my laptop, for example, which is a standard RH9, Gnome terminal windows are UTF-8 but console is latin1). These are both fine locally. There is no configuration on a remote machine which is right for both of them, though.)
I think this is because the character encoding used by the terminal should be in the TERM environment variable, but it is in LANG instead.
Quoted 4 lines
The only problem that may appear if you deliberatedly configured the user environment on the other side for latin1, then you would have to fix it in some way. Eg. configuring LANG depending on SSH* variables in .bashrc.
No. If I have a plain shell with no configuration at all, then both charset-aware programs like Mutt and Emacs, and non-charset-aware code like filename display from "ls" do _not_ automatically display filenames properly on both kinds of client terminal.
In the former case it is because SSH does not automatically convey the appropriate setting for LANG, which (rather dubiously) includes whether to use UTF-8 for display.
In the latter case, "ls" and such, there is nothing SSH can do.
(And that's what makes this relevant to linux-kernel - "ls" has no way to display names correctly on both terminal types precisely because it does not have any information about the character encoding of the filenames returned by readdir()).
The result of all this is that everything works fine as long as you only log in from the kind of terminal which matches the remote machine.
Unfortunately, while the modern GUIs all use UTF-8 (this is a good thing in the long run), both the default Linux console, and most non-Linux terminals, do not use UTF-8.
Therefore file names are generally created and displayed in UTF-8 when using any of the modern GUIs, including GUI terminals, but file names are generally created and displayed in a locale-specific encoding (usually iso-8859-1) when using any console, external terminal, or ssh from an older client.
Btw, as a practical matter, it took me about a year before I figured out how to enter a "£" (POUND) symbol into a message being edited with Mutt and Emacs on a remote server. Until I learned to explicitly set "LANG=en_GB.utf8" on the remote server when I logged in from GNOME Terminal (it was a RH9 box which by default set LANG=en_GB, which is _correct_ for most clients), typing "£" just didn't enter anything.
Third problem (a straightforward Linux bug):
I just did unicode_start on the console, which turns on UTF-8 for that virtual terminal - for display and for keyboard input.
Then I did unicode_stop. Guess what: it put the display back in iso-8859-1 for that virtual terminal, but the keyboard remained stuck in UTF-8 for _all_ virtual terminals. Once in that state, I had difficulty typing the pound sign which appears earlier in this message, and in fact I don't know how to restore the console without rebooting the client machine. "reset" doesn't work; using a different virtual terminal doesn't work.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
#include <hallo.h>
Quoted 8 lines · Jamie Lokier
Quoted 4 lines
Then you have something wrong in the shell configuration of the remote machine. I do not see any problems in having a ssh shell opened from a UTF-8 terminal to a machine where the shell environment is also configured to use UTF-8 environment.
Of course that's fine. What goes wrong is when you connect to that same machine from another terminal which is not UTF-8.
Quoted 6 lines
There are in fact two different problems, and you have ignored them both :)
Firstly, "ls", editors, filenames:
The shell configuration is irrelevant. If I create a file name like "£100.txt" (that's POUND followed by "100.txt") when I'm
Sure, sure, I can read it since I use UTF-8 too.
Quoted 6 lines
connected from a UTF-8 terminal, it creates a filename encoded in UTF-8 and displays it fine.
If I then log in to the same machine from another terminal which displays latin1, then "ls" will _not_ display the name correctly _regardless_ of shell or locale configuration.
I know what you mean and that is why I already proposed a radical solution. Let me repeat it:
- convert all files from the previous charset to UTF-8 overnight if the previous charset was unknown, first make sure that you can guess it for all users and contact users that have files with suspicous filenames (eg. not convertable from Latin1). Or look trough their shell/X config files (*)
- in libc, implement a recoding function to convert file names from LC_CTYPE to the underlying UTF-8 encoding
Done.
(*) There is no other way. Linux developers ignored the diversity of charset/encodings over many years and now the needed information is lost (not stored anywhere in the filesystem)
Quoted 3 lines
If I then create a file called "£100.txt" (same name) using the terminal which displays latin1, it creates a filename encoded in latin1.
Of course. That is what the conversion shoudl be done in Userspace (libc). The kernel itself does not know about used locale.
Quoted 3 lines
Unfortunately, to be compatible with shell utilities, programs like Mutt and Emacs which _are_ aware of the display and input encodings will use the current terminal's encoding when accessing the
That is the correct way, though.
Quoted 2 lines
filesystem. So even those programs create file names with the wrong encoding, if you log in from the wrong kind of terminal.
It is the _right_ enconding in the moment when they create it.
Quoted 10 lines
Secondly, message locale and the shell:
There is no mechanism for SSH to convey which character encoding the remote machine must use for displaying and inputting text, yet client terminals come in different flavours. That is the problem.
(On my laptop, for example, which is a standard RH9, Gnome terminal windows are UTF-8 but console is latin1). These are both fine locally. There is no configuration on a remote machine which is right for both of them, though.)
Yup, I know that problem. At least to display them correctly, you can either run unicode_start (to enable console's own conversion) which sucks when they are chars from completely different language groups, eg. latin and cyrillic. I used dynafont for a while which worked well for displaying characters.
Quoted 2 lines
I think this is because the character encoding used by the terminal should be in the TERM environment variable, but it is in LANG instead.
No. TERM does not have anything to do with locales (LANG).
Regards,
Eduard.
Signature · 2 lines
Selbstlosigkeit ist ausgereifter Egoismus. -- Herbert Spencer
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 4 lines · Eduard Bloch
Quoted 2 lines
I think this is because the character encoding used by the terminal should be in the TERM environment variable, but it is in LANG instead.
No. TERM does not have anything to do with locales (LANG).
No. The locale should not have anything to do with the appropriate byte sequences need to make the terminal display characters.
It is wrong that LANG must have a different value depending on whether I log in using a DEC VT100 or a Gnome Terminal, even though I wish to see exactly the same language, dialect, messages, number formats, currency formats, dates and times.
It is acceptable that LANG may control the encoding stored in files and filenames, but this should be independent of the terminal type.
It is especially wrong that libraries which should be locale-independent - such as curses, slang and readline - must read the LANG variable in addition to TERM. If curses does not read and parse LANG, simple things like the box around a dialog will not line up correctly. This is wrong - it is a terminal characteristic.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
#include <hallo.h>
Quoted 7 lines · Jamie Lokier
Quoted 4 lines
Quoted 2 lines
I think this is because the character encoding used by the terminal should be in the TERM environment variable, but it is in LANG instead.
No. TERM does not have anything to do with locales (LANG).
No. The locale should not have anything to do with the appropriate byte sequences need to make the terminal display characters.
Heh. It would be very nice if we had the situation that you describe, but that is actually not the case. TERM specifies the general capabilities of the terminal. It does _not_ tell the application inside which FONT encoding is used, nor whether it is compatible with multibyte input.
Quoted 4 lines
It is wrong that LANG must have a different value depending on whether I log in using a DEC VT100 or a Gnome Terminal, even though I wish to see exactly the same language, dialect, messages, number formats, currency formats, dates and times.
Nonsense, sorry. How should your application know how to encode its output? How should it know which font is used. I have heard about some magic strings that application can send to the Xterm (when TERM=xterm) to tell it to change the font encoding (similar to the string used to set the window Title used by mc, for example). But this is an extension, not mandatory for a general implentation of a "terminal".
Quoted 2 lines
It is acceptable that LANG may control the encoding stored in files and filenames, but this should be independent of the terminal type.
And what controls the font setting? (see above)
Quoted 3 lines
It is especially wrong that libraries which should be locale-independent - such as curses, slang and readline - must read the LANG variable in addition to TERM. If curses does not read and
See above. Especially since different chars are used to draw graphical characters (lines, boxes, ...), they _must_ know which font encoding they have to expect.
Regards,
Eduard.
Signature · 3 lines
Zufälle sind die Mittel des Schicksals, durch die es seine wichtigsten Pläne mit uns durchführt. -- Charles Tschopp
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 3 lines · Eduard Bloch
TERM specifies the general capabilities of the terminal. It does _not_ tell the application inside which FONT encoding is used, nor whether it is compatible with multibyte input.
It should - especially the multibyte encoding.
The font is irrelevant; our trouble here is *character encoding* which has nothing to do with fonts. Please don't use the incorrect term as there is widespread confusion over it already.
That isn't just about which glyph is displayed in response to each byte. UTF-8 affects terminal escape sequence parsing, and also the relationship between number of non-control bytes transmitted and the distance moved by the cursor.
If I write a UTF-8 string to a VT220-like terminal (such as xterm approximates), some text characters are interpreted as terminal commands. (Hint: 0x9b (which can occur in UTF-8 text) is equivalent to 0x1c 0x5b, the control sequence introducer; there are others too).
When you edit a line with the unix terminal line editor, when you type DEL, it writes BACKSPACE-SPACE-BACKSPACE and removes one byte from the input. That utterly fails to do the right thing on UTF-8 terminals. For example, run the command "cat" by itself, then type "£££", then hit DEL twice - it will show one pound sterling sign. Press enter, and cat will echo the line containing _two_ pound sterling signs.
No setting of LANG or TERM makes that behave correctly.
So, do you think the kernel's line editor should be locale-aware too? :)
Quoted 4 lines
Quoted 4 lines
It is wrong that LANG must have a different value depending on whether I log in using a DEC VT100 or a Gnome Terminal, even though I wish to see exactly the same language, dialect, messages, number formats, currency formats, dates and times.
NB: It's wrong because LANG should be for terminal-independent locale properties, such as which languages I want to use and how I want text files stored.
If I log into a remote machine, I want characters displayed according to the local terminal's requirements, but I want text files and filenames to use the remote machine's locale, naturally.
Quoted 2 lines
Nonsense, sorry. How should your application know how to encode its output?
Increasingly I'm thinking UTF-8-ness should be a terminal capability, like ocrnl. The kernel's own line editor needs to know this property anyway, and it would really help with moving filenames and everything else over to UTF-8 - with no change to the simple unix programs such as the shell utilities.
Quoted 7 lines
Quoted 3 lines
It is especially wrong that libraries which should be locale-independent - such as curses, slang and readline - must read the LANG variable in addition to TERM.
See above. Especially since different chars are used to draw graphical characters (lines, boxes, ...), they _must_ know which font encoding they have to expect.
See "acsc" in the terminfo(5) database. Line & box drawing characters have been treated as a terminal capability for a long time. Case made :)
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Eduard BlochQuoted 5 lines · Eduard Bloch
- convert all files from the previous charset to UTF-8 overnight if the previous charset was unknown, first make sure that you can guess it for all users and contact users that have files with suspicous filenames (eg. not convertable from Latin1). Or look trough their shell/X config files (*)
Hazardous.
Quoted 2 lines
- in libc, implement a recoding function to convert file names from
LC_CTYPEto the underlying UTF-8 encoding
Hmm.. could be fun if somebody is calling 'open', and the UTF-8 encoding requires the insertion of extra characters to encode it - what do you do then? That looks like a security hole just waiting to happen. Probably has lots of lurking corner cases too - what if you creat() a file, then do a readdir() and strcmp() each entry looking for your file (while comparing a filename smashed to UTF-8 to the original unsmashed string)?
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 11 lines · [email protected] wrote:
Quoted 2 lines
- in libc, implement a recoding function to convert file names from
LC_CTYPEto the underlying UTF-8 encodingHmm.. could be fun if somebody is calling 'open', and the UTF-8 encoding requires the insertion of extra characters to encode it - what do you do then?
That looks like a security hole just waiting to happen. Probably has lots of lurking corner cases too - what if you
creat()a file, then do areaddir()andstrcmp()each entry looking for your file (while comparing a filename smashed to UTF-8 to the original unsmashed string)?
Actually, following Eduard's proposal, that would work fine. The file name would be passed to libc in the current encoding, created in UTF-8, libc's readdir() would convert it back (which is always possible without mangling), and strcmp() would be fine.
The real problem comes when you readdir() a directory which contains non-UTF-8 names. Even if you changes your local filesystem, when you go travelling an remotely-mounted filesystem elsewhere may have them. What does Eduard's libc do then? Ignore the names? Mangle them?
Not to mention the extremely unpleasant performance implications.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
#include <hallo.h>
Quoted 11 lines · Jamie Lokier
Quoted 5 lines
That looks like a security hole just waiting to happen. Probably has lots of lurking corner cases too - what if you
creat()a file, then do areaddir()andstrcmp()each entry looking for your file (while comparing a filename smashed to UTF-8 to the original unsmashed string)?Actually, following Eduard's proposal, that would work fine. The file name would be passed to libc in the current encoding, created in UTF-8, libc's
readdir()would convert it back (which is always possible without mangling), andstrcmp()would be fine.
Quoted 4 lines
The real problem comes when you
readdir()a directory which contains non-UTF-8 names. Even if you changes your local filesystem, when you go travelling an remotely-mounted filesystem elsewhere may have them. What does Eduard's libc do then? Ignore the names? Mangle them?
Just pass the uncoverted strings then. Please note that this is exactly what happens today - every application running in UTF-8 locale and facing incompatible filenames has to deal with this problem. I wonder why so many people pretend that the current situation is "less or more okay".
Quoted 1 line
Not to mention the extremely unpleasant performance implications.
You always loose a bit performance when dealing with Unicode. Just accept it.
Regards,
Eduard.
Signature · 2 lines
Lang ist der Weg durch Lehren, kurz und wirksam durch Beispiele. -- Lucius Annaeus Seneca (4-65 n.Chr.)
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to [email protected]Quoted 12 lines
Quoted 4 lines
- convert all files from the previous charset to UTF-8 overnight if the previous charset was unknown, first make sure that you can guess it for all users and contact users that have files with suspicous filenames (eg. not convertable from Latin1). Or look troug=
h
Quoted 1 line
their shell/X config files (*)
Hazardous.
Quoted 2 lines
- in libc, implement a recoding function to convert file names from
LC_CTYPEto the underlying UTF-8 encoding
Quoted 10 lines
Hmm.. could be fun if somebody is calling 'open', and the UTF-8 encoding requires the insertion of extra characters to encode it - what do you do = then?
That looks like a security hole just waiting to happen. Probably has lot= s of
lurking corner cases too - what if youcreat()a file, then do a readdir(= ) andstrcmp()each entry looking for your file (while comparing a filename sma= shed
to UTF-8 to the original unsmashed string)?
The current situation is that so many applications simply treat filenames as arbitrary sequences of bytes. With many encodings, this simply happens to work, and an encoding mis-match will result in some incorrect characters being displayed for byte values > 127. However, some encodings, such as UTF-8, are simply _not_ compatible with the 'you can also treat it like an arbitrary byte string model', and there is a very real potential for security holes in bad implementations if we go down the "it's an arbitrary byte string, but you _should_ store UTF-8 there" route.
Maybe we should forget filename encoding altogether, and start thinking of filenames as arbitrary sequences of _32-bit words_. Existing applications can store their arbitrary byte sequences in the low byte, and new calls can be added to provide Unicode-aware userspace applications with access to the 32-bit space, which _must_ be used for UCS-4.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 6 lines · John Bradford
The current situation is that so many applications simply treat filenames as arbitrary sequences of bytes. With many encodings, this simply happens to work, and an encoding mis-match will result in some incorrect characters being displayed for byte values > 127. However, some encodings, such as UTF-8, are simply _not_ compatible with the 'you can also treat it like an arbitrary byte string model', and there
Excuse me? Would you fscking mind explaining what, in your opinion, UTF-8 is and what makes "simply _not_ compatible" with aforementioned model?
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quote from [email protected]:
Quoted 10 lines · John Bradford
Quoted 6 lines
The current situation is that so many applications simply treat filenames as arbitrary sequences of bytes. With many encodings, this simply happens to work, and an encoding mis-match will result in some incorrect characters being displayed for byte values > 127. However, some encodings, such as UTF-8, are simply _not_ compatible with the 'you can also treat it like an arbitrary byte string model', and there
Excuse me? Would you fscking mind explaining what, in your opinion, UTF-8 is
Read the UTF-8 manual page.
Quoted 2 lines
and what makes "simply _not_ compatible" with aforementioned model?
Byte values > 127 in UTF-8 don't map to single characters, but instead many of them form part of an escape to a larger set of values.
The net effect is that if you have filenames in an existing 8-bit encoding, such as any of the ISO-8859- encodings, and treat them as being in another, similar encoding, you may get some incorrect characters. This is not ideal, of course, but it is not very confusing for end users. You can more or less store arbitrary bytes in the filename, and usually at least get a displayable, re-typeable, somewhat usable result out.
Note that _many_ current applications expect to be able to do just that.
However, with UTF-8, random bytes > 127 may not map to any valid character sequence at all, or may map to a sequence that is not permitted by the spec, but which is, for example, a 31-bit representation of a value < 128. These are a potential source of security vulnerabilities for badly written decoders.
Now, this problem is not limited to UTF-8 - many 16 bit encodings may have similar issues with 'random' byte streams.
However, with my proposed solution, Unicode-aware applications can be adapted to write their filenames as UCS-4, and existing applications which continue to see 8-bit byte streams which they can interpret as they like, will see 7-bit ascii for characters which can be represented in it, and a random character for those which can't. Assuming that those applications treat the byte sequence as an ISO-8859- type character set, (not UTF-8, or a 16-bit character set), this shouldn't be too much of a problem, except where the low byte of the UCS-4 character is \0 or /. We can work around this by replacing such bytes with another character in the 8-bit read routine, (which isn't expected to deal with anything other than 7-bit ASCII 100% correctly, (which is no worse than what we have at the moment, as far as I can see)).
Applications which do treat the 8-bit byte stream as UTF-8 or an existing 16-bit encoding should have only one additional thing to deal with over what they have to deal with today, and that is the potential for a filename created from truncated 32-bit UCS-4 values to contain \0 or /. I suggested above that the kernel could deal with that by substituting another value, but obviously UTF-8 and 16-bit encodings are more sensitive to what that substitute value is, than ISO-8859- type encodings are.
John.
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to John BradfordQuoted 6 lines · John Bradford
Maybe we should forget filename encoding altogether, and start thinking of filenames as arbitrary sequences of _32-bit
words_. Existing applications can store their arbitrary byte sequences in the low byte, and new calls can be added to provide Unicode-aware userspace applications with access to the 32-bit space, which _must_ be used for UCS-4.
You forgot a :-). Right :-/
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Eduard BlochQuoted 5 lines · Eduard Bloch
Yup, I know that problem. At least to display them correctly, you can either run
unicode_start(to enable console's own conversion) which sucks when they are chars from completely different language groups, eg. latin and cyrillic. I used dynafont for a while which worked well for displaying characters.
Sorry, unicode_start doesn't work on most terminals (e.g. the VT100 downstairs or the Putty in the internet cafe), and it's also very antisocial to do when I log in from someone else's Linux console.
-- Jamie
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Eduard BlochQuoted 8 lines · Eduard Bloch
I know what you mean and that is why I already proposed a radical solution. Let me repeat it:
- convert all files from the previous charset to UTF-8 overnight if the previous charset was unknown, first make sure that you can guess it for all users and contact users that have files with suspicous filenames (eg. not convertable from Latin1). Or look trough their shell/X config files (*)
Thankfully isolatin-1 (and all other encodings in use AFAIK) can be converted to UTF-8. IsoLatin1 is also extremly simpe to convert-
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to [email protected]Quoted 9 lines · you
Quoted 4 lines · Robin Rosenberg
Yes, so ext3&co. should be equipped with charset options just the other so it can be fixed by the user or in some cases the mount tools.
Is there a place to store character set information in these file systems?
Bullshit. Just as there is no timezone common for all users, there is no charset common for all of them. Charset of _machine_ doesn't make any sense at all - toy operating systems nonwithstanding.
For us using toy languages, we see characters in filenames, not byte sequences, and if whenever possible users should see the same name regardless of locale.
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 8 lines · Robin Rosenberg
Quoted 7 lines · you
Quoted 5 lines · Robin Rosenberg
Yes, so ext3&co. should be equipped with charset options just the other so it can be fixed by the user or in some cases the mount tools.
Is there a place to store character set information in these file systems?
Quoted 7 lines
Quoted 3 lines
Bullshit. Just as there is no timezone common for all users, there is no charset common for all of them. Charset of _machine_ doesn't make any sense at all - toy operating systems nonwithstanding.
For us using toy languages, we see characters in filenames, not byte sequences, and if whenever possible users should see the same name regardless of locale.
Al says that there can be a hundred of users on the box _simultaneously_, each with different locale. fs should store filenames
in locale-agnostic way.
Signature · 1 line
vda
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 3 lines · vda
Al says that there can be a hundred of users on the box _simultaneously_, each with different locale. fs should store filenames
in locale-agnostic way.
I assume we agree then :-)
-- robin
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Andy IsaacsonQuoted 3 lines · Andy Isaacson
Why on earth is JFS worried about the filename, anyways? Why has it *ever* had *any* behavior other than "string of bytes, delimited with /, terminated with \0" ?
The problem that was addressed in OS/2 was that one user using locale A would create some files using non-ascii characters. Then a user using locale B would access these files, but the characters in those names did not make sense in his locale. Storing the file names in unicode allowed the characters to always translate to the correct characters in the user's locale, when the charset allowed it. I'm not familiar enough with the European locales to give specific examples. It was never an issue in the U.S. :^)
The OS/2 kernel has locale information for each process, so this actually works very well there. I will admit that it was a mistake not to change the default behavior when we ported this to Linux.
Quoted 4 lines
I read your response about OS/2, and maybe I'm just slow, but I don't see what that has to do with anything.
Does JFS on AIX have the same buggy behavior?
I know that JFS1 did not. I'm not sure about JFS2, since it was ported from the same OS/2 code base.
Quoted 3 lines
What behavior was the code originally designed to implement, on OS/2? Why was that behavior chosen rather than "filenames are a string of bytes"?
I hope I explained that well enough above.
Quoted 4 lines
Feel free to point to a "Design of the OS/2 JFS interface" document if such exists and answers my question. :)
-andy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
In reply to Andy IsaacsonQuoted 1 line · Andy Isaacson
Does JFS on AIX have the same buggy behavior?
Nope, it's been tolerant of all 254 bit patterns except \0 and '/' since at least AIX 3.1.2 back in the early 90s. It doesn't even have a concept of "UTF-8 filename" - it considers that a userspace issue.
Now, over the last 15 years I've tripped over a number of *userspace* things that did really stupid things when handed non-ASCII filenames, but that's a different issue...
Re: JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)
Quoted 4 lines · On Thu, 2004-02-12 at 09:26, [email protected] wrote:
On Wed, 11 Feb 2004 18:45:32 CST, Andy Isaacson said:
Now, over the last 15 years I've tripped over a number of *userspace* things that did really stupid things when handed non-ASCII filenames, but that's a different issue...
That's the problem that OS/2 addressed. In OS/2 each application would see the correct charset for its locale, no matter what the locale of the application that created the file was. In Linux, the file system simply doesn't have the information needed to do this, so it was a mistake to try to imitate it.
Signature · 2 lines
David Kleikamp IBM Linux Technology Center