A Great Idea (tm) about reimplementing NLS.
Hello.
I have a Great Idea about improving NLS in the linux kernel and I want somebody with kernel experience consider if it's good or not, just not to waste time on writing code that will be rejected.
First of all, why do I think the current NLS implementation isn't good enough.
Let's look at a situation. I'm using utf-8 as my default system charset, and my friend Vasiliy Pupkin, who uses koi8-r, wants to plug his flash drive (ext3) into my computer. It should work, except all non-us-ascii filenames will be totally unreadable. The problem is even bigger if I have an other friend's hard drive with reiserfs and cp1251 encoded filenames on it. The problem is not only with Russian language for which we have at least 3 common encodings. Everyone who uses non-us-ascii letters can face the same problem, since there are at least 2 encodings for theyr language - utf-8 and an other one used before utf.
Some would suggest not to use non-ascii file names at all, some would say that I should temporary change my locale, some could even offer me a perl script they wrote when faced the same problem. All these solutions are inconvenient and conflict with fundamental VFS concepts.
Instead of adding NLS support to filesystems who don't have it yet, I think there should be a global NLS layer, to convert file names from any to any encoding, independent of file system and transparently to the user.
So what do you think? Is it all nonsense or maybe I should try to implement it?
Please CC me, I'm not subscribed.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi.
Quoted 22 lines
I have a Great Idea about improving NLS in the linux kernel and I want somebody with kernel experience consider if it's good or not, just not to waste time on writing code that will be rejected.
First of all, why do I think the current NLS implementation isn't good enough.
Let's look at a situation. I'm using utf-8 as my default system charset, and my friend Vasiliy Pupkin, who uses koi8-r, wants to plug his flash drive (ext3) into my computer. It should work, except all non-us-ascii filenames will be totally unreadable. The problem is even bigger if I have an other friend's hard drive with reiserfs and cp1251 encoded filenames on it. The problem is not only with Russian language for which we have at least 3 common encodings. Everyone who uses non-us-ascii letters can face the same problem, since there are at least 2 encodings for theyr language - utf-8 and an other one used before utf.
Some would suggest not to use non-ascii file names at all, some would say that I should temporary change my locale, some could even offer me a perl script they wrote when faced the same problem. All these solutions are inconvenient and conflict with fundamental VFS concepts.
Quoted 6 lines
Instead of adding NLS support to filesystems who don't have it yet, I think there should be a global NLS layer, to convert file names from any to any encoding, independent of file system and transparently to the user.
So what do you think? Is it all nonsense or maybe I should try to implement it?
What do you do when a charset doesn't contain a char that another one does?
Compare the two very similar charsets ISO-8859-1 and ISO-8859-15 and have the Euro-sign using ISO-8859-15. Then try to make that into something sane.
Not knocking you or anything, you just have to think about these pitfalls.
// Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (MingW32)
iD8DBQFCrWSeBrn2kJu9P78RAkZNAKCjRkxx4EnZT+C8wblPB/AH63xz2ACfS4m6 IrVy4TwcwWH2Wm1Va+SN0XI=
=SYdz
-----END PGP SIGNATURE-----
On Mon, 2005-06-13 at 14:38 +0400, Alexey Zaytsev wrote:
[ Filenames with another encoding ]
Quoted 4 lines
Some would suggest not to use non-ascii file names at all, some would say that I should temporary change my locale, some could even offer me a perl script they wrote when faced the same problem. All these solutions are inconvenient and conflict with fundamental VFS concepts.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In what way?
Basically you just rename the files. How can this conflict with "fundamental VFS concepts" (and with which).
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 4 lines · Alexey Zaytsev
Instead of adding NLS support to filesystems who don't have it yet, I think there should be a global NLS layer, to convert file names from any to any encoding, independent of file system and transparently to the user.
Thats essentially what we have. The core OS is UTF-8, the fat and one or two other legacy file systems support mapping old and/or inferior encodings into utf-8 (and some other stuff).
Alan
Quoted 10 lines · Bernd Petrovitsch
On Mon, 2005-06-13 at 14:38 +0400, Alexey Zaytsev wrote:
[ Filenames with another encoding ]Quoted 4 lines
Some would suggest not to use non-ascii file names at all, some would say that I should temporary change my locale, some could even offer me a perl script they wrote when faced the same problem. All these solutions are inconvenient and conflict with fundamental VFS concepts.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^In what way?
Basically you just rename the files. How can this conflict with "fundamental VFS concepts" (and with which).
I can't rename files on Pupkin's drive because he won't like it. ;) In the case with a flash drive I can copy all the files to my computer and rename them, but I can't do it with a bigger media like hard disk.
The main idea of VFS is that you can access your files in the same way on any supported file system. But actually you can't simple access different-encoded non-ascii files on a filesystem that has no NLS, like ext or reiser.
Quoted 1 line
Bernd
Quoted 7 lines · Alexey Zaytsev
Quoted 6 lines · Bernd Petrovitsch
On Mon, 2005-06-13 at 14:38 +0400, Alexey Zaytsev wrote:
[ Filenames with another encoding ]Quoted 4 lines
Some would suggest not to use non-ascii file names at all, some would say that I should temporary change my locale, some could even offer me a perl script they wrote when faced the same problem. All these solutions are inconvenient and conflict with fundamental VFS concepts.
Quoted 6 lines
Quoted 4 lines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^In what way?
Basically you just rename the files. How can this conflict with "fundamental VFS concepts" (and with which).I can't rename files on Pupkin's drive because he won't like it. ;)
.... which has IMHO nothing to do with the VFS (or concepts behind it).
Quoted 2 lines
In the case with a flash drive I can copy all the files to my computer and rename them, but I can't do it with a bigger media like hard disk.
You forgot CDs/DVDs and other inherently read-only media with such strange filenames.
Quoted 4 lines
The main idea of VFS is that you can access your files in the same way on any supported file system. But actually you can't simple access different-encoded non-ascii files on a filesystem that has no NLS, like ext or reiser.
I don't think that any filesystem knows about the encoding of every filename - after all it is up to the user which encoding he uses for a given file (and no, no one forces me to use the same encoding on the names of all of "mine" files).
IOW given a FAT filesystem on an USB stick, which codepage should be used?
Perhaps it makes sense to start a prototype with a FUSE (or similar) module. You could use standard libs to convert without messing around in the kernel (and I don't think someone wants to have an encoding conversion layer in the kernel).
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 9 lines · Alan Cox
Quoted 4 lines · Alexey Zaytsev
Instead of adding NLS support to filesystems who don't have it yet, I think there should be a global NLS layer, to convert file names from any to any encoding, independent of file system and transparently to the user.
Thats essentially what we have. The core OS is UTF-8, the fat and one or two other legacy file systems support mapping old and/or inferior encodings into utf-8 (and some other stuff).
Yes, that's how it works, but if I want ext or reiser or whatever to have NLS, I'll have to make them support it (btw, if I do so, wont it be rejected?). I want to move the NLS one level upper so the filesystem imlementations won't have to worry about it any more. I don't have much kernel experience, and none in the fs area, so I can't explain it any better, but hope you get the idea.
Quoted 1 line
Alan
Thank you answering.
Quoted 11 lines · Bernd Petrovitsch
Quoted 4 lines
The main idea of VFS is that you can access your files in the same way on any supported file system. But actually you can't simple access different-encoded non-ascii files on a filesystem that has no NLS, like ext or reiser.
I don't think that any filesystem knows about the encoding of every filename - after all it is up to the user which encoding he uses for a given file (and no, no one forces me to use the same encoding on the names of all of "mine" files).
IOW given a FAT filesystem on an USB stick, which codepage should be used?
Yes, most if not all filesystems don't have any information about file names encoding, but the user can often guess it. Hawing files with differently-encoded names on the same filesystem is nonsense, which could only appear because of the current NLS misfeatures.
Quoted 4 lines
Perhaps it makes sense to start a prototype with a FUSE (or similar) module. You could use standard libs to convert without messing around in the kernel (and I don't think someone wants to have an encoding conversion layer in the kernel).
We already have it in the kernel, it's called nls and it's up to the file system implementation to decide to use it or not. I don't contrive something new, I just want the existing system to work a bit different.
Hi.
A related issue I had is that some codepages don't have a nls module yet.
For example I once or twice needed to mount a vfat filesystem with cp1256 ( arabic windows ) charset. There is no such nls module. I couldn't find any documentation about how to create the module ( maybe I didn't look hard enough ).
Quoted 13 lines · Alexey Zaytsev
Quoted 11 lines · Bernd Petrovitsch
Quoted 4 lines
The main idea of VFS is that you can access your files in the same way on any supported file system. But actually you can't simple access different-encoded non-ascii files on a filesystem that has no NLS, like ext or reiser.
I don't think that any filesystem knows about the encoding of every filename - after all it is up to the user which encoding he uses for a given file (and no, no one forces me to use the same encoding on the names of all of "mine" files).
IOW given a FAT filesystem on an USB stick, which codepage should be used?
Quoted 4 lines
Yes, most if not all filesystems don't have any information about file names encoding, but the user can often guess it. Hawing files with differently-encoded names on the same filesystem is nonsense, which could only appear because of the current NLS misfeatures.
Different users of the same system may have perfectly valid reasons to use different locale settings, and thus different filename encodings. Forcing one thing or another is just a useless restriction, and probably not POSIX compliant.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 6 lines · Alexey Zaytsev
Yes, that's how it works, but if I want ext or reiser or whatever to have NLS, I'll have to make them support it (btw, if I do so, wont it be rejected?). I want to move the NLS one level upper so the filesystem imlementations won't have to worry about it any more. I don't have much kernel experience, and none in the fs area, so I can't explain it any better, but hope you get the idea.
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
There are some good technical reasons too
Encodings don't map 1:1 - two names may cease to be unique
Encodings vary in length - image a file name that is longer than the allowed maximum on your system with your encoding choice - that could occur with KOI8-R to UTF-8 I believe
That said it ought to be possible to use the stackable fs work (FUSE etc) to write a layer you can mount over any fs that does NLS translation.
Quoted 10 lines · Alan Cox
Quoted 6 lines · Alexey Zaytsev
Yes, that's how it works, but if I want ext or reiser or whatever to have NLS, I'll have to make them support it (btw, if I do so, wont it be rejected?). I want to move the NLS one level upper so the filesystem imlementations won't have to worry about it any more. I don't have much kernel experience, and none in the fs area, so I can't explain it any better, but hope you get the idea.
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
I was of the impression that most filesystems (ext3 included) treated file names as a sequence of bytes, and didn't care about encoding. Please correct me if I am wrong.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 15 lines · Alan Cox
Quoted 6 lines · Alexey Zaytsev
Yes, that's how it works, but if I want ext or reiser or whatever to have NLS, I'll have to make them support it (btw, if I do so, wont it be rejected?). I want to move the NLS one level upper so the filesystem imlementations won't have to worry about it any more. I don't have much kernel experience, and none in the fs area, so I can't explain it any better, but hope you get the idea.
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
There are some good technical reasons too
Encodings don't map 1:1 - two names may cease to be unique
Quoted 7 lines
Encodings vary in length - image a file name that is longer than the allowed maximum on your system with your encoding choice - that could occur with KOI8-R to UTF-8 I believe
That said it ought to be possible to use the stackable fs work (FUSE etc) to write a layer you can mount over any fs that does NLS translation.
Or just make a symbolic linked shadow FS with translated filenames (UNTESTED):
cd /tmp
cp -src /mnt/problem_dir .
find problem_dir -exec bash -c "mv \'{}\' \'$(echo {} | iconv -f KOI8-R -t UTF-8)\'" \;
Signature · 3 lines
Rutger Nijlunsing ---------------------------------- eludias ed dse.nl never attribute to a conspiracy which can be explained by incompetence ----------------------------------------------------------------------
Quoted 4 lines · Måns Rullgård
Different users of the same system may have perfectly valid reasons to use different locale settings, and thus different filename encodings. Forcing one thing or another is just a useless restriction, and probably not POSIX compliant.
I agree.
Although some people (like glib2 developers) try to say that filenames should be in UTF-8, this doesn't work, just because the "ls" command assumes that they are in the locale charset. Please fix glibc and/or coreutils and all other programs first.
Signature · 1 line
Alexander E. Patrakov
Quoted 10 lines · Alexander E. Patrakov
Quoted 4 lines · Måns Rullgård
Different users of the same system may have perfectly valid reasons to use different locale settings, and thus different filename encodings. Forcing one thing or another is just a useless restriction, and probably not POSIX compliant.
I agree.
Although some people (like glib2 developers) try to say that filenames should be in UTF-8, this doesn't work, just because the
IMHO, the glib developers are clueless, and the GNOME crew even more so. I remember when the gtk file selection dialog stopped displaying files with "bad" names, unless I set some wacky undocumented environment variable first.
Quoted 2 lines
"ls" command assumes that they are in the locale charset. Please fix glibc and/or coreutils and all other programs first.
I use utf-8 exclusively for my filenames (the few that are not 7-bit ascii). Forcing others who use the system to do the same would cause them a lot of trouble, as they must transfer files to and from Windows machines that use anything but utf-8. The result is that some filenames are in utf-8, some iso-8859-1, and some euc-kr. As long as these stay in each users' home directory, it all works quite well, though.
Signature · 2 lines
Måns Rullgård [email protected]
Hi.
A related issue I had is that some codepages don't have a nls module yet.
For example I once or twice needed to mount a vfat filesystem with cp1256 ( arabic windows ) charset. There is no such nls module. I couldn't find any documentation about how to create the module ( maybe I didn't look hard enough ).
Therefore a filesystem used by old windows versions having arabic names is unusable.
Quoted 9 lines · Islam Amer
Hi.
A related issue I had is that some codepages don't have a nls module yet.
For example I once or twice needed to mount a vfat filesystem with cp1256 ( arabic windows ) charset. There is no such nls module. I couldn't find any documentation about how to create the module ( maybe I didn't look hard enough ).
Therefore a filesystem used by old windows versions having arabic names is unusable.
Did you try looking at some of the already existing ones?
Signature · 2 lines
Måns Rullgård [email protected]
Yes, and couldn't figure out how they were generated, newbie here. I read somewhere the character tables get generated automatically from a UTF8 reference, but can't find that article now.
I am ready to invest time in trying to create (shudder) the module and testing if it fixes the problem I describe, if someone could give me a pointer.
Thanks.
Måns Rullgård napisał(a):
Quoted 4 lines
I use utf-8 exclusively for my filenames (the few that are not 7-bit ascii). Forcing others who use the system to do the same would cause them a lot of trouble, as they must transfer files to and from Windows machines that use anything but utf-8.
But VFAT (and NTFS???) use unicode, i.e. UTF-16 (???). AFAIK
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
Quoted 7 lines · Lukasz Stelmach
Måns Rullgård napisał(a):
Quoted 4 lines
I use utf-8 exclusively for my filenames (the few that are not 7-bit ascii). Forcing others who use the system to do the same would cause them a lot of trouble, as they must transfer files to and from Windows machines that use anything but utf-8.
But VFAT (and NTFS???) use unicode, i.e. UTF-16 (???). AFAIK
No, VFAT and NTFS use an 8-bit encoding, and I think its only NTFS5 that is forced to only use UTF-8 as the encoding (vfat can use any 8-bit encoding, and NTFS4 afaik can as well, and I don't think NTFS5 uses UTF-16 internally),
Forcing people to use unicode isn't a bad thing btw, especially since it is a culture agnostic encoding that can represent wide characters (eg. from Asian languages) in a uniform manner*, and allowing to use multiple languages (eg. Chinese and Japanese) at once without needing to switch encodings.
This also means it fixes the 'bug' where you have multiple encodings for the same language (ie. JIS, SJIS, and EUC_JP; similarly, simplified Chinese has 5 popular encoding methods, and traditional Chinese as three), which allows easier sharing of data between users without needing to muck with encoding.
* Unicode can do so in UTF-8 as well as UTF-16 to describe the entire 20-bit space.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 6 lines · Alexey Zaytsev
Instead of adding NLS support to filesystems who don't have it yet, I think there should be a global NLS layer, to convert file names from any to any encoding, independent of file system and transparently to the user.
So what do you think? Is it all nonsense or maybe I should try to implement it?
I do not understand how this is going to look from userspace perspective. Can you give examples how this will work?
--
vda
Patrick McFarland napisał(a):
Quoted 12 lines · Lukasz Stelmach
Quoted 9 lines
Måns Rullgård napisał(a):
Quoted 4 lines
I use utf-8 exclusively for my filenames (the few that are not 7-bit ascii). Forcing others who use the system to do the same would cause them a lot of trouble, as they must transfer files to and from Windows machines that use anything but utf-8.
But VFAT (and NTFS???) use unicode, i.e. UTF-16 (???). AFAIK
No, VFAT and NTFS use an 8-bit encoding,
I meant that they don't use utf-8 but it is still the unicode. I am not sure i've made myself clear.
Quoted 5 lines
Forcing people to use unicode isn't a bad thing btw, especially since it is a culture agnostic encoding that can represent wide characters (eg. from Asian languages) in a uniform manner*, and allowing to use multiple languages (eg. Chinese and Japanese) at once without needing to switch encodings.
Yes. I also think UTF-8 is a good idea, however it is not an ideal one. It *preferes* Roman encodings since some Asian characters need even four bytes.
IMHO for *every* filesystem there need to be an *option* to:
1. store filenames in utf-8 (that is quite possible today) or any other unicode form.
2. convert them to/from a desired iocharset. I prefere using ISO-8859-2 on my system for not every tool support utf-8 today (hopefuly yet).
Of course if a user whishes to store filenames in some other encoding she should be *able* to do so (that is why i like linux).
Generally. IMHO VFAT is a good example how character encoding needs to be handeled.
Best regards.
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
Quoted 23 lines · Lukasz Stelmach
Patrick McFarland napisał(a):
Quoted 12 lines · Lukasz Stelmach
Quoted 9 lines
Måns Rullgård napisał(a):
Quoted 4 lines
I use utf-8 exclusively for my filenames (the few that are not 7-bit ascii). Forcing others who use the system to do the same would cause them a lot of trouble, as they must transfer files to and from Windows machines that use anything but utf-8.
But VFAT (and NTFS???) use unicode, i.e. UTF-16 (???). AFAIK
No, VFAT and NTFS use an 8-bit encoding,
I meant that they don't use utf-8 but it is still the unicode. I am not sure i've made myself clear.
Quoted 5 lines
Forcing people to use unicode isn't a bad thing btw, especially since it is a culture agnostic encoding that can represent wide characters (eg. from Asian languages) in a uniform manner*, and allowing to use multiple languages (eg. Chinese and Japanese) at once without needing to switch encodings.
Quoted 3 lines
Yes. I also think UTF-8 is a good idea, however it is not an ideal one. It *preferes* Roman encodings since some Asian characters need even four bytes.
That's a simple consequence of having an alphabet with thousands of characters. Besides, fewer of the Asian characters are required to represent the same meaning, so the byte count should come out about the same.
Quoted 4 lines
IMHO for *every* filesystem there need to be an *option* to:
1. store filenames in utf-8 (that is quite possible today) or any other unicode form.
export LC_CTYPE=whatever.utf-8
Quoted 2 lines
2. convert them to/from a desired iocharset. I prefere using ISO-8859-2 on my system for not every tool support utf-8 today (hopefuly yet).
man iconv
Quoted 2 lines
Of course if a user whishes to store filenames in some other encoding she should be *able* to do so (that is why i like linux).
That's the current situation.
Quoted 2 lines
Generally. IMHO VFAT is a good example how character encoding needs to be handeled.
IMHO, VFAT is only a good example of bad design.
Signature · 2 lines
Måns Rullgård [email protected]
Måns Rullgård napisał(a):
Quoted 6 lines
Quoted 4 lines
IMHO for *every* filesystem there need to be an *option* to:
1. store filenames in utf-8 (that is quite possible today) or any other unicode form.
export LC_CTYPE=whatever.utf-8
Translate and store them in utf-8 at kernel level same as VFAT mounted with iocharset option.
Quoted 4 lines
Quoted 2 lines
2. convert them to/from a desired iocharset. I prefere using ISO-8859-2 on my system for not every tool support utf-8 today (hopefuly yet).
man iconv
There are far more programmes than only iconv. First of all readline library is kind of broken because it counts (or at least it did a year ago) bytes instead of characters. I won't use UTF-8 nor force anybody else to do so until readline will handle it properly.
Quoted 4 lines
Quoted 2 lines
Of course if a user whishes to store filenames in some other encoding she should be *able* to do so (that is why i like linux).
That's the current situation.
And it is good in a way, however, i think kernel level translation should be also possible. Either done by a code in each filsystem or by some layer above it.
Quoted 4 lines
Quoted 2 lines
Generally. IMHO VFAT is a good example how character encoding needs to be handeled.
IMHO, VFAT is only a good example of bad design.
It depend's on what it is used for. It is very good fs for removable media. None of linux native filesystems is good for this because of different uids on different machines. Since VFAT uses unicode it is possible to see the filenames properly on systems using different codepages for the same language (1:1 is possible).
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
Quoted 15 lines · Alan Cox
Quoted 6 lines · Alexey Zaytsev
Yes, that's how it works, but if I want ext or reiser or whatever to have NLS, I'll have to make them support it (btw, if I do so, wont it be rejected?). I want to move the NLS one level upper so the filesystem imlementations won't have to worry about it any more. I don't have much kernel experience, and none in the fs area, so I can't explain it any better, but hope you get the idea.
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
There are some good technical reasons too
Encodings don't map 1:1 - two names may cease to be unique
Quoted 8 lines
Encodings vary in length - image a file name that is longer than the allowed maximum on your system with your encoding choice - that could occur with KOI8-R to UTF-8 I believe
That said it ought to be possible to use the stackable fs work (FUSE etc) to write a layer you can mount over any fs that does NLS translation.
Now I quite agree that it isn't a Great Idea to do such conversion in the kernel, but the problem still remains and there is no other place we can do it. I belive that it should be done now and removed after the world finishes to move to utf. Maybe it should not be applyed to the main kernel tree, but I'm sure that at least Russian linux distributions will like it.
Quoted 4 lines · Lukasz Stelmach
There are far more programmes than only iconv. First of all readline library is kind of broken because it counts (or at least it did a year ago) bytes instead of characters. I won't use UTF-8 nor force anybody else to do so until readline will handle it properly.
Well utf8 would sure be nice if everything used it, and getting nice fonts for it was simple.
Quoted 3 lines
And it is good in a way, however, i think kernel level translation should be also possible. Either done by a code in each filsystem or by some layer above it.
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
Quoted 5 lines
It depend's on what it is used for. It is very good fs for removable media. None of linux native filesystems is good for this because of different uids on different machines. Since VFAT uses unicode it is possible to see the filenames properly on systems using different codepages for the same language (1:1 is possible).
VFAT uses unicode? I thought it used the same codepage silyness as FAT did, since after all ti was just supposed to be a long filename extension to FAT. Do they use unicode in the long filenames only?
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
Len Sorensen
Lennart Sorensen napisał(a):
Quoted 6 lines
Quoted 3 lines
And it is good in a way, however, i think kernel level translation should be also possible. Either done by a code in each filsystem or by some layer above it.
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
Quoted 3 lines
VFAT uses unicode? I thought it used the same codepage silyness as FAT did, since after all ti was just supposed to be a long filename extension to FAT. Do they use unicode in the long filenames only?
Yes, it uses unicode. And dos codepages in short ones. To prove this take a vfat floppy and mount it. touch(1) a file on it that has some non latin1 characters. Unmount the floppy then do dd if=/dev/fd0 of=/tmp/floppy bs=1024 count=512. While it's done take some hex editor/viewer and seek the latin1-complaint part of the filename in the floppy file (search for uppercase string). Righ above the short filename you'll find multibyte long one.
Quoted 3 lines
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
Quoted 2 lines · Lennart Sorensen
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
Um, thats impossible, unless you're implying something like the file system not being 8-bit safe. The only thing UTF-8 does is store data in bytes, it doesn't need any real support from the file system.
Quoted 9 lines
Quoted 5 lines
It depend's on what it is used for. It is very good fs for removable media. None of linux native filesystems is good for this because of different uids on different machines. Since VFAT uses unicode it is possible to see the filenames properly on systems using different codepages for the same language (1:1 is possible).
VFAT uses unicode? I thought it used the same codepage silyness as FAT did, since after all ti was just supposed to be a long filename extension to FAT. Do they use unicode in the long filenames only?
I mentioned earlier that VFAT uses 8-bit encodings, none of them (supported by Windows, at least) are Unicode.
Quoted 3 lines
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
I agree. UDF is the true successor to the portable media throne.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 6 lines · Lukasz Stelmach
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
Bingo. Only the operating system itself and software displaying filenames needs to understand Unicode; the file system implementation itself just knows its a string of bytes and nothing else.
Quoted 3 lines
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Uh, 20%? That sounds awfully high. You sure you didn't do something wrong?
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 2 lines · Alan Cox
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
What happens if you 'field upgrade' ext2 to ext3 by adding a journal? That doesn't magically convert !utf-8 to utf-8.
Quoted 3 lines
There are some good technical reasons too
Encodings don't map 1:1 - two names may cease to be unique
Hold up. Unless the original encoding is 'wrong' and has two mapped characters that, in reality, are the same character, no such uniqueness should stop. (This implies the encoding that we switched to 'fixed' said 'bug')
Quoted 3 lines
Encodings vary in length - image a file name that is longer than the allowed maximum on your system with your encoding choice - that could occur with KOI8-R to UTF-8 I believe
Thats a fault of the file system design, not of the encoding. File systems should not have very short filenames.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 11 lines · Alexey Zaytsev
Quoted 3 lines · Alan Cox
That said it ought to be possible to use the stackable fs work (FUSE etc) to write a layer you can mount over any fs that does NLS translation.
Now I quite agree that it isn't a Great Idea to do such conversion in the kernel, but the problem still remains and there is no other place we can do it. I belive that it should be done now and removed after the world finishes to move to utf. Maybe it should not be applyed to the main kernel tree, but I'm sure that at least Russian linux distributions will like it.
I partially agree. I think no userland application should have access to the un-'fixed' file names; they should be fed only Unicode to prevent the spread and acceptance of out of date encodings.
Forcing users to do smart things is often the only way to make them do smart things, and the lack of acceptance of Unicode on Linux in the wild seems to be the only way.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 2 lines · Denis Vlasenko
I do not understand how this is going to look from userspace perspective. Can you give examples how this will work?
IMHO, he means that the userspace would only see Unicode filenames, and the userspace could only give Unicode names back to the kernel. The kernel, using this global NLS layer would translate back and forth, and the userland wouldn't know about it.
Its basically the only sane way to approach the problem of getting the entire Linux community to convert to Unicode.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 6 lines · Patrick McFarland
Quoted 2 lines · Alan Cox
An ext3fs is always utf-8. People might have chosen to put other encodings on it but thats "not our fault" ;)
What happens if you 'field upgrade' ext2 to ext3 by adding a journal? That doesn't magically convert !utf-8 to utf-8.
Ext2/3's encoding has always been utf-8. Period.
There have been some people who have chosen to do something else locally, but that was about as valid as the people who violated SMTP standards by Just Sending 8-bits instead of using MIME.
- Ted
Quoted 5 lines · Theodore Ts'o
Ext2/3's encoding has always been utf-8. Period.
There have been some people who have chosen to do something else locally, but that was about as valid as the people who violated SMTP standards by Just Sending 8-bits instead of using MIME.
Ahh. Whoever made that choice way back at the beginning of Ext2 development rocks. Whoever you are, Thanks.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
[RFC] Filesystem name storage (Was: A Great Idea (tm) about reimplementing NLS.)
In reply to Patrick McFarlandQuoted 11 lines · Patrick McFarland
Quoted 2 lines · Denis Vlasenko
I do not understand how this is going to look from userspace perspective. Can you give examples how this will work?
IMHO, he means that the userspace would only see Unicode filenames, and the userspace could only give Unicode names back to the kernel. The kernel, using this global NLS layer would translate back and forth, and the userland wouldn't know about it.
Its basically the only sane way to approach the problem of getting the entire Linux community to convert to Unicode.
Would the following system for filenames resolve most of the issues people are raising:
First load charset tables into the kernel. These would be stored in files in userspace and could be easily updated, renamed, deleted, etc. Such a table would always be a translation from Unicode <=> Charset. A kernel with this system built in would understand natively "raw", "utf8", "utf16", and "utf32", anything else would need loaded charset tables.
The following mount options would available:
nls_raw=(0|1) [default 1]:
This would cause Linux to pass all chars through unmolested.
This mode
works well on multiuser systems where users want to use their
own NLS
tools, or where the whole system uses UTF-8, including the
filesystems.
This is backwards compatible with the way Linux currently
presents most
(all?) filesystems. If the options "nls_disk" or "nls_user" are
used,
then this option is forced to be zero.
nls_disk=<string-charset>
This specifies the underlying charset which should be used on
the disk
or filesystem itself. This may be "negotiate" for any filesystems which support NLS *and* can identify which charset is in use.
Built in
options are "utf8", "utf16", and "utf32". Defaults to
"negotiate" if
available otherwise "utf8", but only defaults if "nls_raw" is 0.
nls_user=<string-charset>
This specifies the charset which should be presented to the
user. This
may be used to allow a backwards compatibility (IE: A program wants ISO8859-1, but the admin wants the underlying filesystem to use
UTF-8.
Built in options are "utf8", "utf16", and "utf32". Defaults to
"utf8"
if "nls_raw" is 0.
The end result is that specifying either nls_disk or nls_user will turn on automatic NLS conversion, with the unspecified nls_ option being utf8.
If these options are used on bind mounts, they should override the underlying filesystem's mount options (Instead of stacking). This will allow the admin to specify:
# mount -t ext3 -o nls_disk=utf8,nls_user=utf8 /dev/hdb /mnt # mount --bind -o nls_disk=utf8,nls_user=iso8850-1 /mnt/mail /var/spool/mail
if he/she wants to provide backwards compatibility with a legacy mail spooling program. Note: A part of each translation table would be an entry for "Unspecified character", such that any UTF-8 character not mapped in the table could be translated to a sane default, such as '?'. If names collide under such translation, the kernel would need a way to keep track of the collisions (Appended numbers?) and properly re-resolve them when asked.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Quoted 3 lines · Theodore Ts'o
[snip]
Ext2/3's encoding has always been utf-8. Period.
In what way does Ext2/3 know or care about file name encoding? Doesn't it just store an arbitrary 8-byte string? Couldn't someone claim that from the start it was designed to use iso8859-1 just as easily as you can claim it was designed to use utf-8?
Signature · 1 line
Jeremy Maitin-Shepard
Quoted 9 lines · Lukasz Stelmach
Lennart Sorensen napisał(a):
Quoted 6 lines
Quoted 3 lines
And it is good in a way, however, i think kernel level translation should be also possible. Either done by a code in each filsystem or by some layer above it.
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
Quoted 6 lines
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
That's exactly how ext3, reiserfs, xfs, jfs, etc. all work. A few filesystems are tagged as using some specific encoding. If your filesystem is marked for iso-8859-1, what should a kernel with a conversion mechanism do if a user tries to name a file 김?
Quoted 7 lines
Quoted 3 lines
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Filename, timestamps, all the usual.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 7 lines · Patrick McFarland
Quoted 6 lines
Now I quite agree that it isn't a Great Idea to do such conversion in the kernel, but the problem still remains and there is no other place we can do it. I belive that it should be done now and removed after the world finishes to move to utf. Maybe it should not be applyed to the main kernel tree, but I'm sure that at least Russian linux distributions will like it.
Quoted 7 lines
I partially agree. I think no userland application should have access to the un-'fixed' file names; they should be fed only Unicode to prevent the spread and acceptance of out of date encodings.
Forcing users to do smart things is often the only way to make them do smart things, and the lack of acceptance of Unicode on Linux in the wild seems to be the only way.
I'm not going to force anybody to do anything. There is a nubmer of reasons to use unicode, but if somebody finds them not convincing, he is free to use any other encoding. If somebody uses koi8-r as his primary encoding and wants to mount a cp1251 or unicode-encoded file system, he is free to do it, although he should be ready to loose some characters.
Quoted 8 lines · Patrick McFarland
Quoted 6 lines · Lukasz Stelmach
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
Quoted 4 lines
Bingo. Only the operating system itself and software displaying filenames needs to understand Unicode; the file system implementation itself just knows its a string of bytes and nothing else.
Not even the OS needs to know what the bytes mean. Only applications displaying the names have reason to interpret the bytes they are composed of in any specific manner.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 3 lines · In article <[email protected]> you wrote:
Bingo. Only the operating system itself and software displaying filenames needs to understand Unicode; the file system implementation itself just knows its a string of bytes and nothing else.
The filesystem needs to understand and translate the path names, if the Filesystem specification mandates a different encoding for file names (for example UTF-16). Thats why you fequently see translations in legacy filesystems which have national encoding in the on-disk format (think FAT)
And if the filesystem uses unicode, you also need to enforce the naming of files in UTF-8 to do reliable translation. Currently user mode may also send you non-UTF8 national chars.
Bernd
Quoted 6 lines · Lukasz Stelmach
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
That is true. UTF-8 wouldn't cause any more problems than ascii already does, such as some filesystems not allowing : and * in filenames among other characters.
Quoted 7 lines
Yes, it uses unicode. And dos codepages in short ones. To prove this take a vfat floppy and mount it. touch(1) a file on it that has some non latin1 characters. Unmount the floppy then do dd if=/dev/fd0 of=/tmp/floppy bs=1024 count=512. While it's done take some hex editor/viewer and seek the latin1-complaint part of the filename in the floppy file (search for uppercase string). Righ above the short filename you'll find multibyte long one.
Well at least that seems like they did something right when they extended FAT with VFAT. Doesn't make FAT a good filesystem, but it does make the filename extension pretty nice, much as it is an ugly hack too.
Quoted 3 lines
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
The constant rewriting of the same sectors that store the FAT is really bad for some types of flash and other removeably media (like DVD-RAM).
I hadn't noticed any big overhead in UDF, although packet writing may add some overhead itself (I never used packet writing).
Len Sorensen
Quoted 6 lines · Jeremy Maitin-Shepard
Quoted 1 line
Ext2/3's encoding has always been utf-8. Period.
In what way does Ext2/3 know or care about file name encoding? Doesn't it just store an arbitrary 8-byte string? Couldn't someone claim that from the start it was designed to use iso8859-1 just as easily as you can claim it was designed to use utf-8?
Because we've had this discussion^H^H^H^H^H^H^H^H^H^H^H flame war years ago, and despite people from Russia whining that that it took 3 bytes to encode each Cyrillic character in UTF-8, it's where we came out.
The bottom-line though is that if someone files a bug report with ext3 because one user on the system was is creating filenames in Japanese, and another user on the same time-sharing system is creating filenames in Germany, and they fail to interoperate, and they were doing so in their local language, we would laugh at them --- just as people writing mail programs would laugh at people who complained that they were running into problems Just Sending 8-bits instead of using MIME, and could you please fix this business-critical bug?
Or as more and more desktop programs start interpreting the filenames as UTF-8, and people with local variations get screwed, that is their problem, and Not Ours.
So no, we can't prevent anyone from shooting them in the foot. However, if they *do* take the gun, aim it straight downwards, and pull the trigger, we aren't obligated to help.
- Ted
Quoted 20 lines · Lennart Sorensen
Quoted 6 lines · Lukasz Stelmach
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
That is true. UTF-8 wouldn't cause any more problems than ascii already does, such as some filesystems not allowing : and * in filenames among other characters.
Quoted 7 lines
Yes, it uses unicode. And dos codepages in short ones. To prove this take a vfat floppy and mount it. touch(1) a file on it that has some non latin1 characters. Unmount the floppy then do dd if=/dev/fd0 of=/tmp/floppy bs=1024 count=512. While it's done take some hex editor/viewer and seek the latin1-complaint part of the filename in the floppy file (search for uppercase string). Righ above the short filename you'll find multibyte long one.
[SNIPPED...]
Quoted 2 lines
Len Sorensen
You know this problem was "solved" over 20 years ago when it was discovered that file-names could never be long enough. The solution was a container-file which contained as much stuff as necessary to identity the contents of the file that it was associated with. Using this technique, the "real" file didn't need any ASCII identifiers. The real file didn't show up in some directory program, just the contents of the container-file. This same technique could be used for any arbitrary file-identification including characters that haven't been invented yet.
Cheers,
Dick Johnson
Penguin : Linux version 2.6.11.9 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.
Quoted 9 lines · Richard B. Johnson
You know this problem was "solved" over 20 years ago when it was discovered that file-names could never be long enough. The solution was a container-file which contained as much stuff as necessary to identity the contents of the file that it was associated with. Using this technique, the "real" file didn't need any ASCII identifiers. The real file didn't show up in some directory program, just the contents of the container-file. This same technique could be used for any arbitrary file-identification including characters that haven't been invented yet.
Why am I suddenly reminded of apple's idiotic filesystem forks for resources and data? Such a pain when trying to transfer files to other types of filesystems. Modifying the files themselves also doesn't seem like the right solution.
As for filenames never being long enough, I don't think that is true. Filenames CAN be too long, but I don't see very many people think 250 characters makes for a useful filename. Most people seem happy with 50 or so being a good limit even though many systems support much longer. 8 wasn't enough, and 25 or 30 was sometimes a bit short, but usually enough. Not having enough filename length doesn't seem to be a problem in need of a solution on most systems anymore.
Len Sorensen
Quoted 5 lines · Patrick McFarland
Quoted 3 lines
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Uh, 20%? That sounds awfully high. You sure you didn't do something wrong?
Fixed packet recording under UDF records data in 32/39 of sectors; this alone is 18%; I can easily see 2% in UDF itself.
More specifically, each packet has a seven sector overhead. UDF sets the packet size to 32.
Signature · 1 line
Rob
Quoted 2 lines · Lennart Sorensen
Most people seem happy with 50 or so being a good limit even though many systems support much longer.
50 characters or 50 bytes? Because in the case of UTF-8, if you do a lot of three byte characters (which require four bites to encode), 50 bytes is very short.
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 7 lines · Patrick McFarland
Quoted 2 lines · Lennart Sorensen
Most people seem happy with 50 or so being a good limit even though many systems support much longer.
50 characters or 50 bytes? Because in the case of UTF-8, if you do a lot of three byte characters (which require four bites to encode), 50 bytes is very short.
What do you mean by three-byte characters requiring four bytes to encode? Is a three-byte character not a character encoded using three bytes?
As for 50 bytes being too short, many of the multibyte characters are equivalent to several English characters, so fewer of them are required. You have a point, though.
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 5 lines · Måns Rullgård
Quoted 4 lines · Patrick McFarland
Quoted 2 lines · Lennart Sorensen
Most people seem happy with 50 or so being a good limit even though
many systems support much longer.
Quoted 7 lines
Quoted 3 lines
50 characters or 50 bytes? Because in the case of UTF-8, if you do a lot of three byte characters (which require four bites to encode), 50 bytes is very short.
What do you mean by three-byte characters requiring four bytes to encode? Is a three-byte character not a character encoded using three bytes?
(implication of utf8 and not utf16 goes here)
Very few Unicode characters require three bytes, instead of the usual one or two.
For one byte you just have the byte.
For two bytes, you really have three: a control code stating "the following two bytes are a two byte character", and then the two bytes.
For three bytes, you really have four bytes: a control code stating "the following three bytes are a three byte character" and then the three bytes.
Unless I've completely misunderstood the Unicode specification, this is what is going on.
Quoted 3 lines
As for 50 bytes being too short, many of the multibyte characters are equivalent to several English characters, so fewer of them are required. You have a point, though.
Any English characters (ie, the first 127 ascii characters) map directly to the first 127 Unicode characters (if thats what you meant).
Signature · 4 lines
Patrick "Diablo-D3" McFarland || [email protected] "Computer games don't affect kids; I mean if Pac-Man affected us as kids, we'd all be running around in darkened rooms, munching magic pills and listening to repetitive electronic music." -- Kristian Wilson, Nintendo, Inc, 1989
Quoted 10 lines · Patrick McFarland
Quoted 9 lines · Måns Rullgård
Quoted 7 lines · Patrick McFarland
Quoted 2 lines · Lennart Sorensen
Most people seem happy with 50 or so being a good limit even though
many systems support much longer.50 characters or 50 bytes? Because in the case of UTF-8, if you do a lot of three byte characters (which require four bites to encode), 50 bytes is very short.
Quoted 8 lines
Quoted 3 lines
What do you mean by three-byte characters requiring four bytes to encode? Is a three-byte character not a character encoded using three bytes?
(implication of utf8 and not utf16 goes here)
Very few Unicode characters require three bytes, instead of the usual one or two.
I wouldn't the Chinese, Japanese, and Korean characters "very few", and those all require (at least) three bytes.
Quoted 1 line
For one byte you just have the byte.
Correct.
Quoted 7 lines
For two bytes, you really have three: a control code stating "the following two bytes are a two byte character", and then the two bytes.
For three bytes, you really have four bytes: a control code stating "the following three bytes are a three byte character" and then the three bytes.
Wrong. The first byte indicates the total size of the character, but it also contains data, like this:
0xxxxxxx
110xxxxx 10xxxxxx
1110xxxx 10xxxxxx 10xxxxxx
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
Refer to the Unicode standard, section 3.9 for the full details.
Quoted 7 lines
Quoted 3 lines
As for 50 bytes being too short, many of the multibyte characters are equivalent to several English characters, so fewer of them are required. You have a point, though.
Any English characters (ie, the first 127 ascii characters) map directly to the first 127 Unicode characters (if thats what you meant).
Let me clarify with an example. The common Korean name Kim consists of three ascii characters. The Hangul spelling, ~, is encoded in utf-8 using three bytes. Even though a three-byte character was used, the number of bytes is the same.
Signature · 2 lines
Måns Rullgård [email protected]
Looks like something ate the Hangul. I'll try again, without any other non-ascii characters.
Quoted 12 lines
Quoted 7 lines
Quoted 3 lines
As for 50 bytes being too short, many of the multibyte characters are equivalent to several English characters, so fewer of them are required. You have a point, though.
Any English characters (ie, the first 127 ascii characters) map directly to the first 127 Unicode characters (if thats what you meant).
Let me clarify with an example. The common Korean name Kim consists of three ascii characters. The Hangul spelling, 김, is encoded in utf-8 using three bytes. Even though a three-byte character was used, the number of bytes is the same.
Quoted 4 lines · In article <[email protected]> you wrote:
(implication of utf8 and not utf16 goes here)
Very few Unicode characters require three bytes, instead of the usual one or two.
UTF-8 2 bytes end with U+07ff which covers only Latin, Cyrillic, Hebrew and Arabic.
All JCK Unified Ideographs (U+4E00-) and Extensions (U+3400-) have 3 byte encodings with UTF-8. Some of the B Extensions even use 4 bytes (U+20000-)
Quoted 1 line
For one byte you just have the byte.
For ASCII you have one byte.
Quoted 2 lines
For two bytes, you really have three: a control code stating "the following two bytes are a two byte character", and then the two bytes.
Umm, thats a bit missleading. UTF-8 works with bit not byte prefixes. Unicode code points are integers and depending on the encoding represented as multiple code points, which can be represented as bytes.
Quoted 2 lines
Unless I've completely misunderstood the Unicode specification, this is what is going on.
You might want to look up Joel's Tutorial or just browse the Unihan Database:
http://www.joelonsoftware.com/articles/Unicode.html
http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=3400
http://www.unicode.org/cgi-bin/UnihanGrid.pl?codepoint=U+07F1&useutf8=false
Greetings
Bernd
Quoted 7 lines · Patrick McFarland
Quoted 2 lines · Lennart Sorensen
Most people seem happy with 50 or so being a good limit even though many systems support much longer.
50 characters or 50 bytes? Because in the case of UTF-8, if you do a lot of three byte characters (which require four bites to encode), 50 bytes is very short.
I would think most languages that need 2 or 3 bytes per character would need a lot less characters, although I think I can think of a few cases where that isn't true.
Well how about making it '50 characters seems plenty for most people to be happy'. If you can handle a couple hundred bytes that should be ok, and I think most filesystems can.
Len Sorensen
Quoted 7 lines · Patrick McFarland
(implication of utf8 and not utf16 goes here)
Very few Unicode characters require three bytes, instead of the usual one or two.
For one byte you just have the byte.
Quoted 8 lines
For two bytes, you really have three: a control code stating "the following two bytes are a two byte character", and then the two bytes.
For three bytes, you really have four bytes: a control code stating "the following three bytes are a three byte character" and then the three bytes.
Unless I've completely misunderstood the Unicode specification, this is what is going on.
You have probably slightly misunderstood UTF8 at least. UTF8 tries very hard to make sure you can't mistake the characters for ascii, so it makes the first byte contains some 1's follwed by one zero. The number of 1's indicates how many bytes the character contains, after the 0 the remaining bits is used to store bits for the character. The remaining bytes are all 10xxxxxx which stores another 6 bites of the character code. One is required to use the shortest form of utf8 that can store the character you are encoding.
x's are where the bits for the character number go:
0xxxxxxx encodes character 0-127
110xxxxx 10xxxxxx encodes character 128-2047
1110xxxx 10xxxxxx 10xxxxxx encodes characters 2048-65535
etc up to
1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx encodes characters 67108864-2147483647
As far as I know, unicode doesn't currently define anything past 20bits or so, so probably 4bytes is the most you will see in normal use, with 3 bytes covering quite a large number of the characters.
Quoted 2 lines
Any English characters (ie, the first 127 ascii characters) map directly to the first 127 Unicode characters (if thats what you meant).
Well utf8 also is backwards compatible with ascii to make handling text files simpler. You could encode the ascii characters using the other part of UTF8 except that would violate the rule of using the shortest form possible.
Len Sorensen
Quoted 8 lines · [email protected] (Lennart Sorensen) writes:
You have probably slightly misunderstood UTF8 at least. UTF8 tries very hard to make sure you can't mistake the characters for ascii, so it makes the first byte contains some 1's follwed by one zero. The number of 1's indicates how many bytes the character contains, after the 0 the remaining bits is used to store bits for the character. The remaining bytes are all 10xxxxxx which stores another 6 bites of the character code. One is required to use the shortest form of utf8 that can store the character you are encoding.
Some characters can be encoded in several equally shortest ways. For instance, characters with multiple diacritics can have these applied in different orders. One of these is designated the canonical encoding, and should be used in favor of the others. Those things, among others, are what makes unicode difficult to deal with.
Signature · 2 lines
Måns Rullgård [email protected]
Måns Rullgård napisał(a):
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
Quoted 4 lines
That's exactly how ext3, reiserfs, xfs, jfs, etc. all work. A few filesystems are tagged as using some specific encoding. If your filesystem is marked for iso-8859-1, what should a kernel with a conversion mechanism do if a user tries to name a file 김?
Return -ENOENT? I am guessing. But please tell me what should do userland software if it runs with locale set to something.iso-8859-2 and finds 김 in the directory? That is the same problem. And for now ISO 8-bit encodings are far more popolar and usefull with contemporary tools than UTF-8. That is why I think suggestion of a layer in the kernel that would translate filenames form utf-8 stored on the media to e.g. latin2 used by tools is quite reasonable. Especially when there is more than one encoding for a particular language (think Russian, Polish). Even more, with such a facility transition would be much more greaceful since you could have utf-8 filesystem and then you can worry about tools other tools. The filesystem is already populated with UFT-8 names.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Filename, timestamps, all the usual.
That's why IMHO FAT is quite enough for this purpose.
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
Re: [RFC] Filesystem name storage (Was: A Great Idea (tm) about reimplementing NLS.)
In reply to Kyle MoffettKyle Moffett napisał(a):
Quoted 5 lines
Would the following system for filenames resolve most of the issues people are raising:
First load charset tables into the kernel. These would be stored in files in userspace
[...]
Just like keyboard maps?
Quoted 2 lines
The following mount options would available:
nls_raw=(0|1) [default 1]:
[...]
Quoted 1 line
nls_disk=<string-charset>
[...]
Quoted 1 line
nls_user=<string-charset>
It sounds quite reasonable to me :-)
Signature · 1 line
Było mi bardzo miło. Trzecia pospolita klęska, [...]
Quoted 1 line
Łukasz< Już nie katolicka lecz złodziejska. (c)PP
fredagen den 17 juni 2005 15.23 skrev Måns Rullgård:
Quoted 1 line
Some characters can be encoded in several equally shortest ways.
No they cannot. How to encode characters i explicitly and well defined. If you don't follow the rules you are simply not producing UTF-8, but something else.
Every unicode character has exactly one UTF-8 representation.
-- robin
Quoted 10 lines
fredagen den 17 juni 2005 15.23 skrev Måns Rullgård:
Quoted 1 line
Some characters can be encoded in several equally shortest ways.
No they cannot. How to encode characters i explicitly and well defined. If you don't follow the rules you are simply not producing UTF-8, but something else.
Every unicode character has exactly one UTF-8 representation.
-- robin
You are confused between unicode characters and unicode codepoints.
Every unicode codepoint has exactly one UTF-8 representation.
Unicode characters may use one ore more unicode codepoints.
Some characters have also representation with one codepoint, but not all.
For example
LATIN CAPITAL LETTER A WITH ACUTE
have presentation 0041 0301
That is two unicode codepoints. That character
have also other (compatibility) representation
that is 00C1
But consider (somewhat imaginary) character
LATIN CAPITAL LETTER A WITH GRAVE AND CIRCUMFLEX
that have presentation 0041 0300 0302
but it have also presentation 0041 0302 0300
Both presentations are equal short.
/ Kari Hurtta
Quoted 1 line · In article <[email protected]> you wrote:
Every unicode character has exactly one UTF-8 representation.
Every unicode code point has exactly one UTF-8 representation, however there are for a few glyphs multiple code points. And this is not only a problem beause of homoglphys which look like/similiar, but also because of combining characters vs. legacy characters. However thats more an issue of the user interface (think IDN exploits).
Personally I think the on-disk filesystem format should be required to be UTF-8, and its an open discussion if the syscalls accept UTF-8 or locale byte encodings. Currently its a mess. We can learn from Windows here:)
Greetings
Bernd
lördagen den 18 juni 2005 20.06 skrev Kari Hurtta:
Quoted 7 lines
Quoted 7 lines
fredagen den 17 juni 2005 15.23 skrev M åns Rullgård:
Quoted 1 line
Some characters can be encoded in several equally shortest ways.
No they cannot. How to encode characters i explicitly and well defined. If you don't follow the rules you are simply not producing UTF-8, but something else.
Quoted 5 lines
Quoted 3 lines
Every unicode character has exactly one UTF-8 representation.
-- robin
You are confused between unicode characters and unicode codepoints.
Yes. i forgot about them. Thank you for reminding me.
-- robin
Quoted 7 lines · In article <[email protected]> you wrote:
As for filenames never being long enough, I don't think that is true. Filenames CAN be too long, but I don't see very many people think 250 characters makes for a useful filename. Most people seem happy with 50 or so being a good limit even though many systems support much longer. 8 wasn't enough, and 25 or 30 was sometimes a bit short, but usually enough. Not having enough filename length doesn't seem to be a problem in need of a solution on most systems anymore.
pathname length however is a bigger issue. 255bytes is pretty short, especially if you distributed file systems with deeply nested mountpoints.
Greetings
Bernd
Quoted 11 lines · Lukasz Stelmach
Måns Rullgård napisał(a):
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
What do you do if the underlying filesystem can not store some unicode characters that are allowed on others?
That's why UTF-8 is suggested. UTF-8 has been developed to "fool" the software that need not to be aware of unicodeness of the text it manages to handle it without any hickups *and* to store in the text information about multibyte characters.What characters exactly you do mean? NULL? There is no NULL byte in any UTF-8 string except the one which terminates it.
Quoted 6 lines
Quoted 4 lines
That's exactly how ext3, reiserfs, xfs, jfs, etc. all work. A few filesystems are tagged as using some specific encoding. If your filesystem is marked for iso-8859-1, what should a kernel with a conversion mechanism do if a user tries to name a file 김?
Return
-ENOENT? I am guessing.
Doesn't seem very friendly.
Quoted 2 lines
But please tell me what should do userland software if it runs with locale set to something.iso-8859-2 and finds 김 in the directory?
I suppose it will display ęš (0x80 doesn't seem be a printable iso-8859-2 character). You told it to use iso-8859-2 in the first place, so what do you expect?
Quoted 2 lines
That is the same problem. And for now ISO 8-bit encodings are far more popolar and usefull with contemporary tools than UTF-8.
ISO 8-bit encodings are more common with characters they can represent. These are a small minority of all characters commonly used.
Quoted 8 lines
That is why I think suggestion of a layer in the kernel that would translate filenames form utf-8 stored on the media to e.g. latin2 used by tools is quite reasonable. Especially when there is more than one encoding for a particular language (think Russian, Polish). Even more, with such a facility transition would be much more greaceful since you could have utf-8 filesystem and then you can worry about tools other tools. The filesystem is already populated with UFT-8 names.
How is the kernel to know what to translate to/from?
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I think UDF is a better filesystem for many types of media since it is able to me more gently to the sectors storing the meta data than VFAT ever will be.
I've tried cd packet writing with UDF and it gives insane overhead of about 20%. What metadata you'd like to store for example on your flashdrive or a floppy disk?
Filename, timestamps, all the usual.
That's why IMHO FAT is quite enough for this purpose.
FAT has a bad habit of constantly hammering the same sectors over and over. This can wear out cheap flash media in no time at all.
Signature · 2 lines
Måns Rullgård [email protected]
Hi!
Quoted 20 lines
Quoted 6 lines
Quoted 1 line
Ext2/3's encoding has always been utf-8. Period.
In what way does Ext2/3 know or care about file name encoding? Doesn't it just store an arbitrary 8-byte string? Couldn't someone claim that from the start it was designed to use iso8859-1 just as easily as you can claim it was designed to use utf-8?
Because we've had this discussion^H^H^H^H^H^H^H^H^H^H^H flame war years ago, and despite people from Russia whining that that it took 3 bytes to encode each Cyrillic character in UTF-8, it's where we came out.
The bottom-line though is that if someone files a bug report with ext3 because one user on the system was is creating filenames in Japanese, and another user on the same time-sharing system is creating filenames in Germany, and they fail to interoperate, and they were doing so in their local language, we would laugh at them --- just as people writing mail programs would laugh at people who complained that they were running into problems Just Sending 8-bits instead of using MIME, and could you please fix this business-critical bug?
Quoted 3 lines
Or as more and more desktop programs start interpreting the filenames as UTF-8, and people with local variations get screwed, that is their problem, and Not Ours.
Actually the day we have rm utf-8-ed, we have a problem. Someone will create two files that have same utf name, encoded differently, and will be in trouble. Remember old > \* "hack"? utf-8 makes variation possible...
If we are serious about utf-8 support in ext3, we should return -EINVAL if someone passes non-canonical utf-8 string.
Pavel
Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
Quoted 4 lines · Pavel Machek
Actually the day we have rm utf-8-ed, we have a problem. Someone will create two files that have same utf name, encoded differently, and will be in trouble. Remember old > \* "hack"? utf-8 makes variation possible...
They are different to POSIX as they are different byte sequences
Quoted 3 lines
If we are serious about utf-8 support in ext3, we should return
-EINVALif someone passes non-canonical utf-8 string.
That would ironically not be standards compliant
Hi!
Quoted 6 lines
Quoted 4 lines
Actually the day we have rm utf-8-ed, we have a problem. Someone will create two files that have same utf name, encoded differently, and will be in trouble. Remember old > \* "hack"? utf-8 makes variation possible...
They are different to POSIX as they are different byte sequences
Does POSIX really say that all weird characters must be accepted in path name?
Quoted 4 lines
Quoted 2 lines
If we are serious about utf-8 support in ext3, we should return
-EINVALif someone passes non-canonical utf-8 string.That would ironically not be standards compliant
I don't see how we can claim ext3 is utf-8 then. If application vendors believed us and accepted that ext3 filenames are in utf-8, they'd do wrong thing because kernel is perfectly willing to feed them non-utf-8 things.
Pavel
Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
Quoted 11 lines · Pavel Machek
Hi!
Quoted 6 lines
Quoted 4 lines
Actually the day we have rm utf-8-ed, we have a problem. Someone will create two files that have same utf name, encoded differently, and will be in trouble. Remember old > \* "hack"? utf-8 makes variation possible...
They are different to POSIX as they are different byte sequences
Does POSIX really say that all weird characters must be accepted in path name?
POSIX only requires [A-Za-z0-9._-].
Andreas.
Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected] SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."
Måns Rullgård napisał(a):
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
That's exactly how ext3, reiserfs, xfs, jfs, etc. all work. A few filesystems are tagged as using some specific encoding. If your filesystem is marked for iso-8859-1, what should a kernel with a conversion mechanism do if a user tries to name a file 김?
Return
-ENOENT? I am guessing.Doesn't seem very friendly.
Well, if user marks her fs as iso-8859-1 that means that she doesn't want it to contain filenames unrepresentable in this particular codepage. Aleksey has begun the whole thread because in Russia there are several, equally popular, different encodings for the same alphabet. And in this context his proposal is quite good: develope general, fs independent NLS layer.
Quoted 7 lines
Quoted 2 lines
But please tell me what should do userland software if it runs with locale set to something.iso-8859-2 and finds 김 in the directory?
I suppose it will display ęš (
0x80doesn't seem be a printable iso-8859-2 character). You told it to use iso-8859-2 in the first place, so what do you expect?
ls(1) displays either \0nnn or ?. Or maybe some other mangling could be done, however, octal representation seems to be ok.
Quoted 7 lines
Quoted 2 lines
That is the same problem. And for now ISO 8-bit encodings are far more popular and usefull with contemporary tools than UTF-8.
ISO 8-bit encodings are more common with characters they can represent. These are a small minority of all characters commonly used.
OK. Let me be more general: fixed char width encodings. AFAIK Japanese encodigs use 16bits, yet it is still fixed width.
Quoted 11 lines
Quoted 8 lines
That is why I think suggestion of a layer in the kernel that would translate filenames form utf-8 stored on the media to e.g. latin2 used by tools is quite reasonable. Especially when there is more than one encoding for a particular language (think Russian, Polish). Even more, with such a facility transition would be much more greaceful since you could have utf-8 filesystem and then you can worry about tools other tools. The filesystem is already populated with UFT-8 names.
How is the kernel to know what to translate to/from?
Mount options. See the letter from Kyle Moffett
<[email protected]>
[ good filesystem for portable media ]
Quoted 4 lines
Quoted 1 line
That's why IMHO FAT is quite enough for this purpose.
FAT has a bad habit of constantly hammering the same sectors over and over. This can wear out cheap flash media in no time at all.
Maybe. I don't think that digital cameras or audio players will suppoty UDF though. But that is something completly differnent.
Signature · 1 line
By³o mi bardzo mi³o. Trzecia pospolita klêska, [...]
Quoted 1 line
£ukasz< Ju¿ nie katolicka lecz z³odziejska. (c)PP
Quoted 8 lines · Alan Cox
On Sul, 2005-06-19 at 18:55, Pavel Machek wrote:
...Quoted 3 lines
If we are serious about utf-8 support in ext3, we should return
-EINVALif someone passes non-canonical utf-8 string.That would ironically not be standards compliant
Which standards?
The standards I've read (mostly XML- and web-related specs) do say that non-standard UTF-8 octet sequences should be rejected.
Daniel
Signature · 2 lines
Daniel Barclay [email protected]
Quoted 2 lines · Daniel B.
Quoted 1 line · Alan Cox
On Sul, 2005-06-19 at 18:55, Pavel Machek wrote:
[...]
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
If we are serious about utf-8 support in ext3, we should return
-EINVALif someone passes non-canonical utf-8 string.That would ironically not be standards compliant
Which standards?
Probably POSIX, SuSv3 and similiar.
Quoted 2 lines
The standards I've read (mostly XML- and web-related specs) do say that non-standard UTF-8 octet sequences should be rejected.
There you have basically text files with some structure in it and the definiton/requirement that is is UTF-8.
At kernel level these are also just byte streams and the kernel doesn't know or care which charset, encoding, file format, font etc. the data is used with or interpreted several layers higher, e.g. for presenting to the user. And the same holds for filenames.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 1 line · Daniel B.
Which standards?
Traditional unix namespace is a sequence of bytes with '/' as a seperator and \0 as a terminator. There are no other restrictions. UTF-8 is essentially a retrofit onto that.
Quoted 2 lines
The standards I've read (mostly XML- and web-related specs) do say that non-standard UTF-8 octet sequences should be rejected.
If you follow the thread further various people pointed out that POSIX and other standard documents are actually more restrictive in their guarantees so my belief by a strict standards reading is wrong. It'll break a few apps if you enforced it (lots if you took the minimal posix requirement).