NLS mappings for iso-8859-* encodings
Quoted 6 lines · Urban Widmark
On Tue, 7 May 2002, Petr Vandrovec wrote:
ncpfs should perhaps not use iso8859-x to read filenames in some cp* encoding. The default nls you can specify is strange, is it the default for chars on the filesystem or the default to use for display?
Quoted 8 lines
isofs uses it for display (and has no need for a second nls table). smbfs uses it for display and has a second default for the remote chars. ncpfs uses it as default for both display and remote.
vfat also uses it for both on-disk and display.I think ncpfs should demand that the user sets two defaults and if that isn't done no default translation is made (just do a memcpy in
ncp__vol2ioandncp__io2vol). That's what smbfs does anyway.
Yes, it looks like a good idea.
Quoted 6 lines
In unicode the 0x80-0x9F does not contain any printable characters, but they are defined. I know one table for iso8859-1 that lists that part as being empty/undefined, but it's not an iso document.
For someone setting their default to iso8859-1 that patch is probably ok, but what happens when someone sets it to a variable length encoding? (sjis)
They still have a problem - but they'll probably know what to do as they had to change default NLS from iso8859-1 to something else.
Quoted 3 lines
But if you have checked that you are not mapping two values to the same thing (which would break the back-and-forth translation that smbfs does) I don't see how that patch can harm anything.
Yes, I checked it. After changing iso* all singlebyte encodings except cp874 contain unique mapping for all byte values (cp874 is unique, but some values are unmappable).
Thanks,
Petr Vandrovec [email protected]
Quoted 9 lines · Petr Vandrovec
Quoted 4 lines · Urban Widmark
On Tue, 7 May 2002, Petr Vandrovec wrote:
But if you have checked that you are not mapping two values to the same thing (which would break the back-and-forth translation that smbfs does) I don't see how that patch can harm anything.Yes, I checked it. After changing iso* all singlebyte encodings except cp874 contain unique mapping for all byte values (cp874 is unique, but some values are unmappable).
Wrong. The NLS tables do not guarantee unique mapping. So all fs which do "back-and-forth" translation are broken, the only encoding which really works is UTF-8.
We found out the hard way in ntfs. An example: take CP936 (GB2312).
Take a Unicode character between 0x4e00 and 0x9fa5, i.e. the CJK Ideograph range (yes we found examples using these characters on various (chinese?) websites).
Convert to gb2312 using NLS and then back to Unicode and you end up with a Unicode character in the range 0xF900-0xFA2D, i.e. the CJK Compatibility Ideographs.
Concrete example we ran into with ntfs, Unicode character 0x884C (a CJK Ideograph). Translates to gb2312 character sequence 0xD0, 0xD0, and then back to Unicode character 0xFA08 (a CJK Compatibility Ideograph).
I double checked the translation manually and I also checked the original translation tables on the microsoft website and this is indeed what happens. If you looks at the translation table there are several Unicode characters mapping to the gb2312 character sequence 0xD0, 0xD0, but obviously this only maps back to a single Unicode character.
Also if you lookup the Unicode character database 2.1 (I checked rev 2.1.8) from the Unicode Consortium it specifies this as correct:
[snip]
4E00;<CJK Ideograph, First>;Lo;0;L;;;;;N;;;;; 9FA5;<CJK Ideograph, Last>;Lo;0;L;;;;;N;;;;; [snip]
FA08;CJK COMPATIBILITY IDEOGRAPH-FA08;Lo;0;L;884C;;;;N;;;;; [snip]
This means that Unicode itself is not a one-to-one mapping. Apparently multiple characters have the same meanings... )))-:
I never imagined I would find something so braindamaged in Unicode but there you go!
Basically this means, at least for NTFS, but I think it is the same for all file systems, that on directory lookups, either we have to search the directory by just looking at EVERY directory entry and converting each to the current NLS and comparing for identity to the name being searched for or we have to use UTF-8 as that guarantees to preserve back-and-forth mappings one-to-one (I believe).
Doing a directory lookup where the whole directory is scanned linearly is incredibly slow and the overhead of having to convert every single directory entry to compare it every time a lookup() happens is very large, so I don't want to implement that on NTFS, so if anyone complains to me about their character translation not working properly they will just hear use UTF-8 and it will work.
But you will have to find UTF-8 fonts and user space support code in order to see the correct output displayed. Otherwise you just see random characters in your filenames... But at least It Works(TM).
Best regards,
Anton
Signature · 5 lines
"I've not lost my mind. It's backed up on tape somewhere." - Unknown -- Anton Altaparmakov <aia21 at cantab.net> (replace at with @) Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/