Skip to contentSkip to conversation

Racy NLS behaviour in FAT (and possible other fs)

DISCUSSION Pierre Ossman 2 participants 5 messages Feb 19, 2007 – Feb 20, 2007

lore
Share & tools
lore
#0 lore raw

Hi,

I'm experiencing a rather odd behaviour with the character set conversion. If I mount a vfat fs with utf8 and then create a file with invalid utf-8 sequences, the file will briefly exist with these invalid sequences, then quickly convert to a stripped version.

I haven't found an easy way to catch the race, but if I have nautilus open it tends to catch it now and then (I get a file name with "<?>" replacing each bad byte).

The race also seems to corrupt the in-memory state of the fs now and then. I managed to create a file where "ls" shows "?" for most fields. Data seemed to have made it to disk ok though (fsck didn't complain and a remount showed everything as it should be).

Third, there seems to be a problem with not all syscalls being subjected to the NLS transformation. Example:

$ echo foo > baråäö.txt
$ ls
foo.txt
$ echo foo > baråäö.txt
bash: baråäö.txt: File exists

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#1 lore raw
Quoted 1 line · Pierre Ossman

Hi,

Hi,

Quoted 4 lines

I'm experiencing a rather odd behaviour with the character set conversion. If I mount a vfat fs with utf8 and then create a file with invalid utf-8 sequences, the file will briefly exist with these invalid sequences, then quickly convert to a stripped version.

Yes. utf8 support is broken, and it will fail to convert letter case on many case. And it's why that is not recommended.

Quoted 3 lines

I haven't found an easy way to catch the race, but if I have nautilus open it tends to catch it now and then (I get a file name with "<?>" replacing each bad byte).

`?' is meaning the character can't be converted.

Quoted 4 lines

The race also seems to corrupt the in-memory state of the fs now and then. I managed to create a file where "ls" shows "?" for most fields. Data seemed to have made it to disk ok though (fsck didn't complain and a remount showed everything as it should be).

It seems readdir() was success, but stat() was failed. And the dosfsck doesn't check filename correctly.

Those problem seem to be relateing to poor utf8 support...

Quoted 8 lines

Third, there seems to be a problem with not all syscalls being subjected to the NLS transformation. Example:

$ echo foo > baråäö.txt
$ ls
foo.txt
$ echo foo > baråäö.txt
bash: baråäö.txt: File exists
Signature · 1 line
OGAWA Hirofumi <[email protected]>
#2 lore raw
Quoted 10 lines · OGAWA Hirofumi
Quoted 5 lines

I'm experiencing a rather odd behaviour with the character set conversion. If I mount a vfat fs with utf8 and then create a file with invalid utf-8 sequences, the file will briefly exist with these invalid sequences, then quickly convert to a stripped version.

Yes. utf8 support is broken, and it will fail to convert letter case on many case. And it's why that is not recommended.

Is there any ongoing work to fix this? UTF-8 is standard on more or less every distribution these days.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#3 lore raw
Quoted 11 lines · Pierre Ossman
Quoted 7 lines · OGAWA Hirofumi
Quoted 4 lines

I'm experiencing a rather odd behaviour with the character set conversion. If I mount a vfat fs with utf8 and then create a file with invalid utf-8 sequences, the file will briefly exist with these invalid sequences, then quickly convert to a stripped version.

Yes. utf8 support is broken, and it will fail to convert letter case on many case. And it's why that is not recommended.

Is there any ongoing work to fix this? UTF-8 is standard on more or less every distribution these days.

Yes. But sorry, I don't have any plan and time to fix it now.

If you are using "iocharset=utf8" now, "codepage=cp???,iocharset=xxx,utf8" might help a bit.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#4 lore raw
Quoted 2 lines · OGAWA Hirofumi

Yes. But sorry, I don't have any plan and time to fix it now.

I know the feeling. :)

I just wanted to know where things stand as I have little insight into vfat development.

Quoted 3 lines

If you are using "iocharset=utf8" now, "codepage=cp???,iocharset=xxx,utf8" might help a bit.

These tests was with the "utf8" option.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org