Skip to contentSkip to conversation

Long file names in VFAT broken with iocharset=utf8

DISCUSSION Andrey Borzenkov 5 participants 11 messages May 7, 2007 – May 9, 2007

lore
Share & tools
lore
#0 lore raw

This was posted in one of Russian forums. It was not possible to archive (under Linux, using tar) vfat directory where files had long Russian names (really long - over 150 - 170 characters) - tar returned stat failure. When looking with plain ls, file names appeared truncated.

Now looking at current (2.6.21) fat driver, __fat_readdir allocates large enough buffer (PAGE_SIZE-522) for UTF-8 name; but for iocharset=utf8 it calls uni16_to_x8() which artificially limits length of UTF-8 name to 256 ... which is obviously not enough for long UTF-8 Russian string (2 bytes per character) not to mention the - theoretical - general case of 6 bytes UTF-8 characters.

Similar problem has apparently vfat_lookup()->...->fat_search_long() call chain. Except this appears to be broken even in case of "utf8", because fat_search_long allocates fixed 256 bytes buffer for UTF-8 name.

Am I off track here?

-andrey

#1 lore raw

Hi Andrey!

Quoted 11 lines · Andrey Borzenkov

This was posted in one of Russian forums. It was not possible to archive (under Linux, using tar) vfat directory where files had long Russian names (really long - over 150 - 170 characters) - tar returned stat failure. When looking with plain ls, file names appeared truncated.

Now looking at current (2.6.21) fat driver, __fat_readdir allocates large enough buffer (PAGE_SIZE-522) for UTF-8 name; but for iocharset=utf8 it calls uni16_to_x8() which artificially limits length of UTF-8 name to 256 ... which is obviously not enough for long UTF-8 Russian string (2 bytes per character) not to mention the - theoretical - general case of 6 bytes UTF-8 characters.

Quoted 6 lines

Similar problem has apparently vfat_lookup()->...->fat_search_long() call chain. Except this appears to be broken even in case of "utf8", because fat_search_long allocates fixed 256 bytes buffer for UTF-8 name.

Am I off track here?

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard-coded value of 256 somewhere inside the kernel smells like a bug.

Ciao,
Roland

--
TU Muenchen, Physik-Department E18, James-Franck-Str., 85748 Garching Telefon 089/289-12575; Telefax 089/289-12570
--
CERN office: 892-1-D23 phone: +41 22 7676540 mobile: +41 76 487 4482 --
Any society that would give up a little liberty to gain a little security will deserve neither and lose both. - Benjamin Franklin -----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P+++ L+++ E(+) W+ !N K- w--- M+ !V Y+ PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++
------END GEEK CODE BLOCK------

#2 lore raw
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is PATH_MAX and there is NAME_MAX, and only the latter (which is 260 for vfat) matters here.

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#3 lore raw
Quoted 10 lines · Andreas Schwab
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is PATH_MAX and there is NAME_MAX, and only the latter (which is 260 for vfat) matters here.

Do you imply that Linux is unable to represent full VFAT names (255 UCS2 charaters) by design? Hmm ... testing ... looks like it,

touch $(perl -e 'print "ф" x 200;') -> Name too long.

So what do we do with long VFAT names? As initial post shows, they do exist.

#4 lore raw
Quoted 8 lines · Andrey Borzenkov
Quoted 7 lines · Andreas Schwab
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

Quoted 6 lines
Quoted 3 lines

There is PATH_MAX and there is NAME_MAX, and only the latter (which is 260 for vfat) matters here.

Do you imply that Linux is unable to represent full VFAT names (255 UCS2 charaters) by design? Hmm ... testing ... looks like it,

Yes. But I think it's not design, and it should be fixed.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#5 lore raw
In reply to Roland Kuhn

Hi,

Quoted 25 lines · Roland Kuhn
Quoted 25 lines

This was posted in one of Russian forums. It was not possible to archive
(under Linux, using tar) vfat directory where files had long Russian names
(really long - over 150 - 170 characters) - tar returned stat failure. When
looking with plain ls, file names appeared truncated.

Now looking at current (2.6.21) fat driver, __fat_readdir allocates large
enough buffer (PAGE_SIZE-522) for UTF-8 name; but for iocharset=utf8 it calls
uni16_to_x8() which artificially limits length of UTF-8 name to 256 ... which
is obviously not enough for long UTF-8 Russian string (2 bytes per character)
not to mention the - theoretical - general case of 6 bytes UTF-8 characters.

Similar problem has apparently vfat_lookup()->...->fat_search_long () call
chain. Except this appears to be broken even in case of "utf8", because
fat_search_long allocates fixed 256 bytes buffer for UTF-8 name.

Quoted 7 lines
Quoted 2 lines

Am I off track here?

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is a nasty issue here. FAT is limited by 255 unicode chars or so. So, we would need to count number of unicode chars of filename.

That's not implemented currently.

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#6 lore raw
Quoted 11 lines · OGAWA Hirofumi
Quoted 6 lines
Quoted 1 line

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is a nasty issue here. FAT is limited by 255 unicode chars or so. So, we would need to count number of unicode chars of filename.

That's not implemented currently.

Note also there is PATH_MAX and NAME_MAX; the latter is 255 I believe.

POSIX allows NAME_MAX to vary on a filesystem by filesystem basis, although that is not currently implemented in Linux.

-hpa

#7 lore raw
In reply to OGAWA Hirofumi

Hi!

Quoted 10 lines · OGAWA Hirofumi
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is a nasty issue here. FAT is limited by 255 unicode chars or so. So, we would need to count number of unicode chars of filename.

No, we don't. At least not when looking at the POSIX spec, which explicitly mentions _bytes_ and _not_ unicode characters. So, to be on the safe side, FAT filesystems would need to support a NAME_MAX of roughly 6*255+3=1533 bytes (not to mention the hassles of forbidden sequences, etc.; do we need to count zero-width characters?) and report it through pathconf() to userspace, then userspace could do with that whatever it liked.

What happened to: "file names are just sequences of octets, excluding '/' and NUL"? Adding unicode parsing to the kernel is completely useless _and_ a big trouble maker.

Ciao,
Roland

--
TU Muenchen, Physik-Department E18, James-Franck-Str., 85748 Garching Telefon 089/289-12575; Telefax 089/289-12570
--
CERN office: 892-1-D23 phone: +41 22 7676540 mobile: +41 76 487 4482 --
Any society that would give up a little liberty to gain a little security will deserve neither and lose both. - Benjamin Franklin -----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P+++ L+++ E(+) W+ !N K- w--- M+ !V Y+ PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++
------END GEEK CODE BLOCK------

#8 lore raw
Quoted 19 lines · Roland Kuhn
Quoted 11 lines
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is a nasty issue here. FAT is limited by 255 unicode chars or so.
So, we would need to count number of unicode chars of filename.

No, we don't. At least not when looking at the POSIX spec, which explicitly mentions _bytes_ and _not_ unicode characters. So, to be on the safe side, FAT filesystems would need to support a NAME_MAX of roughly 6*255+3=1533 bytes (not to mention the hassles of forbidden sequences, etc.; do we need to count zero-width characters?) and report it through pathconf() to userspace, then userspace could do with that whatever it liked.

Quoted 3 lines

What happened to: "file names are just sequences of octets, excluding '/' and UL"? Adding unicode parsing to the kernel is completely useless _and_ a big trouble maker.

The UCS2 in FAT is just on-disk format of the filename. So...

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#9 lore raw
In reply to Roland Kuhn
Quoted 9 lines · Roland Kuhn

No, we don't. At least not when looking at the POSIX spec, which explicitly mentions _bytes_ and _not_ unicode characters. So, to be on the safe side, FAT filesystems would need to support a NAME_MAX of roughly 6*255+3=1533 bytes (not to mention the hassles of forbidden sequences, etc.; do we need to count zero-width characters?) and report it through pathconf() to userspace, then userspace could do with that whatever it liked.

Quoted 4 lines

What happened to: "file names are just sequences of octets, excluding '/' and NUL"? Adding unicode parsing to the kernel is completely useless _and_ a big trouble maker.

"Filenames are just octets" have never applied to alien filesystems like VFAT.

-hpa

#10 lore raw
In reply to Roland Kuhn
Quoted 14 lines · Roland Kuhn

Hi!

Quoted 10 lines · OGAWA Hirofumi
Quoted 5 lines · Roland Kuhn

PATH_MAX specifically counts _bytes_ not characters, so UTF-8 does not matter. ISTR that PATH_MAX was 256 at some point, but I just quickly grepped /usr/include and found various mention of 4096, so where's the central repository for this configuration item? A hard- coded value of 256 somewhere inside the kernel smells like a bug.

There is a nasty issue here. FAT is limited by 255 unicode chars or so.
So, we would need to count number of unicode chars of filename.

Quoted 5 lines

No, we don't. At least not when looking at the POSIX spec, which explicitly mentions _bytes_ and _not_ unicode characters. So, to be on the safe side, FAT filesystems would need to support a NAME_MAX of roughly 6*255+3=1533 bytes (not to mention the hassles of forbidden sequences, etc.; do we need to count zero-width characters?)

How is this issue related to character *width* at all?

Quoted 8 lines

and
report it through pathconf() to userspace, then userspace could do with that whatever it liked.

What happened to: "file names are just sequences of octets, excluding '/' and NUL"? Adding unicode parsing to the kernel is completely useless _and_ a big trouble maker.

Who speaks about unicode parsing? UCS2 - UTF-8 transformation does and requires no parsing; this is simply conversion between on-disk and in-kernel representation (like endian conversion). Anyway we are doing it now already; how support for larger name length limit changes it?

-andrey