Skip to contentSkip to conversation

JFS default behavior (was: UTF-8 in file systems? xfs/extfs/etc.)

DISCUSSION Nicolas Mailhot 7 participants 12 messages Feb 12, 2004 – Feb 13, 2004

lore
Share & tools
lore
#0 lore raw

Not specifying the file name encoding (either per fs type, per partition or per filename) is plain dangerous. It is not a userspace problem - flash/hotplug disks move, users on the same system can have different locales and try to share files, a user can change his locale to another one (hear the screams of RH users forcibly converted to utf8 which had to fix years of storage which filenames were suddenly borked)

See also the sun zip encoding bug - everyone uses zip files in Java, zip authors thought a filename is "just a bunch of bytes" and didn't put filename encoding info in the zip format, and now java zip handling goes boom since numerous encodings are unicode-incompatible. It's slowly getting its way to the top-25 most reported java bugs.

(of course as usual US users/coders are not hit and do not feel concerned)

The only reason we got by with it so far is linux localisation was poor, and systems didn't scale high enough to permit high number of users per system (reducing locale collision risks)

The only reason we might get by in the future is everyone will be using utf8.

But that's not a reason not to fix the core problem - I don't want to spent hours fixing filenames next time someone comes up with a new encoding. Please put valid encoding info somewhere or declare filenames are utf-8 od utf-16 only - changing user locale should not corrupt old data.

Cheers,

Signature · 1 line
Nicolas Mailhot
#1 lore raw
Quoted 5 lines · you

But that's not a reason not to fix the core problem - I don't want to spent hours fixing filenames next time someone comes up with a new encoding. Please put valid encoding info somewhere or declare filenames are utf-8 od utf-16 only - changing user locale should not corrupt old data.

Yes!

-- robin

#2 lore raw
In reply to Nicolas Mailhot
Quoted 5 lines · Nicolas Mailhot

But that's not a reason not to fix the core problem - I don't want to spent hours fixing filenames next time someone comes up with a new encoding. Please put valid encoding info somewhere or declare filenames are utf-8 od utf-16 only - changing user locale should not corrupt old data.

If you attach encoding to names for a whole filesystem, you will get really unpleasant bugs including security holes because some names won't be writable, so the fs will either return error codes when those names are used, or silently alter the names.

-- Jamie

#3 lore raw
Quoted 11 lines · Jamie Lokier
Quoted 5 lines · Nicolas Mailhot

But that's not a reason not to fix the core problem - I don't want to spent hours fixing filenames next time someone comes up with a new encoding. Please put valid encoding info somewhere or declare filenames are utf-8 od utf-16 only - changing user locale should not corrupt old data.

If you attach encoding to names for a whole filesystem, you will get really unpleasant bugs including security holes because some names won't be writable, so the fs will either return error codes when those names are used, or silently alter the names.

Depends on how to handle those undecodeble file names. non-ascii filenames are probably a security issue (negative characters) with some apps. Making them inaccessible is definitely not ok. I proposed one version, although it might be a good idea to look at those file systems that handle the problem already so a uniform solution could be used that makes all filenames accessible regardless of which characters are used and doesn't cause unneccessary confusion as to what is the name.

-- robin

#4 lore raw
In reply to Jamie Lokier
Quoted 11 lines · Jamie Lokier
Quoted 5 lines · Nicolas Mailhot

But that's not a reason not to fix the core problem - I don't want to spent hours fixing filenames next time someone comes up with a new encoding. Please put valid encoding info somewhere or declare filenames are utf-8 od utf-16 only - changing user locale should not corrupt old data.

If you attach encoding to names for a whole filesystem, you will get really unpleasant bugs including security holes because some names won't be writable, so the fs will either return error codes when those names are used, or silently alter the names.

You can have security holes now just by tricking an app into reading files written by another app which disagreed on the locale.

And as for the filename problems :
- just mangle existing invalid filenames when a default encoding is agreed upon
- refuse to write new files with invalid filenames just like you would with the few names forbidden in ascii - apps will learn to cope.

Some convention is needed, expecting it to materialise without os enforcement is deluding oneself, getting a change like this in place will definitely be painful but the current situation is far from painless for a lot of people.

Regards,

Signature · 1 line
Nicolas Mailhot
#5 lore raw
Quoted 5 lines · Nicolas Mailhot

And as for the filename problems :
- just mangle existing invalid filenames when a default encoding is agreed upon
- refuse to write new files with invalid filenames just like you would with the few names forbidden in ascii - apps will learn to cope.

What names forbidden in ASCII?

#7 lore raw
Quoted 5 lines · On Fri, Feb 13, 2004 at 01:24:33PM -0500, [email protected] wrote:
Quoted 1 line · On Fri, 13 Feb 2004 18:15:42 GMT, [email protected] said:

What names forbidden in ASCII?

Anything with a / or a \0 in it. ;)

You try and pass something _without_ \0 in it to the kernel ;-)

#8 lore raw
Quoted 1 line · [email protected] wrote:

You try and pass something _without_ \0 in it to the kernel ;-)

:)

But seriously, even that is a security issue when someone requests a URL containing "%00", or some text contains a filename to operate on and the name contains \0.

For example, if I write a Perl regular expression to reject paths from the outside world containing "..": m{(?:/|^)\.\.(?:/|\z)}, it will fail to notice when given the path "..\0" that the kernel will treat it identically to "..". Potential security hole, depending on the context.

-- Jamie

#9 lore raw
In reply to [email protected]
Quoted 8 lines · On Fri, 13 Feb 2004 [email protected] wrote:
Quoted 5 lines · Nicolas Mailhot

And as for the filename problems :
- just mangle existing invalid filenames when a default encoding is agreed upon
- refuse to write new files with invalid filenames just like you would with the few names forbidden in ascii - apps will learn to cope.

What names forbidden in ASCII?

I think that all ASCII characters below 0x20 are forbidden in Unix file-names and others shown in the reference cited and "disapproved".

http://www.med.nyu.edu/rcr/rcr/nyu_vms/unixfileanddirectorynames.htm

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#10 lore raw

Re: JFS default behavior

Quoted 2 lines · Richard B. Johnson

I think that all ASCII characters below 0x20 are forbidden in Unix file-names

Not true. Filenames in Unix are defined as

3.169 Filename
A name consisting of 1 to {NAME_MAX} bytes used to name a file. The characters composing the name may be selected from the set of all character values excluding the slash character and the null byte. The filenames dot and dot-dot have special meaning. A filename is
sometimes referred to as a pathname component .

Only NUL and / are special.

Signature · 1 line
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖
#11 lore raw
In reply to Richard B. Johnson
Quoted 9 lines · Richard B. Johnson
Quoted 7 lines · On Fri, 13 Feb 2004 [email protected] wrote:
Quoted 5 lines · Nicolas Mailhot

And as for the filename problems :
- just mangle existing invalid filenames when a default encoding is agreed upon
- refuse to write new files with invalid filenames just like you would with the few names forbidden in ascii - apps will learn to cope.

Quoted 7 lines
Quoted 1 line

What names forbidden in ASCII?

I think that all ASCII characters below 0x20 are forbidden in Unix file-names and others shown in the reference cited and "disapproved".

http://www.med.nyu.edu/rcr/rcr/nyu_vms/unixfileanddirectorynames.htm

That's not really a formal definition of what's allowed. It's a recommendation for users on how to avoid detecting applications that cannot handle all file names, i.e. buggy applications. Try

	touch "$(/bin/ls -1|head)"

and you will find apps that can handle the nice filename and those that cannot. I'm definitely not endorsing them and it would probably be wise to implement a system policy that allows administrators to ban such names as they represent security holes and all sorts of problems.

Some filesystems forbid these names, but unix doesn't.

-- robin