git on MacOSX and files with decomposed utf-8 file names

Hi,

I have some files like "Lüftung.txt" in my repository. The strange thing is that I can pull / add / commit / push those files without problem but git-status always complains that thoes files are untraced (but not missing). My assumption is that it's a problem with the way MacOSX stores the file names (decomposed UTF-8). So something like "Lüftung.txt" becomes "Lüftung.txt".

It seems that git-status does two things:
1. Find files under version control (i.e. search for missing files)
2. Find files not under version control (i.e. search for untracked files)

I guess that the first look-up succeeds because MacOS X converts composed UTF-8 to decomposed UTF-8 when searching for a file. But it seems that the second look-up takes the file names as-is (decomposed) without converting them to composed UTF-8.

Is there an easy way to fix this behaviour? It's really annoying to see all those "untracked" files that are already under version control when executing a git-status.

Regards,
Mark

Hi,

This is a known problem. Unfortunately, noone has implemented a fix, although if you're serious about it, I can point you to threads where it has been hinted how to solve the issue.

FWIW the issue is that Mac OS X decides that it knows better how to encode your filename than you could yourself.

Ciao,
Dscho

More like, Mac OS X has standardized on Unicode and the rest of the world hasn't caught up yet. Git is the only tool I've ever heard of that has a problem with OS X using Unicode.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

As far as I know, Subversion has basically exactly the same problem, and any time you consume/produce files on Mac OS X that are be consumed/produced on other platforms you will run into this kind of issue, with any software.

Tell Mac OS X to write a file with "ó" in the file name ("\xc3\xb3" in UTF-8), and it will "normalize" it prior to writing by converting it into a decomposed form (that is, ASCII "o" followed by "\xcc\x81", or "combining acute accent"). So they're both valid Unicode, both valid UTF-8, and they encode exactly the same characters but the byte stream is different.

If you only work on Mac OS X then this will never be a problem because all the files you create and therefore all the files you add to your Git repository will have their names in decomposed UTF-8. But when you start cloning repositories containing files added on other systems, systems which might use precomposed rather than decomposed UTF-8 then you'll run into exactly this kind of problem. The git.git repo has one such file itself (gitweb/test/Märchen, if I remember correctly, which Git reports as untracked).

Now, Mac OS X's behaviour is not entirely "insane" as some would claim; there is indeed a rationale behind it even if you don't agree with it, but it *does* produce some unfortunate teething problems for people wanting to use Mac OS X in a cross-platform environment.

Here are some Apple docs on the subject:

http://developer.apple.com/qa/qa2001/qa1173.html

http://developer.apple.com/qa/qa2001/qa1235.html

I personally wish that UTF-8 didn't allow different normalization forms; then this kind of problem wouldn't arise. But it has arisen and we have to live with it. Some workarounds have been proposed for Git, but I haven't seen any convincing proposals yet.

Cheers,
Wincent

Hi,

No. That's not at all the problem. Mac OS X insists on storing _another_ encoding of your filename. Both are UTF-8. Both encode the _same_ string. Yet they are different, bytewise. For no good reason.

Stop spreading FUD. Git can handle Unicode just fine. In fact, Git does not _care_ how the filename is encoded, it _respects_ the user's choice, not only of the encoding _type_, but the _encoding_, too.

Okay?

Hth,
Dscho

"FUD" is a bit strong, don't you think? HFS+ is the way it is and it would be nice if Git could deal with it.

The problem is that HFS+ normalizes filenames to avoid multiple files that appear to have the same name (eg "M<A WITH UMLAUT>rchen" vs "Ma<UMLAUT MODIFIER>rchen", in gitweb/test). This is sort of like case sensitivity, but filenames are normalized when a file is _created_. Git, not unreasonably, expects a file to keep the name it was created with.

As far as I can tell, as long as you add all your internationally becharactered files to git from an HFS+ file system using a gui or command-line completion, you'll be okay; trouble starts when you check in a file with the composed form of a character, by typing the name on the command line (I'm not sure about this one) or committing on another OS. Git will store the filename in composed form, but the Mac's filesystem will decompose the filename when you check the file out.

The result looks like this:

vredefort:[git]% git status
# On branch master
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to track)

(this is directly after checking out git.git @ v1.5.4-rc3)

There are two things to note here. One is that Git thinks that there is a new file called "gitweb/test/Märchen" (decomposed) when it's "really" just the same "gitweb/test/Märchen" (precomposed) that's in the repository. The other is that git _thinks_ that the "gitweb/test/ Märchen" (precomposed) it's expecting is still there, because the filesystem, when asked for "gitweb/test/Märchen" in any form will return the file "gitweb/test/Märchen" (decomposed).

Trying to check out the "next" branch at this point is a pain since next's "Märchen" would overwrite the untracked "Märchen".

I can't provide links to any previous discussions about this, but here's Apple's Technical Q&A on the subject:

http://developer.apple.com/qa/qa2001/qa1235.html

Finding a sane way of allowing git to handle this behaviour is left as an exercise for the reader.

Eyvind Bernhardsen

To be more exact encoding used to _create_ file differs from encoding returned when _reading directory_...

...which means that sequence of bytes differ. And Git by design is (both for filenames and for blob contents) encoding agnostic.

HFS+ is just _stupid_. And unfortunately Git doesn't support stupid filesystems (e.g. case insensitive filesystems) well.

Jakub Narebski
Poland
ShadeHawk on #git

There's two different ways to do filesystem encodings. One is to have the fs simply not care about encoding, which is what the linux world seems to prefer. Sure, this is great in that what you create the file with is what you get back, but on the other hand, given an arbitrary non-ASCII file on disk, you have absolutely no idea what the encoding should be and you can't display it without making assumptions (yes you can use heuristics, but you're still making assumptions). Filesystems like HFS+ that standardize the encoding, on the other hand, make it such that you always know what the encoding of a file should be, so you can always display and use the filename intelligently. It also means it plays much nicer in a non-ASCII world, since you don't have to worry about different normalizations of a given string referring to different files (it's one thing to be case-sensitive, but claiming that "föo" and "föo" are different files just because one uses a composed character and the other doesn't is extremely user-unfriendly). On the other hand, what you create the file with may not be what you read back later, since the name has been standardized. It's hard to say one is better than the other, they're just different ways of doing it. However, I have noticed that everybody who's voiced an opinion on this list in favor of the encoding-agnostic approach seem to be unwilling to accept that any other approach might have validity, to the extent of calling an OS/filesystem that does things different stupid or insane. This strikes me as extremely elitist and risks alienating what I expect to be a fast-growing group of users (i.e. OS X users).

I'm willing to give Linus a free pass on calling other OS's stupid and insane, as I don't think Linux would exist as it does today without his strong opinions, but I don't think this should give carte blanche to the rest of the community for this inflammatory behavior.

I should note that I'm only taking the time to discuss this because, despite the fact that I'm new to git, I really like it and I want it to work better. And one area that it has a problem with is the de-facto filesystem on my OS of choice. However, attempts to discuss the problem invariable end up with multiple people calling my OS stupid and insane simply because it differs in a particular design decision. This is not a good way to build a community or to build a better product, and I hope it can be improved.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

By the way, calling HFS+ stupid, or rather calling at least two different normalizations of UTF-8 (two different encodings) used for writing and reading filenames stupid is wrong _for me_. I have quoted Linus here, when I think I should use other description.

For me it looks like a layering violation... but my knowledge about filesystem is cluse to nil. IMHO it is VFS and libc which should do the translating.

But using one encoding to create file, and another when reding filenames is strange. It is IMHO better to simply refuse creating filenames which are outside chosen encoding / normalization. But having different encodings used for reading and writing on the level of filesystem access (not on level of UI) is strange.

First, it is Git philosophy and very core of design to be encoding agnostic (to be "content tracker"). Second, using the same sequence of bytes on filesystem, in the index, and in 'tree' objects ensures good performance... this is something to think about if you want to add patches which would deal with HFS+ API/UI quirks.

[cut]

Jakub Narebski
Poland

It's not using different encodings, it's all Unicode. However, it accepts different normalization variants of Unicode, since it can read them all and it would be folly to require everybody to conform to its own special internal variant. But it does have to normalize them, otherwise how would it detect the same filename using different normalizations? Also, it may seem strange to have different names between reading and writing, but that's only if you think of the name as a sequence of bytes - when treated as a sequence of characters, you get the same result. In other words, you're used to filenames as bytes, HFS+ treats filenames as strings.

Sure, it makes sense from a performance perspective, but it causes problems with HFS+ and any other filesystem that behaves the same way. In the previous discussion about case-sensitivity, somebody suggested using a lookup table to map between git's internal representation and the name the filesystem returns, which seems like a decent idea and one that could be enabled with a config parameter to avoid penalizing repos on other filesystems. But I don't know enough about the internals of git to even think of trying to implement it myself.

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

That's a singularly *stupid* argument.

Here, let me rephrase that same idiotic argument:

"But it does have to uppercase them, otherwise how would it detect the
same filename using different cases?"

..and if you don't see how that's *exactly* the same argument, you really are stupid.

The fact is, normalization is wrong.

It's wrong when you normalize upper/lower case (no, the word "Polish" is not the same as "polish"), and it's equally wrong when you normalize for "looks similar".

No. HFS+ treats users as idiots and thinks that it should "fix" the filename for them. And it causes problems.

It causes problems for exactly the same reasons case-independence causes problems, because it's EXACTLY THE SAME ISSUE. People may think that "but they are the same", but they aren't. Case matters. And so does "single character" vs "two character overlay".

Does it always matter? Hell no. But the problem with a filesystem that thinks it knows better is that when it *sometimes* matters, the filesystem simply DOES THE WRONG THING.

Can't you understand that?

Linus

You're right, it doesn't actually have to store the normalized form. And yes, it's possible to compare without normalizing them. Admittedly, I don't know much about the implementation details of unicode, but I would assume that the easiest way to compare two strings is to normalize them first. But in the case of the filesystem, normalization actually is important if you're thinking about filenames in terms of characters rather than bytes. When I feed the filesystem a given unicode string, it has to find the file I'm talking about - should it do a relatively expensive unicode-sensitive comparison of all the filenames with the one I gave it, or should it just normalize all names and do the much cheaper lookup that way? I don't know about you, but I'd prefer to let my filesystem normalize the name and run faster.

There's a difference between "looks similar" as in "Polish" vs "polish", and actually is the same string as in "Ma<UMLAUT MODIFIER>rchen" vs "M<A WITH UMLAUT>rchen". Capitalization has a valid semantic meaning, normalization doesn't. The only way to argue that normalization is wrong is by providing a good reason to preserve the exact byte sequence, and so far the only reason I've seen is to help git. Applications in general don't care one whit about the byte sequence of the filename, they care about the underlying file the name represents. Additionally, it would be a terrible experience for a user to enter "Märchen" and have the application say "sorry, I can't find this file" simply because the application used decomposed characters and the filename used composed characters. Unless the user is knowledgeable about the OS, filesystems, and unicode, they wouldn't have a hope of figuring out what the problem was.

How do you figure? When I type "Märchen", I'm typing a string, not a byte sequence. I have no control over the normalization of the characters. Therefore, depending on what program I'm typing the name in, I might use the same normalization as the filename, or I might miss. It's completely out of my control. This is why the filesystem has to step in and say "You composed that character differently, but I know you were trying to specify this file".

There are valid reasons for case to matter, but what reason is there for "single character" vs" two character overlay" to matter in filenames? They're different representations of the exact same string, and that's what a filename is - a string.

It seems like your arguments stem from the assumption that the user cares about the byte sequence that represents the filename, which is wrong. The user has no idea what the byte sequence is - the user cares about the string. Normalization is meant to help computers, not users, and claiming that different normalizations of the same string produces different meaningful strings is complete bunk.

If you were to have two different files on your system, both of them called "Märchen", but one precomposed and one decomposed, how would you specify which one you wanted? Unless Linux has a special text input system which gives the user control over the normalization of their typed characters, you'd have to write out the UTF-8 bytes manually.

I just don't understand this insistence on treating the specific byte sequence that makes up the filename as significant.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

That simply isn't true.

Normalization actually has real semantic meaning. If it didn't, there would never ever be a reason why you'd use the non-normalized form in the first place.

Others have argued the exact same thing for capitalization. "A" is the same letter as "a". Except there is a distinction.

The same is true of "a<UMLAUT MODIFIER>" and "<a WITH UMLAUT>". Yes, it's the same "chacter" in either case. Except when there is a distinction.

And there *are* cases where there are distinctions. Especially inside computers. For one thing, you may not be talking about "characters on screen", but you may be talking about "key sequences". And suddenly "a<UMLAUT MODIFIER>" is a two-key sequence, and "<a WITH UMLAUT>" is a single-key sequence, and THEY ARE DIFFERENT.

See?

"a" and "A" are the same letter. But sometimes case matters.

Multi-character UTF-8 sequences may be the same character. But sometimes the sequence matters.

Same exact thing.

Git doesn't care. Just use the *same* sequence everywhere. Make sure something doesn't change it. Because if something changes it, git will track it.

Pure and utter garbage.

What you are describing is an *input method* issue, not a filesystem issue.

The fact that you think this has anything what-so-ever to do with filesystems, I cannot understand.

Here's an example: I can type Märchen two different ways on my keyboard: I can press the 'ä' key (yes, I have one, I have a Swedish keyboard), or I could press the '¨' key and the 'a' key.

See: I get 'ä' and 'ä' respectively.

And as I send this email off, those characters never *ever* got written as filenames to any filesystem. But they *did* get written as part of text-files to the disk using "write()", yes.

And according to your *insane* logic, that write() call should have converted them to the same representation, no?

Hell no! That conversion has absolutely nothing to do with the filesystem. It's done at a totally different layer that actually knows what it is doing, and turned them both into \xc3\xa4 (and then, the email client probably will turn this into Latin1, and send it out as a single-byte '\xe4' character).

See? Putting the conversion in the filesystem IS INSANE. You wouldn't make the filesystem convert the characters in the data stream (because it would cause strange data conversion issues) AND FOR EXACTLY THE SAME REASON it shouldn't do it for filenames either!

And your claim that "you have no control over the normalization of characters" is simply insane. Of course you have. It's just not supposed to be at the filesystem level - whether it's a write() call or a creat() call!

Linus

Actually, there is no good reason for non-normalized forms (deficient software not able to deal with some of the normalized forms is not a good reason: such software should be fixed).

It is just that the file system is a rather quirky place for enforcing the normalization. One should not be able to get unnormalized forms created easily in the first place, be it command line or script.

No. Input methods are not the same as their resulting string. I can even produce some ASCII characters on my keyboard in more than one way and would not expect them to lead to different codes.

How nice. We are actually in agreement here.

Yup. But that does not mean that normalization is a bad idea. It is just that the filesystem is not the right place for it.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

I'd actually agree, and it then boils down to the second sane choice I gave earlier:

- don't accept data you don't like

if you don't like non-normalized names, don't create them. That's fine.

But don't go normalizing them behind the users back.

Oh, absolutely. You can - and often should - normalize in the application (or have libraries to do it for you).

Not silently and behind peoples backs.

Linus

My understanding is that normalization is there to help the computer. That doesn't give it any semantic meaning, because all normal forms of a given string still represent the exact same string to the user.

The argument for case insensitivity is different than the argument for normalization. I certainly hope you understand why they are different arguments, or there's really no point in going further.

You're right, sometimes the sequence matters. As in key sequences. But we're not talking about key sequences, we're talking about strings. Just because it matters sometimes doesn't mean it matters all the time.

And how am I supposed to use the same sequence everywhere? When I type "Märchen", I don't know which form I'm typing, nor should I. It's not something that I, as a user, should have to know. Especially if I pass this name through various other utilities before using it - I have no idea if another utility is going to end up normalizing the name, and it shouldn't matter, as they are equivalent strings.

On a US keyboard I only have one way of typing ä, and I have no idea whether it ends up precomposed or decomposed in the resulting byte stream. And I don't care. Because I'm typing characters, not bytes. I could be typing in a file in ISO-Latin-1 and I still wouldn't care, because it looks the same to me. If my filesystem did make a distinction between the normal forms, and I see that I have a file named "Märchen", how am I supposed to type that at my keyboard? I don't know which normal form it's using.

The fact that you think the normalization of the string matters, I don't understand.

What a fabulous straw man argument you just put together. I hope you don't need me to point out why this argument is fundamentally flawed.

I'm speaking as a user, and as such, I shouldn't even have to know that it's possible to write the same character in multiple different ways. As a user, HFS+ behaves exactly the way I want it to. You were talking earlier about not messing with the "user data", but what is the "user data"? It's the string, not the byte sequence. That's all I care about - the string. That's all the OS cares about, that's all any application I use cares about, and that's all git should care about.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

THAT IS NOT TRUE!

How the hell does the computer know what the string means?

Hint: it does not.

The fact is, the user may use a non-normalized string on purpose. It's not your place to say that the user is wrong. Your "undestanding" is simply wrong. Two strings are *different* if they are [un]normalized differently.

Really.

The exact same way the word Polish and polish are different, just because they are capitalized differently.

You do not understand.

In *order* to do case-insensitivity, you generally need to normalize (and do other things too - normalization is just *one* of the things you need to do).

So if you are a case-insensitive filesystem, then normalization is sane.

But if you aren't, then there is no reason to normalize.

You define "string" to be something totally made-up.

In your world "string" means "normalized". BUT IT'S NOT TRUE!

You define normalization to be a property of strings, without any actual backing for why that would be.

The fact is, *looks the same* is very very different from *is the same*.

But you seem to be too stupid to undestand the differce.

Linus

Hi,

Actually, no. Even an case-challenged filesystem should keep the _original_ name around, if only for the exact same argument you used
earlier: if the user chooses to capitalise some letters, but not others, it is not the filesystem's place to "correct" that.

Ciao,
Dscho

Hi,

For the record, HFS+ is case-insensitive but case-preserving so I believe they keep the original filename around. I don't have the spec in front of me, but from memory I believe that this is what they do.

But I think that focusing on HFS+ is loosing sight of the real problem. It's not about encoding at the filesystem, but encoding inside the git structures.

Best regards,

Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [email protected]
Use XMPP!

Hi,

For the record, that's only the default setting. AFAIK you can configure it to care about case, too.

Also for the record, the whole thread was about HFS+ _not_ keeping the original filename around, but _only_ a normalised version of it.

So far I have not seen anyone talking _seriously_ about this issue. Only a few shouts "you should support", and a few shouts back "I don't care about insane filesystems".

Therefore, I fully agree with you that we're losing sight of the real problem.

Ciao,
Dscho

You're right. The normalization only really needs to happen as part of the name comparison itself.

Linus

The thing is, you seem to argue that what OS X does helps you as the user.

But you are arguing based on incorrect assumptions.

First off, we've had years and years and years of usage of non-corrupting filesystems (pretty much every UNIX OS around since day 1, and many other OS's too), and it's simply not true that it's a problem. You see the filename in the file dialog, and you open it, and you're done. OS X isn't any "easier" in this regard.

In fact, this whole thread comes from the fact that the OS X choice that you *think* is easier, is in fact not easier at all. It's not easier for the user, it's not easier for the application programmer, and the really sad part is that it's very much *not* easier for OS X itself either (ie they had to literally write extra code with nasty tables to do it, and it really does hurt them in performance and complexity).

And _that_ is why the OS X situation is so sad. Apple literally added extra code to make things slower and more complex *and* harder to use reliably.

Does it show up in normal behaviour? Of course not. You'd probably never see it in real life outside of test-suites. People simply don't even tend to use filenames outside of US-ASCII, and when they do use them, input methods really *do* tend to do the normalization for you.

But when it comes to automation (which is what computers are all about), the OS X choice is literally the wrong one. And there's no _upside_. It's all downside. Which is why it's so stupid.

I bet it only exists because OS X engineers didn't really even think about it, and they just assumed that "normalization is helpful". They took your stance - thinking it was worth it, without ever really thinking it through.

Linus

I believe it exists because HFS+ was created at a time when the Mac was moving from a multi-encoding world (which was a nightmare) to a Unicode world and they wanted to remove ambiguity in filenames. But I wasn't around when they made this decision so this is just a guess.

-Kevin Ballard

I do agree. And I think starting out case-insensitive (something they must really hate by now) also made it less of an issue. When you're case-insensitive, the issues with any UTF-8 normalization are simply swamped by all the issues of case, so you probably don't even think about it very much.

The big problem with any name rewriting is that I can open file 'xyz', and I literally have a very hard time knowing whether that file I know I opened and created has anything to do with the file 'Xyz' that I see when I do a readdir().

Are they the same? Maybe. But it's literally hard to tell on OS X. I can do an fstat() on my file descriptor and on the directory entry, and if they get the same d_ino they *probably are the same entry, but even then it actually could have been a hardlink (and my 'xyz' is really *another* name for it entirely, and the filesystem is actually case-sensitive and 'Xyz' was a *different* name that somebody else did!).

See? If you're creating a content tracker, these kinds of issues are not "idle chatter". It's really *really* important. Was that file the one I was told to track? Or was it a temporary file that was just hardlinked?

This is why case-insensitivity is so hard: you have a very real "aliasing" on the filesystem level, where all those really *different* pathnames end up being the same thing.

And all the same issues show up with utf-8 rewriting, so if you normalize utf-8 names, you actually end up having almost all the same problems that a case-insensitive filesystem has. They're just much rarer in practice, so you just won't hit them as often - but when you do, they are equally painful!

(In fact, they can be a whole lot *more* painful, because now they are really rare, and really confusing when they happen!)

But if you come from a case-insensitive background, all the UTF-8 rewriting really looks like such a small problem compared to all the horrid problems that you had with different locales and cases, so I suspect they didn't even realize what a big mistake they did!

Linus

Those of us who grew up on a case-insensitive filesystem don't find there to be any problem with it. I can count on one hand the number of times I've run into a problem caused by a case-insensitive filesystem. That number is 1. And that 1 time is when git screwed up trying to track CS4536 and cs4536 in the same directory (see earlier thread).

That's only true if you don't know what type of filesystem you're on. And, in the vast majority of cases (in fact, a content tracker is the only exception I can think of), it doesn't matter. If the user said 'xyz' and you can stat() it, great, that's what the user wanted! Just because it's really called 'Xyz' on the filesystem doesn't make any difference.

But git is a content tracker, so even if it's really a different hardlink that shouldn't matter, it's still referencing the same content. Go ahead and track whatever name the user specified originally, as long as it maps to a file on disk with the expected content you're set. If the file is really called 'foo' and I told git to track 'Foo', I'm perfectly happy with it continuing to think 'foo' is 'Foo' until I use 'git mv Foo foo'.

I don't see that as being a problem. Think of it, if you will, as if every single file simply had an implicit hardlink for every possible case or normalization variant. The whole point of the filename is that it is meta-information, used as an identifier and not as actual content, and thus it is perfectly fine for it to be a real string, subject to interpretation, rather than treated as a sacred binary blob like content is. The whole purpose of the name is to identify the inode in question, and case and normalization aren't particularly relevant here. As long as we can identify the file, we're happy.

Again, as someone who grew up in a case-insensitive world, there's no problems here. I wish I could tell you that it causes problems, I wish I could agree with you, but I can't.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I guess you haven't used unix tools much. The ever-popular HEAD perl utility (which does an HTTP HEAD against a URL), when installed, silently overwrites the head shell utility, which is used for all sorts of things, some even in startup scripts. Ooops! I've been hit by this more than once - and if you google for it, it hurt a lot of people.

Hmmm. Many important tools - that I wouldn't want to ever fail! - have similar needs to git. Backup/restore and file replication tools for example.

Ok - but how do you track the directory then (in git's terms, the tree). There's no way to tell what the user wants. Does the user want a copy of the file with different capitalization, or is the OS playing games?

I don't think you *actually* want it subject to interpretation.

Probably because you have been surrounded by tools that have a lot of extra code to cope with the case insensitive way of life, and learned to not do things that are completely valid, just to avoid trouble. Which is ok, but I don't think it makes the OS design decision defensible.

cheers,

m

I can imagine. However, I've never been hit by such a situation. This doesn't mean a case-insensitive filesystem is a problem per se, it means interactions between a case-insensitive and a case-sensitive filesystem can be a problem. That doesn't mean either way is "correct" it just means both don't work well together.

I like ice cream, and I like steak, but I sure don't think a mixture of steak and ice cream would go well together. Do you?

Both of which would be replicating the directory contents, not a listing of files specified by the user. If, as a user, I were to say "please replicate file FOO" and the file was really called "foo", I wouldn't be in the least surprised to see the tool take me at my word and produce a file called "FOO" with the contents of "foo". But in general, things like this operate on the filesystem, not on the user args.

If I say "track FOO", I probably mean it. So go ahead and track "FOO", even if you end up tracking the contents of file "foo". I certainly won't blame the tool for doing what I told it.

Sure I do. I find it very convenient, for example, to say "cd documents/school" when I really want to go to "Documents/School". Similarly, if I'm trying to reference gitweb/tests/Märchen, I'm quite happy to not have to figure out what normalization the filename is using and attempt to replicate that (especially as I have no idea which normalization my input mechanism uses - unlike Linus, I don't have a key dedicated to ä, and even if I did I wouldn't necessarily expect it to use precomposed vs decomposed). I can't think of a single reason why I'd want to be able to have 2 different files named "Märchen" on my disk. On the other hand, treating unicode normalization as significant can pose security risks - how am I to know that the file that is named "foo.txt" is really the same file "foo.txt" that I last saw? Someone I know on IRC sent me this image[1], which shows 6 files all apparently named "foo.txt" on a disk image. This is possible because on a case-sensitive HFS+ volume, the file system doesn't ignore ignorables when comparing filenames (it does on a case-insensitive HFS+ system), and so all of those filenames look identical up until you actually pipe their names through xxd and look at the byte sequence. When this sort of tomfoolery is possible, I simply cannot trust the names of any of my files anymore.

[1]: http://sailor月.com/imgs/ignorable.png

Extra code? I don't think so. The only reason I'd need extra code is if I were attempting to explicitly detect the "real" filename for a user-supplied argument, by scanning the directory contents until I found a file that was equivalent to the given argument. But there's no reason to do that. None of the code I've ever written, or any of the code I've ever seen, has had to do any extra work because it was on a case-insensitive filesystem. I contribute to a packaging system for the Mac called MacPorts, and I've never seen any patches on any of the 4000+ ports to handle case insensitivity (granted, I haven't looked at every port, but I've looked at a significant fraction). It's a complete non-issue.

The content of files is sacred. The filename is only there to provide a handle to locate the contents. I don't see any problem with expanding the equivalency scope of the filename to accept multiple encodings and cases. The only arguments I can see that have any validity at all are the ones that sound like "we use case-sensitive filesystems, and your case-insensitivity and normalization are causing problems with our tools! Conform to our world!". As I said above, this isn't a problem of case-insensitivity or normalization, it's a problem of interaction between two incompatible viewpoints. All I want to do is make git play nicer in an HFS+ world, and this would be far easier if you guys were willing to admit this is a problem that should be solved in the tool rather than a problem with the system.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

For those on Mac OS X: it is possible to create a case-sensitive HFS+ partition and use it with git. You even can just create a disk image and mount it. However, I wouldn't quite try to use it as startup filesystem...

-Geert

PS. I'm working on a proposal/patch for addressing the UFS/case sensitivity issues.
Will try to mail something later this week.

This is starting to stray far afield, but the first thing I did when I got a Macbook was to reinstall it with case-sensitive HFS as the boot file system. Works fine, including with git. The only problem I have had is that FileVault does not work. There are rumored to be some third-part apps that do not work but I do not use that many of those anyway.

andrew

The main problem with this approach is you know for certain that using HFSX as the boot partition is barely tested by Apple, and certainly untested by third-party apps. This means the potential for breakage is extremely high.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

No, actually, HFSX boot partitions are fairly well tested by Apple and most 3rd-party programs. I had one for a while and the only problems I encountered were with programs ported from Windows without Mac versions, such as "Microsoft Office for Mac" and "World of Warcraft". "Quake 4" has a few quirks which are easily worked around.

Cheers,
Kyle Moffett

Perhaps the big name companies might do some testing on HFSX, but I can guarantee most third-party programs will not be tested under HFSX.

Also, World of Warcraft isn't a ported program. It was developed for the Mac concurrently with the Windows version. Same with MS Office - it's an entirely different team (the Mac BU) developing MS Office for Mac independently of the Windows version, not a porting job. However, if you're saying these two big-name programs had problems, I wouldn't be surprised to see many more problems on various other third-party apps from smaller companies.

In any case, "just use HFSX" is still not an appropriate solution to the problem, especially since that will only take care of case sensitivity and not the utf-8 stuff.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I was going to post this earlier, but wanted to search the archives first. Here are the commands assuming you don't want to or can't partition a drive and format as ufs (I don't care for HFS+ much). I can't believe I didn't find the command in the git list archives, so voilà:

$ hdiutil create -size 300m -fs UFS foo.dmg
...............................................................................
created: /Users/mitch/foo.dmg
$ hdiutil attach foo.dmg
/dev/disk2          	GUID_partition_scheme          	
/dev/disk2s1        	Apple_UFS                      	/Volumes/untitled
$ cd /Volumes/untitled && git clone git://git.kernel.org/pub/scm/git/ 
git.git
... snipped ...
$ cd git && git status
# On branch master
nothing to commit (working directory clean)

After git clone in HFS+ land...

$ git status
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)

Should I just add this to the wiki? Then we can all go back to ignoring the insane filesystems.

Mitch

Definitely.

While it's a nice workaround, it really is just that (a workaround) because performance will be suboptimal in a repository running on a disk image (and many of switched to Git because of its speed).

Cheers,
Wincent

Not only is it suboptimal, it's also not acceptable, plain and simple. If an individual wants to do that, sure, but it's simply not an appropriate solution in general for this problem. I certainly don't want to have to attach a disk image every time I want access to anything I keep in a git repo, nor do I want to be restricted to keeping everything within a certain filesystem on disk. Additionally, while I'm not certain it's impossible, it's certainly very difficult to attach a disk image without anybody logged into the system at the GUI, as diskarbitrationd won't be running.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Hi,

If it's not acceptable, do something about it (and I don't mean writing 50 emails). If you don't want to do something about it, I have to assume that you accept it as-is.

Ciao,
Dscho

I never said I don't want to do anything about it. However, I do believe that it will take a significant investment of time and energy to learn all the gooey details of how git handles filenames and how the index works and all that jazz, which is knowledge that other people already have. I believe that, for me to solve this problem independently, it may require so much time that it never gets done (after all, I am fairly busy). However, if other people who already have this knowledge are willing to help, that would make this task far easier, especially given that if nobody else even acknowledges that this is a problem I don't have much hope of getting a patch accepted.

So again, I'm certainly going to try, but working by myself it simply may never get done.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

torsdagen den 17 januari 2008 skrev Kevin Ballard:

(This is only for those that think the problem should be solved somehow. The rest can move on - nothing to see here)

You may look at http://rosenberg.homelinux.net/cgi-bin/gitweb/gitweb.cgi?p=GIT.git;a=log;h=i18n
for inspiration. It's pretty obsolete by now and only a "proof of concept", i.e. it can be done, not that it necessarily should be done exactly this way.

Basically it intercepts the user's access to git, i.e. certain commands and how files are named (since those names represent a user interface). Then it assumes the internal encoding is UTF-8 (or garbage) converting to and from the user's local encoding. The heuristics is based on the assumption that a string (even random onesthat looks like UTF-8, with a very high probablity actually is UTF-8 encoded.

The test cases might be usable almost as is.

-- robin

I hope you're right (about them hating it), but we'll see. They've just opened the source for the ZFS port they're working on. By the time it goes final and becomes the default FS, replacing HFS+, probably within a couple of years, we'll see if they make the same two design decisions which cause the kinds of problems being discussed here (case-insensitivity, and ubiquitous FS-level UTF-8 normalization).

I've done a dumb search in the ZFS source code for "CASE" and see that it can in theory support case-insensitivity as an optional feature. The potential is there for Apple to use this. I personally hope that they don't, because as has already been pointed out, these little tricks tend to make life more difficult for users rather than helping them (the day I have two files in the same directory called "Märchen" and want to specify one of them on the command line I'll worry about that when I come to it).

http://fuzzy.wordpress.com/2007/06/09/zfsandfilesystemoptions/

Cheers,
Wincent

Side note: the thing is, the reason people shouldn't worry about it is that this is a *trivial* thing to handle. You really don't even need to know what you're doing. And you can test it today, easily.

Having two (differently encoded) files like that is really no different from the traditional UNIX FAQ of "how do I remove a file starting with '-'" or even more closely "how do I remove a file that has a character in it that I cannot get at the keyboard".

In other words, on a bog-standard UNIX (and yes, in this case, I bet OS X works fine too for this test), just try this

	filename1=$(echo -e "hello\002there")
	filename2=$(echo -e "hello\003there")
	echo Odd file > "$filename1"
	echo Another odd file > "$filename2"

and now you have a filename that is actually rather hard to type on the command line. In fact, for me they even *look* the same:

	[torvalds@woody ~]$ ll hello*
	-rw-rw-r-- 1 torvalds torvalds  9 2008-01-17 08:23 hello?there
	-rw-rw-r-- 1 torvalds torvalds 17 2008-01-17 08:23 hello?there

See?

Even in my graphical browser, those two filenames look 100% *identical*. I could give you a screen-shot, but I'm lazy. Just take my word for it, or just fire up konqueror on Linux (but it may well depend on the particular font you're using).

[ And yes, for other browsers, you might have something that shows them as
different characters - depending on the font, it might show up as a small box with [00 02] vs [00 03] in it, for example. But that's also actually 100% true of the two different encodings of 'ä' - you could easily have a file broswer that shows the multi-character as a multi-character, exactly to distinguish them and show that one of them isn't "normalized"!

The point is, once the filesystem doesn't corrupt the data, it's always easy to get at, and there is never any ambiguity. ]

How is this different from "Märchen" spelled with two different encodings for that "ä"?

I'll tell you: it's not at all different. It's 100% the exact same issue.

And does that make you perhaps go "Hunh? How do I remove it, or open it?"

And the fact is, those "idential looking" filenames (and thus they must be the same, and something should have normalized them to the same thing, no?) are obviously two different files, and they are *really*easy* to edit and look at.

Fire up that graphical browser again, and it doesn't even matter whether the filename looks identical or not, it shows up as two different files, and you can drag them around independently, rename them there, and at least my file browser shows clearly which is which, because I get a small icon with a preview in it, so I directly see which one is the "Odd file" and which one is the "Another odd file".

So the whole "but they _look_ the same" argument is just total BS. In just about all character encodings there has always been unique and different "characters" that _look_ the same on screen, and it has never really made them actually *be* the same, and it has never been a valid argument for them being considered the same.

Because even when they *look* the same, that file browser that didn't show the difference in names visually, still showed them correctly as two separate files, and I could still just rename them by hand by right-clicking on them and picking "rename".

So "look the same" is really not a new thing, nor is it even a really hard thing. Yes, people can get confused by it, but hey, people can get confused by *anything*. People get confused by filenames starting with a "-", yet nobody sane really says that filenames cannot start with a dash.

Linus

Sorry, but you're using different characters that look the same. But Kevins point was that it's a different thing if you use two characters that look the same or the same character with different encodings. This makes this HFS-specific problem different from the "look the same"- or the "case-insensitivity"-issues.

BTW: I also read about your argument that you wouldn't convert file data to normalized UTF-8 (I agree with you that this would be nonsense) and therefore filenames shouldn't be converted too. This is something where I have to disagree because a filename (like ctime, mtime, atime, ...) are meta data (while file contents isn't) and - until now - I would've guessed that you agree on this point because git doesn't care about filenames but contents.

IMHO it would be the best solution when git stores all string meta data in UTF-8 and converts it to the target systems file system encoding. That would fix all those problems with different locales and file system encodings ...

However, I have to agree that the enforced character set conversion causes more problems than it solves.

Regards,
Mark

No, the encoding was the same -- UTF-8. MacOSX converts one sequence of Unicode characters to *another* sequence, which are canonical equivalent, but being canonical equivalent does not mean they are the same characters. In the same way, as being compatible equivalent does not mean being the same. As well as, being case-insensitive equivalent does not mean being the same... Do you remember DOS? It stored all filenames in upper-case, so they original and stored names are case-insensitive equivalent, but they are not the same!

Dmitry

Hi,

+1.

And I would suggest the use of RFC 3454 as the guidelines for UTF-8 normalization.

Best regards,

Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [email protected]
Use XMPP!

The problem is that there is no way to know what the "target system encoding" is.

And it wouldn't actually solve the bigger problem on OS X anyway: as long as you are case-insensitive, you'll have all the same problems (ie the insane OS X filesystem presumably thinks that "MÄRCHEN" and "Märchen" are also identical, because they are "equivalent" names).

Linus

Hi,

Correct. Storing or using a normalized version of the filename is only part of the problem.

The full problem is:

User A <-> filesystem A <-#-> git < ...... > git <-#-> filesystem B <-

You have to encode/decode/normalize on all the <-#-> and there is no magic bullet. Each user would have to tell git "Hey I'm using utf-8" or "Hey, I'm a masochist using HFS+".

But I think its important for git to store the filenames in something that at least permits this kind of scenario.

All encoding/decoding/normalization is of course optional, and for git, it still is a sequence of bytes.

Correct. HFS+ has bigger problems. I'm not sure if this is enough to solve it.

But it would solve two linux users using different encodings.

And given that the filtering layers are optional, you have to configure them, it wont bite nobody.

Best regards,

Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [email protected]
Use XMPP!

Hi,

-1.

It's just too arrogant to force your particular preferences down the throat of every git user.

Ciao,
Dscho

Hi,

Do you agree that you need to store or at least calculate a normalized version of each filename to see if you are already tracking the file, to take in account all the the filesystems out there who are not case-preserving, case-sensitive?

If so, do you think those rules should be an option? Or a preference?

Should I specify in my config file that I want my filenames to be normalized?

Ignoring encoding, and case-sensitive issues in the git index creates problems for those people who want/need to use non-ascii chars in their filenames, and have some change of being able to collaborate with other users on different operating systems.

Best regards,

Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [email protected]
Use XMPP!

It's not arrogant to make a suggestion. Where is your alternative solution?

However, what about storing an additional information like the file system encoding (for every file)? This would result in the same behaviour (and speed) as today as long as the file system encoding is the same. Conversion will only be done when the targets file system encoding is different.

BTW: This reminds me of the code page switching stuff back in the times of MS-DOS 4/5. This really wasn't funny.

Regards,
Mark

But that's exactly the case he gave - 'ä' vs 'a¨' are exactly that:
different strings (not even characters: the second is actually a multi-character) that just look the same.

You try to twist the argument by just claiming that they are the same "character". They aren't, unless you *define* character to be the same as "glyph". Of course, if you claim that, then you can always support your argument, but I claim that is a bogus and incorrect axiom to start with!

Too many people confuse "character" and "glyph". They are different.

See, for example

http://en.wikipedia.org/wiki/Unicode

and notice the *many* places where they try to make that distinction between "character" and "glyph" clear (and also "code values", which are the actual bytes that encode a character).

See also

http://en.wikipedia.org/wiki/Unicode_normalization

and realize that a Unicode sequence is a sequence of *characters* even if it is not normalized! Those things are still characters, when they are the "simpler" non-combined characters.

You are trying to make a totally BOGUS argument, and you base it on the INCORRECT basis that the TWO characters 'a'+'¨' somehow aren't independent characters. They *are*. They are *different* characters from 'ä', even though they may be "Canonically equivalent" as a sequence.

The fact is that "equivalent" does not mean "same". Why cannot people accept that?

Linus

Ahhhh ... now I understand.

Regards,
Mark

Hi,

This is very true.

I'll shut up now if you can answer me one question, because it really is a problem for my team.

We have people using windows, people using Macs, and people using several flavors of Linux desktops. They all have different settings and if I add a file like áéióú that happens to be UTF-8 encoded, it will reach a iso-latin-1 user as visual garbage. git will track the file perfectly, we know that, because the sequence of bytes that my system used to create the file will be the same on all "sane" systems, but the file will look "funny" to some users, and we get complaints for some less enlightened ones.

The answer is that users should not create filenames with non-ascii characters if they want a consistent experience, right?

This is just so that I can write a best practices document to them...

Best regards,

Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: [email protected]
Use XMPP!

That's the easist thing to do if you want to assure that things will mostly work across multiple different OS's, with different levels of sanity. You might also want to include that it's a bad idea to create two filenames that are identical on case-insensitive filesystems, i.e., "makefile" and "Makefile", or "foo.H" and "foo.h" which even though it works Just Fine on Linux, will likely cause problems on Windows and MacOS filesystems, and other systems that are insane with respect to case insensitivity.

- Ted

Yes.

I can't really suggest anything else than trying to make everybody use UTF-8.

[ Not just for filenames, by the way - this is one of the reasons I think
it is so *important* to not corrupt filenames, exactly because this is in no way filename-specific at all, and filenames are generally "textual data" exactly the same way a text-file is.

But only totally insane people think that you should force-normalize text-files, even though all the issues are obviously all the same regardless of whether it's a filename or a word in textfile. ]

And yes, I also realize that it's not going to be realistic. We're probably *closer* to that than we used to be, but I don't think you can even make Windows think FAT is UTF-8.

I don't know how NTFS works (I know it is Unicode-aware, and I think it encodes filenames in UCS-2 or possibly UTF-16, but there is an obvious 1:1 translation to UTF-8, and since we use C strings, I'd assume/hope Windows actually uses that unambiguous translation for any filenames).

Under modern Linux and OS X, UTF-8 is basically the only way (older Linux distros may be set up for Latin1, but at least the newer ones seem to all default to a UTF-8 locale).

Oh, absolutely. That takes care of 99.9% of all source projects. Even then you can have problems with case insensitivity (the Linux kernel sources are all US-ASCII filenames, for example, but *literally* has many files that are identical if you ignore case, and that's not unheard of).

So yes, to a first approximation, the answer is to simply avoid using anything but US-ASCII. It's seldom a big limitation when talking about filenames.

Linus

torsdagen den 17 januari 2008 skrev Linus Torvalds:

It's UTF-16 (when needed). I think it's all in the Linux kernel for you to see.

UTF-16 (was UCS-2 until MS did a s/UCS-2/UTF-16/ on the documentation).

It uses the local 8-bit codepage, which is not UTF-8, often some latin-inspired thingy, but in Asia multi-byte encodings are used. In western Europe it is Windows-1252, which is almost, but not exactly iso-8859-1. Oh, and then we have the cmd prompt which has another encoding in 8-bit mode.

I think there is a cygwin patch that converts to and from UTF-8. An application can choose to use the "A" or "W" interfaces. The W-API's are the real ones and the others' are just wrappers that convert to and from UTF-16 before anything happens (i.e. CreateFileA is slower than CreateFileW and so on).

-- robin

.. well, FAT certainly wasn't. But yes, VFAT probably is. Not that I want to look at it ;)

Well, if it uses a 8-bit codepage, then that means that as far as the POSIX filename interface is concerned, it has nothing what-so-ever to do with Unicode (ie unicode is just a totally invisible internal encoding issue, not externally visible).

I assume you have to use some insane Windows-only UCS-2 filename function to actually see any Unicode behaviour.

Sad. Because there really is no reason to use a local 8-bit codepage when you could just use UTF-8.

So the CreateFileW() is the "native UTF-16 interface", and CreateFileA() is the 8-bit codepage one that has nothing to do with Unicode and is purely some local thing.

But for a UNIX interface layer, the most logical thing would probably be to map "open()" and friends not to CreateFileA(), but to CreateFileW(utf8_to_utf16(filename)).

Once you do that, then it sounds like Windows would basically be Unicode, and hopefully without any crazy normalization (but presumably all the crazy case-insensitivity cannot be fixed ;^).

So it probably really only depends on whether you choose to use the insane 8-bit code page translation or whether you just use a sane and trivial UTF8<->UTF16 conversion.

Anybody know which one cygwin/mingw does?

Linus

Yes, the default code page for the command prompt uses so-called OEM encoding, and GUI programs uses another one, which MS calls as "ANSI" encoding. However, if you use Cygwin, then you have ANSI encoding in the command prompt. So, in the same command prompt window, you can have Cygwin programs using one encoding and other window console programs using a different encoding.

Some people tried to set the current code page to 65001, which is the Microsoft code page for UTF-8. However, it seems that does not work very well.

http://support.microsoft.com/kb/175392
http://blogs.msdn.com/michkap/archive/2006/03/13/550191.aspx

It seems to me that Win32 API functions work correctly with UTF-8 (after all, they are just wrappers over UTF-16 functions), but Microsoft's C library cannot handle UTF-8 (or any other encoding that requires more than two bytes per character).

There is a patch for Cygwin that adds UTF-8 support for it, however, Cygwin maintainers do not like it, so it is not integrated. I think Cygwin 1.7 will support UTF-8, but I have no idea how soon it will be released.

I don't know much about mingw, but if I am not mistaken, mingw relies on Microsoft's C library, so I suppose it uses an "OEM" code page for console programs by default.

Dmitry

fredagen den 18 januari 2008 skrev Linus Torvalds:

I just had to investigate this a bit, so on a Vista machine I started a cmd prompt and typed mode con: cp select=65001, selected the lucida font and then echo å >x.txt and opened it in notepad and it was UTF-8 encoded. So there might be some hope after all. I don't know how to change the encoding for non-console apps. I leave that as an excercise for the list.

-- robin

Yes, but have you tried to run any batch file? At least, on WinXP all batch files silently stopped working after choosing 65001, and I don't know what else gets broken, because Microsoft C library does not work with encoding that requires more than two bytes per character.

It is not difficult to change the current encoding in any Windows application, the real issue is that neither Microsoft C library nor Cygwin library does not work correctly with UTF-8. There is a patch for Cygwin though...

Dmitry

Dmitry Potapov:

Indeed. On Windows, you should avoid using UTF-8 and instead use UTF-16 everywhere. That usually works better, and if you run on an NT-based system it will convert all the data to WinAPI to UTF-16 anyway.

\\// Peter - http://www.softwolves.pp.se/

Errr... doesn't UTF-16 (as compared to USC-2) sometimes (for some exotic characters) require more than two bytes per character?

Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Cygwin does not yet support doing the smart thing. At the moment you can only open() files in the current 8 bit codepage. There is a patch floating around to allow using UTF-8, but it was rejected for inclusion because it was considered too hackish. Instead work has been ongoing for some time to replumb the internal representation of Windows filenames to use UTF-16 instead of plain chars, so that conversion overhead can be held at a minimum. In conjuction with dropping Win9x/ME support this also means the Native APIs like NtCreateFile() can be used directly, as they are more low level than the Win32 -A and -W functions and expose more flexibility, such as the ability to implement the openat() family of functions natively (no pun intended) without emulation. These two items (unicode and dropping non-NT windows) are the big features for 1.7.

Of course since a lot of what Cygwin does is translate paths in sometimes unobvious and complicated ways, there's a lot of path handling code to adapt, so it's taking a while.

Incidently, the ridiculously short MAX_PATH of 260 on Windows comes from the Win32 -A version of the functions. The -W API and the Native API can cope with paths of up to 32k wide chars, so a side benefit of this should be the ability to finally stop running into length limits. Of course there's always a catch: when using long filenames with the Win32 -W API or the Native API you can only use absolute paths, so either you have to live with the 260 limitation for relative paths or you keep track of the current directory and always do a rel->abs conversion. Or better, if you stick to the Native API you can do a directory handle relative openat-type thing which I suppose starts to sound relatively sane. However, there's another catch here: For some time Cygwin has maintained a separate and private value of CWD behind Windows' back, and only synced the two when spawning a non-Cygwin binary. This allows Windows to happly think the process' CWD is always C:\ or whatever, and not hold an open handle to the actual CWD. In turn Cygwin uses this to allow POSIX filesystem behavior of being able to unlink the current dir, which some programs or build systems assume they can do but is not possible in straight Win32. This is a roundabout way of saying that going back to actually having to keep a handle to CWD open again in order to do relative paths might be complicated.

Brian

I just don't understand why you insist that the filename is data, when it is clearly metadata. The filename has two purposes: the identify the file to the user, and to provide a handle with which to reference the file contents. The specific byte sequence is in no way sacred.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Uhh. And exactly how do you know the difference, and why should it matter?

A lot of data is metadata. Look at the git index file. It's *all* metadata. Does that mean that the OS has the right to corrupt it?

IOW, why do you seem to argue that metadata something you can corrupt, but not then "regular" data?

Why is it ok to change a filename, when that same filename may *also* be encoded by the user in a regular data file (think about MD5SUM files, for example, that include the pathname, but now the pathname is part of the file data, not on a filesystem).

So filenames are data, they're metadata, they're whatever. None of that means that it's acceptable to corrupt them, or gives the OS any reason to say that it "knows better" than the user in how users use them. It's still the *users* metadata, not the filesystems own metadata!

In many cases, users use filenames *as* data, ie the filename actually has a meaning in itself, not just as a handle to get the file contents.

If this was truly metadata that isn't visible to the user, and not under the users control (ie indirect block numbers etc), then you'd have a good point. At that point, it's obviously entirely up to the filesystem how the heck it encodes it.

But that's not what filenames are. Filenames are an index specified by the user, not by the computer.

Linus

Linus Torvalds:

But they are not different strings, they are canonically equivalent as far as Unicode is concerned. They're even supposed to map to the same glyph (if the font has an "ä", it should display it in both cases, if it has an "a" and a combining diaeresis, it should make up one).

You cannot do a binary comparison of text to see if two strings are equivalent.

Whereas you are confusing characters and code points.

"ä" and "a¨" use different code points, but they encode the same character, and from the user's perspective it is the *character* that is interesting (although he might confuse it with the glyph).

Actually, NTFS is a bit broken. It sees file names as a string of 16-bit words. It doesn't check that it is valid UTF-16, or even valid UCS-2, it allows almost anything.

Apple made Mac OS X handle filenames properly, by seeing that file names are a string of characters, not code points, so they use a canonical form for all characters (personally, I would have preferred the pre-composed form, though).

\\// Peter - http://www.softwolves.pp.se/

There are canonically equivalent, but they are different sequences of characters as Unicode is concerned. In one case, we have one character in the other case, we have two characters that canonically equivalent to the first one.

By defition, sequences of characters that are canonically equivalent are both visual and functional equivalent...

Of course, you can't. Who argues otherwise?

I am afraid it is you who confuses "characters" with "abstract characters", there is no place in the standard saying that "characters" are "abstract characters" only. On contrary, the term "characters" is used to refer non abstract characters.

Dmitry

Dmitry Potapov:

Perhaps it's just a case of confusion about naming conventions. I tend to use "character" as a "grapheme cluster", i.e a "user character" (to the end user, "ä" and "a"+diaeresis is the same character, no matter if they would display as different glyphs), whereas some people use "character" as a "code point", which would be more of a "programmer character". And then there are some people that still use "character" interchangibly for "bytes" or "code units" (for UTF-16; a pair of surrogate code units is still only one "code point").

\\// Peter - http://www.softwolves.pp.se/

Fuck me with a spoon.

Why the hell cannot people see that "equivalent" and "same" are two totally different meanings.

.. and this is relevant how? They are different strings. Not the same.

Equivalence doesn't matter. Equivalence is *evil*. Equivalence is what gives us case-insensitive filesystems ("because the names are equivalent").

Filesystems don't *want* equivalence. They want a much stronger exactness guarantee. Exactly because sometimes the differences matter.

Linus

As far as I can tell, the only time you ever run into the problems you've described on a filesystem which treats filenames as unicode strings (and therefore is free to normalize), are when you're trying to interact with a filesystem that treats filenames as sequences of bytes.

This doesn't mean treating filenames as unicode strings is wrong, it just means that the world would be much better if every filesystem had the same behaviour here. It's kinda like the endian issue, except there's no simple solution here.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Hi,

[please do not top post. Just delete everything you do not reply to]

If you read the first message in this thread, you would probably know that the problem exists on Mac even without any other filesystem being involved. My understanding of it is that is caused by HFS+ converting one sequence of Unicode characters (generated by Mac keyboard driver) to another sequence using "fast decomposed" conversion.

[And please stop calling by normalization what is not. Mac does NOT normalize Unicode strings, it uses some sub-standard conversion, which neither produce a normalized string nor is guaranteed to be stable across versions of Unicode.]

Actually, there is, if you care to do something. You can write a wrapper around readdir(3) that will recodes filenames in Unicode Normal Forms C. This does not require much knowledge of Git -- what it requires the desire to do something to solve the problem. Of course, this step alone is not a complete solution (it does not solve case-insensitive issue), but the first step in the right direction...

BTW, Git is far from being only software that ran into this problem with Mac. But not being first, we can benefit from other people experiences:
http://osdir.com/ml/network.gnutella.limewire.core.devel/2003-01/msg00000.html

Dmitry

In this case, git's index counts as a filesystem that treats filenames as sequences of bytes. But yes, it is possible, though somewhat difficult, to produce this problem on just HFS+. It's far more common when the file was originally added on a different filesystem

From what the HFS+ technote says, it produces a variant of Normal Form D. This variant, while not guaranteed to be stable across versions of HFS+, but in practice it is stable.

What would you prefer I call it?

I'm not sure how that would solve anything. Sure, it would provide a stable, known encoding for git to compare filenames against, but that would only work if the filename is known to be Unicode, and as it has been pointed out on other filesystems the filename can be whatever encoding the user chooses (which, IMHO, is a flaw).

It looks like their problem was binary compatibility with strings from other clients that were using Normal Form C instead of Normal Form D. git's problem is that it's only even using a known encoding on HFS+.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

There is no such thing in the standard as a variant of NFD. Moreover, even if this conversion were described in the standard, it would never called as normalization, because normalization means conversion that makes all equivalent strings having identical binary representations. HFS+ conversion does not met this criterion, so it not normalization.

Apple calls it as decomposition, which is correct even if it is not full decomposition as stated in the technote.

I believe that Git internally should use only UTF-8 for encoding file names, commit messages, etc. The problem with some other filesystems should be addressed separately (by those who work on those systems or at least have access to them). Regardless interoperability with other systems, this change alone should solve the issue that was described in the first message of this thread.

Dmitry

If somebody wants to do this, then readdir() isn't the only place, but yes, readdir() is one of the places.

I suspect that if we were to just do the "turn into NFC on readdir() on OS X", that might actually be good enough to hide most of the problems. The issue isn't just that OS X mangles the filenames, it's that it picks a particularly *stupid* way to mangle them (the decomposed forms), which means that OS X will actually not just corrupt "odd cases" of Unicode, but will corrupt the obvious and *common* Latin1 translations of Unicode.

I don't know if NFC is better for other locales, but I doubt it. Usually people want to do the *composite* forms, not the *de*composed forms.

A trivial example of this for some cross-OS issue:

- let's say that you have a file "Märchen" on just about *any* other OS than OS X. It could be Latin1 or it could be Unicode, but even if it is Unicode, I can almost guarantee that the 'ä' is going to be the *single* Unicode character U+00e4 (utf-8: "\xc3\xa4", latin1: "\xe4")

So from a cross-OS standpoint, that's the *common* representation, and yes, you can create the file that way (I don't know what happens if you actually create it with the Latin1 encoding, but I would not be surprised if OS X notices that it's not a valid UTF sequence and assumes it's Latin1 and converts it to Unicode)

- But on OS X, because of Apples *insane* choice of normal form, it will then be turned into "a¨". I doubt *anybody* else does that. If you have to normalize it, NFD is just about the *worst* choice.

So yeah, even just re-coding it as NFC on readdir() would at least mean that any OS X git client would be MORE LIKELY to pick the same representation as git clients on other OS's.

It wouldn't solve all problems (and it would almost certainly create a few new ones), but it would likely at least increase compatibility between systems.

So doing the NFC conversion on readdir() on OS X is probably a good idea, and probably is the simplest way to make it interact better with other OS's. And it's definitely safe on OS X, since OS X _already_ corrupted the name, so we're not losing any information (in contrast, on other systems, doing a NFC conversion would possibly lose encoding detail _and_ might be incorrect simply because they might not use Unicode in the first place).

Anybody want to creat a compat layer around "readdir()" that does that NFC conversion on OS X but not elsewhere?

Linus

FWIW: I just made a test and it seems that MacOS X refuses the creation of a file with this invalid name.

Maybe I'll try it.

Regards,
Mark

Hi,

I think a better approach would be to try to match the name to what we have in the index. Then we could implement case-insensitivity and MacOSX workaround at the same time.

Ciao,
Dscho

I thought about that, but the problem is that HFS+ _already_ mangled names from what the user entered (and what is used by anyone else) to some sub-standard form, which no one outside of Mac likes or uses. Thus, bringing filenames back to the NFC form (which is what almost anyone uses) is the only sane thing do, because no one outside of Mac really needs to know about this HFS+ specific craziness.

So I really dislike the idea that due to some HFS+ specific conversion, we may end up having some strangely encoded names in a Git repository. Sane people enter names only in NFC, so why should they suffer because of some insane conversation made by filesystem behind everyone's back? And I am not entertaining the idea of having this Mac OS/X specific workaround outside of Mac OS/X.

Besides, writing a wrapper around readdir() is not difficult. We already have git-compat-util.h, which redefines some functions for some platforms, so I don't see any problem with writing a wrapper around readdir().

Dmitry

Well, more importantly, most of the important cases actually don't have an index entry yet.

For example, what about "git add"? That's when it really matters that you add things in a sane format, and by definition, you don't have an index entry to try to match to.

So once you aim for NFC in "git add", now the index will generally be in NFC anyway (since I agree that that's what you'd normally get on non-OSX systems), so there is little point in then matching the index.

But no, it won't fix all problems. I do suspect it would make them less obvious in practice, though.

Linus

Hi,

So? That's why I said "match", not "compare for identity".

To be a little bit more precise: I think a viable plan would be to

- have a config switch which determines what type of filename mangling we allow the host OS to perform (Unicode "normalisation", case mongering), and leave _everybody_ alone who left that switch unset,

- "overload" readdir() (by the famous git_X(); #define X git_X trick),

- have the overloaded readdir() _know_ which is the current prefix, and load the index if it has not yet been loaded (but probably into a static variable to avoid reloading, and to avoid interfering with the global "cache" instance).

It _could_ be wise to store the "normalised" forms at one stage (instead of the index) to speed up comparison -- the prefix has to be normalised for readdir()s purposes, too, then.

This is possible with the HFS+ problem, since we know exactly how HFS+ tries to "help", and for case insensitivity too, I think. But it may be restricting ourselves for other filename "equivalences" we might want to handle one day.

BTW: I cannot think of anything else than readdir() which should have the "problem" of reading back a name that the user did not specify. What am I missing?

No. I think that would be a serious mistake. If you add a file on MacOSX (with a _mangled_ filename, think of "git add ."), git should not try to be as clever as HFS+ and "remangle" it.

It _is_ UTF-8, so what's the problem?

As for the HFS+ specfic conversion: like the CRLF issue, I am opposed to have a "solution" affecting other people than those on broken system. So I very much _want_ it to be an HFS+ specific conversion.

Exactly.

Ciao,
Dscho

For what it's worth, their choice wasn't entirely "insane" ie. it did have an element of rationality: that decomposed forms are a little bit simpler to sort.

Of course, this doesn't excuse them for creating a file system that interacts so horridly with basically everything else out there.

Cheers,
Wincent

No they are *not*.

In many languages, 'ä' does *not* sort like 'a' at all, and if you think it does, you'll sort at least Finnish and Swedish totally wrong (åäö are real letters, and they sort at the *end* of the alphabet, they have nothing what-so-ever to do with the letters 'a' or 'o').

The fact that in *some* languages the decomposed forms sort as the base letter is immaterial. It's only true in some cases.

So no, sort order is not it. To sort right, you need to use the a real Unicode sort (and the decomposed form is *not* going to help you one bit, quite the reverse).

It may be that a case compare is easier in NFD (ie you basically only do the case-compare on the base letter).

Linus

But there is no way to know whether 'ä' in a document is the Finnish 'ä' or a 'ä' from, say, German, that sorts after 'a'.

Unicode sort is not enough, there is no language indicator in an Unicode document, which is why Unicode, while solving a bunch of problems, has its very own, cf. the infamous CJK problem.

But that's all very OT.

Mike

... without knowing the locale. Correct.

That's why sorting is locale-dependent, even in Unicode. And why you should always sort using the *combined* character, not think that you can sort by decompsed sequence.

That said, even then you get the wrong thing. Some things cannot be sorted character by character at all, and have semantical sorting at a higher level entirely. I think most European family names are traditionally sorted by effectively using the prefixes (ie d', von, etc) as a secondary sort key (so even though they are in front, they sort as if they were at the _end_ of the name).

So unicode doesn't help with sorting, and you shouldn't even try to find sort rules in the Unicode spec or tech reports. But in general, decomposing the characters just makes things worse, not better. To sort well, you tend to need the bigger picture, not the details.

Of course, for something like git, we sort by binary value, because we also require the sort to be not just well-defined, but *stable*. A sort based on any kind of unicode rule is rather likely to change over time.

Linus

That said, the locale doesn't necessarily express the language in which the document is written. It's easy enough to read documents that are not written in your native language on the net. That's already what we are both doing right now. Fortunately, HTTP and HTML have ways to indicate the language in which a document is written in, but that leaves out plain mail, for instance.

That said, the "decomposed" version of UTF-8 has nice side effects on OSX, with UTF-8 encoded RockRidge ISO-9660 volumes (with or without Joliet ; OSX will use RockRidge by default when it's there), for instance.

Mike

AFAIK, the RockRidge standard prescribes to use the portable character set, and it has nothing to do with Unicode. Basically, it is a subset of ASCII.

http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap06.html

So, I don't think UTF-8 encoded filenames are valid regardless whether they are decomposed or not.

Dmitry

Actually, it prescribes to use the portable *filename* character set, which is even more restrictive than just portable character set.

http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#tag_03_276

Anyway, there is no place for UTF-8 in it.

Dmitry

.. and quite commonly, there are multiple languages per document.

The good news is that sorting is almost never relevant or done over general documents. You sort almost only well-behaved data, and quite often the exact order is less than important: and when it is, you have very specific rules (which probably seldom have anything what-so-ever to do with general unicode ;).

Well, Unicode already handles the "reading" part, just not the sorting.

I think Unicode in general (and UTF-8 in particular) is a great thing. I do not argue against Unicode at all. It's what I use myself.

The thing I argue against is that they force normalization (and then, as a secondary complaint, their insane choice of target format).

Linux is generally UTF-8 too, and does all of this much better. No forced normalization, and it uses UTF-8 everywhere as the encoding model. Joliet and RR works beautifully.

(I don't think RR is NFD, btw. It's the standard microsoft UTF-16 without normalization, afaik. I think you can happily generate a Rock Ridge disk that has two _different_ filenames that OS X cannot tell apart, but that both Linux and Windows can see peoperly)

Linus

That's what I get for believing Wikipedia ("This makes sorting far simpler"):

http://en.wikipedia.org/wiki/UTF-8#Mac_OS_X

Cheers,
Wincent

Could people _please_ stop this already?

I think the sane people see the difference between equivalence and sameness, and we established that a filesystem that mangles the filenames behind user's back is a bad design. Anybody who followed the thread and still does not agree with you is, eh, "ugly-and-stupid", as you might say ;-). You cannot educate them all.

The thing is, even if you mange to educate them all, that broken filesystem, and other filesystems with similar brokenness, do not go away. If your ultimate objective is to declare that it is the right thing for git not to support such broken filesystems, and to make everybody agree to it, that is fine. Please keep pouring fuel to the fire. But if that is not the case, we would need to devise a way to help lives easier for the unfortunate people who are stuck on such filesystems. They may not even realize that they are unfortunate now, and I agree that some education is justified, but this thread has raged on long enough to salvage any salvageable lost souls (the remaining ones may be beyond salvation but let's not waste time on them).

I'd rather see our mental bandwidth spent on coming up with a workable workaround for such broken filesystems, while not hurting use of git on sane platforms.

I fear it might have to end up to be very messy and slow, though.

Hi,

Welcome, voice of reason.

Actually, I see some value in calling them names, see
http://video.google.nl/videoplay?docid=-4216011961522818645 for why.

I was almost starting with hacking on this, but then the discussion annoyed me too much, and I asked myself for who I think I'd do this.

IMHO those people should ask "how could I begin to work on this".

Instead, they started a useless flamewar.

Now, back to the issue: Robin posted a link to his UTF-8 work. While it is way too intrusive, and not limited to filenames at all, I think it has a few good pointers.

Ciao,
Dscho "who needs to calm down now"

Random thought. Would it make sense to implement a git paranoid mode to autodetect name mangling.

I.e. After opening or creating a file by name we do a readdir in the same directory to make certain we can find that same name/inode combination. Then on name-mangling systems we can autodetect they exist and limit ourselves to just what they don't mangle with no prior knowledge. By refusing to process names that actively
get mangled. For small directories that you frequently see in
development it shouldn't even be that slow.

Eric

Inside init-db where we already check how the filesystem behaves, we could have an autodetection. A rough equivalent of what I had in mind is:

mkdir -p "Märchen/Märchen"
if test "$(cd Märchen && echo M*)" = "Märchen" then
: not mangling
else
git config core.namemangle true
fi

(of course we do that in C not in shell).

I wonder if that is good enough. Git repositories can be copied over to different filesystems.

Nicolas

Do you mean "cp -a"? If I am not mistaken we already have that issue, due to core.filemode, when user does that across filesystems with different behaviours.

There is not much we can do against "cp -a" other than telling users that some configurations need to be adjusted.

Linus Torvalds:

I'd prefer not to.

It is relevant because the Mac OS file system stores file names as a sequence of Unicode code points, in a (apparently slightly modified) normalized form, whereas Git prefers to see file systems that store file names as a sequence of octets, which may, or may not, actually map to something that the user would call characters.

I happen to prefer the text-as-string-of-characters (or code points, since you use the other meaning of characters in your posts), since I come from the text world, having worked a lot on Unicode text processing.

You apparently prefer the text-as-sequence-of-octets, which I tend to dislike because I would have thought computer engineers would have evolved beyond this when we left the 1900s.

But the real issue is that Git cannot use it's filenames as string of octets on Mac OS X, since the file system doesn't handle it. So Git needs to do something sensible. That's part of porting. Preferrably that would involve supporting real Unicode file names, which would also work on Windows (through it's UTF-16 file APIs), and in part on other systems (through conversion to the systems' locale encoding).

\\// Peter - http://www.softwolves.pp.se/

I agree. Every single problem that I can recall Linus bringing up as a consequence of HFS+ treating filenames as strings is in fact only a problem if you then think of the filename as octets at some point. If you stick with UTF-8 equivalence comparison the entire time, then everything just works.

Granted, this is a problem when you have to operate on a filesystem that thinks of filenames as octets, but as I said before, this doesn't mean the HFS+ approach is wrong, it just means it's incompatible with Linus's approach.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Git's data model relies on SHA-1 hashing of data, including filenames. So at some level, git _has_ to treat data as octets, and "equivalent" strings must be the same at the octet level (or else you lose all of the useful properties that the hashing data model provides). You can argue about where in the program conversion and normalization occur, but I don't think you can get around the fact that you're going to need to think of the "filename as octets at some point."

-Peff

At *some* point everything stored in computers is a sequence of octets. In fact, the whole point of the Unicode standard is to define characters and how to map each character to a unique number (code points) and then how to encode this number into sequence of octets.

There are more than one equivalence comparison. The unicode standard defines at least two, and for some other purpose you may want to use some others, but for some reason you are trying to present that to work with text means to follow only one type of equivalence the entire time...

Dmitry

git calculates hashes over filenames and sorts them. This is not a mere question of "UTF-8 equivalence comparison".

It also is a problem when operating on a filesystem that considers "ä" a single utf-8 character instead of decomposing it.

It is not the business of a file system to juggle with filename representations.

David Kastrup

No, it's a question of hashing algorithm. And it's one that's fairly easily solved simply by picking a specific nonambiguous UTF-8 encoding before hashing.

What makes you say that?

You're right, that probably belongs in the VFS layer, but the behavior is the same either way. You can't leave it up to user-space libraries to enforce a filesystem encoding, because you can't rely on all clients to behave properly.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

UTF-8 is a *single* encoding, and it maps every Unicode character to a unique binary representation. So, it is completely nonambiguous.

Dmitry

In this case, encoding refers to normalization form, as other people have used it in the conversation besides me.

I suggest you stop trying to find inconsequential stuff to argue about, especially when a tiny bit of critical thinking would reveal the answer.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I thought we spoke about HFS+, and it does not use any normalization form, because normalization should produce binary identitical strings for equivalent strings and HFS+ conversion does not. So, it looks like you redefine both words "encoding" and "normalization" here.

All your arguments based on confusion and the fact that some other people were probably confused does not make your arguments any more valid.

IMHO, most of your arguments are inconsequential stuff, so I am not sure what I am supposed to do about your writings. Probably, it does not make sense to respond your mails anymore...

As to critical thinking, it definitely reveals that Apple's choice was far from being. Is it so difficult to accept?

Anyway, if you think that you know better than other how to properly deal with the problem, why don't you try to actually *do* something and write some code that works as your propose.

Dmitry

There exists more than one "normalization form" (even across MacOS platforms), and git is cross-platform. And people can't be made to agree on normalization forms, anyway. You are aware that Unicode code points are shared between some Chinese and Japanese signs, and that stroked forms might be composed differently in different languages? We don't need to go to the Far East, anyway: in Turkish, İ and i are equivalent, as are I and ı, whereas in other European languages, I is instead equivalent to i. In the Netherlands, ÿ is IIRC equivalent to ij. And so on.

Now that you have established that you are the only person on the list capable of critical thinking, how about going elsewhere where you can find similarly sharp critical thinkers?

David Kastrup, Kriemhildstr. 15, 44793 Bochum

You say "I agree", BUT YOU DON'T EVEN SEEM TO UNDERSTAND WHAT IS GOING ON.

The fact is, text-as-string-of-codepoints (let's make the "codepoints" obvious, so that there is no ambiguity, but I'd also like to make it clear that a codepoint *is* how a Unicode character is defined, and a Unicode "string" is actually *defined* to be a sequence of codepoints, and totally independent of normalization!) is fine.

That was never the issue at all. Unicode codepoints are wonderful.

Now, git _also_ heavily depends on the actual encoding of those codepoints, since we create hashes etc, so in fact, as far ass git is concerned, names have to be in some particular encoding to be hashed, and UTF-8 is the only sane encoding for Unicode. People can blather about UCS-2 and UTF-16 and UTF-32 all they want, but the fact is, UTF-8 is simply technically superior in so many ways that I don't even understand why anybody ever uses anything else.

So I would not disagree with using UTF-8 at all.

But that is *entirely* a separate issue from "normalization".

Kevin, you seem to think that normalization is somehow forced on you by the "text-as-codepoints" decision, and that is SIMPLY NOT TRUE. Normalization is a totally separate decision, and it's a STUPID one, because it breaks so many of the _nice_ properties of using UTF-8.

And THAT is where we differ. It has nothing to do with "octets". It has nothing to do with not liking Unicode. It has nothing to do with "strings".

In short:

- normalization is by no means required or even a good feature. It's something you do when you want to know if two strings are equivalent, but that doesn't actually mean that you should keep the strings normalized all the time!

- normalization has *nothing* to do with "treating text as octets". That's entirely an encoding issue.

- of *course* git has to treat things as a binary stream at some point, since you need that to even compute a SHA1 in the first place, but that has *nothing* to do with normalization or the lack of it.

Got it? Forced normalization is stupid, because it changes the data and removes information, and unless you know that change is safe, it's the wrong thing to do.

One reason _not_ to do normalization is that if you don't, you can still interact with no ambiguity with other non-Unicode locales. You can do the 1:1 Latin1<->Unicode translation, and you *never* get into trouble. In cotnrast, if you normalize, it's no longer a 1:1 translation any more, and you can get into a situation where the translation from Latin1 to Unicode and back results in a *different* filename than the one you started with!

See? That's a *serious*problem*. A system that forces normalization BY DEFINITION cannot work with people who use a Latin1 filesystem, because it will corrupt the filenames!

But you are apparently too damn stupid to understand that "data corruption" == "bad", and too damn stupid to see that "Unicode" does not mean "Forced normalization".

But I'll try one more time. Let's say that I work on a project where there are some people who use Latin1, and some people who use UTF-8, and we use special characters. It should all work, as long as we use only the common subset, and we teach git to convert to UTF-8 as a common base. Right?

In your *idiotic* world, where you have to normalize and corrupting filenames is ok, that doesn't work! It works wonderfully well if you do the obvious 1:1 translation and you do *not* normalize, but the moment you start normalizing, you actually corrupt the filenames!

And yes, the character sequence 'a¨' is exactly one such sequence. It's perfectly representable in both Latin1 and in UTF-8: in latin1 it is a two-character '\x61\xa8', and when doing a Latin1->UTF-8 conversion, it becomes '\x61\xc2\xa8', and you can convert back and forth between those two forms an infinite amount of times, and you never corrupt it.

But the moment you add normalization to the mix, you start screwing up. Suddenly, the sequence '\x61\xa8' in Latin1 becomes (assuming NFD) '\xc3\xa4' in UTF-8, and when converted back to Latin1, it is now '\xe4', ie that filename hass been corrupted!

See? Normalization in the face of working together with others is a total and utter mistake, and yes, it really *does* corrupt data. It makes it fundamentally impossible to reliably work together with other encodings - even when you do converstion between the two!

[ And that's the really sad part. Non-normalized Unicode can pretty much
be used as a "generic encoding" for just about all locales - if you know the locale you convert from and to, you can generally use UTF-8 as an internal format, knowing that you can always get the same result back in the original encoding. Normalization literally breaks that wonderful generic capability of Unicode.

And the fact that Unicode is such a "generic replacement" for any locale is exactly what makes it so wonderful, and allows you to fairly seamlessly convert piece-meal from some particular locale to Unicode:
even if you have some programs that still work in the original locale, you know that you can convert back to it without loss of information.

Except if you normalize. In that case, you *do* lose information, and suddenly one of the best things about Unicode simply disappears.

As a result, people who force-normalize are idiots. But they seem to also be stupid enough that they don't understand that they are idiots. Sad.

It's a bit like whitespace. Whitespace "doesn't matter" in text (== is equivalent), but an email client that force-normalizes whitespace in text is a really *broken* email client, because it turns out that sometimes even the "equivalent" forms simply do matter. Patches are text, but whitespace is meaningful there.

Same exact deal: it's good to have the *ability* to normalize whitespace (in email, we call this "text=flowed" or similar), and in some ceses you might even want to make it the default action, but *forcing* normalization is total idiocy and actually makes the system less useful! ]

Linus

Maybe because it's 1.5 times bigger for any text in chinese, japanese or korean ?

Mike

Code point is a unique numerical value assigned to every Unicode character. Also, every Unicode character has a uniqie name assigned to it. There are some other non-unique properties that every Unicode has. So, to say that a Unicode character is just a code point is not exactly correct, because the code point is one of properties of a unicode character. But, yes, any Unicode character can be identified by its code point. So, it is one to one relation.

Dmitry

Linus,

(slightly offtopic) are you praising UTF-8 as storage format (for disk and network) or in general? UTF-8-aware string ops like counting characters seem to me a horrendous thing at the ASM level.

More on topic, I suspect Kevin's experience is more on end-user apps, where input sanitization and even canonicalisation are common practice. From a kernel and filesystems POV, a filename is data as sacred as file data. On the webapp world, we "corrupt" user input liberally to avoid XSS attacks and the like. In some cases, these practices are stupid and can be replaced with escaping data properly, but in other cases, the web platform is so broken that there's no option.

At least in Moodle we store *exactly* what the user POSTed and cleanup^Wcorrupt it when displaying it, so that if it does happen that the cleanup was buggy, we never corrupted the data.

So no point in calling eachother stupid this much. Once is enough ;-) And no point in arguing that something that is ok for an end-user app is a good design decision for an OS.

martin

Huh? Why? Just count all characters in the range 00-bf. That's the exact character count of utf-8 characters.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

I'm praising UTF-8 (without normalization) as a wonderful format where you can do 99.9% of everything without ever caring about all the expensive stuff.

But in order to do that, you really need to avoid normalization, and you also need to accept mis-formed UTF-8 strings (because even if it is real UTF-8, the string may actually be just a fragment of some larger string).

Once you do that (and _only_ if you do that), then UTF-8 is actually a wonderful thing. You can consider it to be a traditional "everything is a stream of bytes", and everything that only cares about a stream of byte will work wonderfully well.

And then, the (actually relatively few) things that want to do things like show things on the screen, or check for equivalence, or worry about width of the characters, *those* can still do so.

So the beauty of UTF-8 is that you can switch between thinking of it like just a binary blob and thinking of it like text, and everythign works (including the traditional C null-termination).

And yes, that was obviously the explicit design goal. It's a good thing.

Sure. And I'm not arguing against them. Knowing the rules for combining characters is really important for input and output.

Absolutely. It's what the kernel does, and I think that's what perl does too for their "strings". It works really well. It also allows you to handle binary data (ie data that *really* isn't text) with shared routines etc etc.

And that's the beauty of non-normalized (and possibly badly formed) UTF-8.

Linus

*thanks* for these notes. Very useful, and...

...

I find the above amusing -- different worlds we live in. Programming webapps means that 90% of the code deals with a bit of metaprogramming (with lots of string manipulation) to talk SQL to a backend, and then doing lots of string manipulation on the data the DB returns, which ends up in humongous strings of goop otherwise known as HTML+CSS+JS. After waiting for the DB to return data, over 50% of cpu time is spent in regexes, concatenations, counting words, array ops, etc. So it is pretty significant.

So now I have to worry about cost and correctness of stuff that I took for granted in the pre-unicode days - strtolower() can be quite expensive and... buggy! But that's mainly due to Unicode, not UTF8. I think the only slowdown I can pin on UTF-8 is in counting chars, and probably slower regexes. Not that I deal with the C implementation of any of this stuff -- and so happy about it! ;-)

</offtopic>

(...)

I had a few issues with Perl v5.6's utf-8 handling that wasn't binary safe (fread() to a fixed-length buffer would break the input if a unicode char landed across the boundary - ouch!) -- made me think that you couldn't do this in binary safe ways. So I tend to tell Perl to treatfiles as binary, and switch to utf-8 in specially chosen spots. I suspect that 5.8 is a bit saner about this, but I'm not taking chances.

cheers,

martin

I could say the same thing about you.

I'm not saying it's forced on you, I'm saying when you treat filenames as text, it DOESN'T MATTER if the string gets normalized. As long as the string remains equivalent, YOU DON'T CARE about the underlying byte stream.

Alright, fine. I'm not saying HFS+ is right in storing the normalized version, but I do believe the authors of HFS+ must have had a reason to do that, and I also believe that it shouldn't make any difference to me since it remains equivalent.

Sure it does. Normalizing a string produces an equivalent string, and so unless I look at the octets the two strings are, for all intents and purposes, the same.

You're right, but it doesn't have to treat it as a binary stream at the level I care about. I mean, no matter what you do at some level the string is evaluated as a binary stream. For our purposes, just redefine the hashing algorithm to hash all equivalent strings the same, and you can implement that by using SHA1 on a particular encoding of the string.

Decomposing and recomposing shouldn't lose any information we care about - when treating filenames as text, a<COMBINING DIARESIS> and <A WITH DIARESIS> are equivalent, and thus no distinction is made between them. I'm not sure what other information you might be considering lost in this case.

I don't believe you. See below.

When have I ever said that Unicode meant Forced normalization?

Wrong.

Wrong. '\x61\x18' in Latin1, when converted to UTF-8 (NFD) is still '\x61\xc2\xa8'. You're mixing up DIARESIS (U+00A8) and COMBINING DIARESIS (U+0308).

I suspect this is why you've been yelling so much - you have a fundamental misunderstanding about what normalization is actually doing.

See above as to why you're not losing the information you so fervently believe you are.

People who insult others run the risk of looking like a fool when shown to be wrong.

Sure, it all depends on what level you need to evaluate text. If we're talking about english paragraphs, then whitespace can be messed with. When we're talking about unicode strings, then specific encoding can be messed with. When talking about byte sequence, nothing can be messed with.

In our case, when working on an HFS+ filesystem all you have to care about is the unicode string level. The specific encoding can be messed with, and the client shouldn't care. Problems only arise when attempting to interoperate with filesystems that work at the byte sequence level.

The only information you lose when doing canonical normalization is what the original byte sequence was. Sure, this is a problem when working on a filesystem that cares about byte sequence, but it's not a problem when working on a filesystem that cares about the unicode string.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

That's horribly broken, for a couple of reasons. First of all, changing the hash algorithm breaks compatibility with existing repositories; sure, you can try to guess what will least likely break existing repository (which won't be the native MacOSX normalization algorithm, since it's more likely the combined character will likely be used on other environments), but there's still no guarantee there aren't filenames that use some other form of byte-string for the filename.

Secondly, the hash algorithm would not be stable. Unicode is not static, and new characters can get added that may be composable, and thus would be normalized differently. This is one of the reasons why Unicode is so horribly broken as a standard. It was originally created by representatives from the printing world that were horribly clueless about what was needed with respect to canonicalization representation, so they compromised allowed both forms, not realizing what a massive f*ckup this would cause later on. So people have over the years piled kludges on top of kludges in order to make Unicode "work".

So we can't blame all of the craziness on the MacOS designers, although they have seen to have been very creative about how to take a bad situation and make it worse....

- Ted

You seem to be under the impression that I'm advocating that git treat all filenames as unicode strings, and thus change its hashing algorithm as described. I am not. I am saying that, if git only had to deal with HFS+, then it could treat all filenames as strings, etc. However, since git does not only have to deal with HFS+, this will not work. What I am describing is an ideal, not a practicality.

In other words, what I'm saying is that treating filenames as strings works perfectly fine, *provided you can do that 100% of the time*. git cannot do that 100% of the time, therefore it's not appropriate here. The purpose of this argument is to illustrate that treating filenames as strings isn't wrong, it's simply incompatible with treating filenames as byte sequences.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Well, why are you arguing on the git list about precisely that (when you reponsed to Linus), then?

No, it's still broken, because of the Unicode-is-not-static problem. What happens when you start adding more composable characters, which some future version of HFS+ will start breaking apart?

Presumably the whole *reason* why HFS+ was corrupting strings was so that "stupid applications" that only did byte comparisons would work correctly. But when you upgrade from Mac OS 10.5 to 10.6, and it adds support for new composable characters, and you now take a USB hard drive that was hooked up to a MacBook Air, running one version of MacOS, and hook it up to another Macintosh, running another version of MacOS, the normalization algorithm will be different, so the byte comparisons won't work.

So all of this extra work which MacOS put in to corrupt filenames behind our back doesn't actually do any good; applications still need to be smart, or there will be rare, hard to reproduce bugs nevertheless. So if MacOS wants to supply Unicode libraries that compare strings keeping in mind Unicode "equivalences" it can be our guest (although how they deal with different versions of Unicode with different equivalence classes will be their cross to bear). BUT MacOS X SHOULD NOT BE CORRUPTING FILENAMES. TO DO SO IS BROKEN.

Even Microsoft got this right; its filesystem is case-preserving, but it has case-insensitive lookups. Hence, it is not corrupting filenames behind the application's back, unlike MacOS.

- Ted

Because of the way in which an argument evolves. This started out as "HFS+ is stupid because it normalizes", and I was arguing that said normalization wasn't stupid. This turned into an argument as to why HFS+ wasn't stupid for normalization, which is basically this argument of the ideal. Yes, I realize that it's not producing any practical results, but I'm stubborn (as, apparently, are most of you), and I believe that if the official stance of the git project is "HFS+ is stupid" then there's a lower chance of a patch being accepted then if people accept that "HFS+ is different in an incompatible fashion".

If you need a static representation, you normalize to a specific form. And in fact, adding new composable characters doesn't matter, since if they didn't exist before, you couldn't have possibly used them. Unless you mean adding new composed forms of existing simpler characters, at which point you seem to be arguing for NFD instead of NFC.

I doubt that HFS+ normalized so that "stupid applications" could do byte comparisons. But even if that were the case, see previous paragraph.

Your entire argument is based on the assumption that HFS+ "corrupts" filenames in order to allow dumb clients to do byte comparisons, and I don't believe that to be the case. In fact, it's only considered a corruption if you care about the byte sequence of filenames, and my argument is that, on HFS+, you aren't supposed to care about the byte sequence.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Sure you can. Suppose you unpack the same tar file or zip file that contains one of these new-fangled characters, one on a MacOS 10.5 system, and one on a MacOS 10.9 system. How HFS+ will corrupt that filename will differ depending which version of MacOS you are running. Hence, normalizing the filename when you store it is stupid and broken. MacOS and its applications and libraries want to do normalization in the privacy of its own address space, that's it's business. It can pursue any fetish it wants, among consenting adults. Safe, sane and consensual, and all that... well, consensual, anyway. I'm not sure about "safe" and "sane"....

My arguement is basically is that there is absolutely no value in what HFS+ is doing, by corrupting filenames --- if you want to call it "normalizing" them, fine, but since Unicode is not static, so you can't even call it a "canonical" form. It's just some random corruption of what was passed in at open(2) time, that can and will change depending on what version of MacOS you are running.

If you want to play the insane Unicode game of "equivalent" characters, you have to do it at comparison time, so there's no point trying to "normalize" them when you store them. It doesn't buy you anything, and it causes all sorts of pain.

OK, what's your reason for why HFS+ corrupts filenames? What do you think is its excuse? What problem does it solve? If the answer is "no reason at all, but because it *can*", according to the Great God Unicode, then that's really not very impressive....

- Ted

Note: resent to list due to bounce.
Original CC list: [email protected], [email protected], [email protected], [email protected], [email protected]

You're making the huge assumption that the HFS+ normalization algorithms will change. As the technote states:

"Platform algorithms tend to evolve with the Unicode standard. The HFS Plus algorithms cannot evolve because such evolution would invalidate existing HFS Plus volumes."

Again with the huge assumptions.

It must have bought somebody something, or they never would have done it.

I have no idea why HFS+ stores filenames in a normalized form, and further I am smart enough to know that speculating is completely pointless. I assume the authors had a good reason (which should be a safe assumption, filesystem authors are a smart bunch). The reason may not be valid anymore, but if it was valid back in 1998, then I can accept it without complaining.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Great, so even worse. Does the tech note then specify exactly what version of Unicode HFS+ is using to do its "normalization"? Or exactly what characters it will normalize? After all, Unicode has added all sorts of characters since 1998, and I'm sure some of them were combining characters.

And you *really* want to continue argue that a sane thing for a cross-platform system to do is to pervert its hash algorithm to take into account *one* particular OS that happened to freeze a normalization algorithm at some arbitrary point in time, approximately nine years ago? Talk about the tail wagging the dog!! Especially when you can't even justify why it was done nine years ago!

Your faith in the HFS+ designers is touching.

Well, I *AM* a filesystem designer (ext2/ext3/ext4), and well before 1998, I knew that trying to do anything with Unicode normalization was a fool's errand. So if you're going to blindly trust filesystme designers (not something I would recommend, actually :-), trust me. What HFS+ is doing is dumb, dumb, dumb.

And even if *you* can accept it, why should the git designers pervert any core part of git's design to support this behaviour? Especially if it's legacy behaviour which will hopefully be going away, say when MacOS adopts ZFS --- there's an opportunity for them to start afresh, and not make the same mistakes they made nine years ago!

So why don't you suggest some kind of sane fix in the Mac specific code that doesn't impact any core part of git, such as its hash algorithm? It would be far more productive than trying to defend a
bad design decision made nine years ago.... :-}

- Ted

I suggest you go back and read the emails where I specifically stated that I'm *not* suggesting this.

And your arrogance is troubling. Do you really believe you are so smart you can claim the HFS+ designers had no reason for this decision?

Again, I'm not saying that they necessarily did the "correct" thing, as I can't evaluate that without knowing their reason. I'm just saying there must have been a reason.

And why do you believe MacOS is going to adopt ZFS? Sure, they might, but assuming stuff about the future is just as bad as assuming stuff about the past. And git should "pervert" itself because of the simple fact that git has a problem on HFS+. Keeping your code "pure" is all well and good, except it's not particularly practical. If the git project has any interest in being a viable system on OS X, it really should behave properly. I'm sure you have various "perversions" for other cases.

How many times must I say I never suggested actually changing git's hashing algorithm? And if you want me to suggest a fix to git that works, first you have to wait for me to learn how git's internals work, and frankly, I have too much work on my plate right now to devote the time necessary to learning git's internals well enough to fix this problem.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

LOL! Spare us the flamefesting and you will have plenty of time for learning git internals. You might even learn something.

cheers,

martin

Again with the bouncing. I gotta figure out how to fix this.
Original CC list: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Ah, so I'm flaming while you are providing a well-reasoned and articulate argument? Glad to know the difference.

In any case, you should be very familiar with the fact that writing emails and learning code are two vastly different activities that require vastly different amounts of concentration and time. I'm responding to email while doing other things - were I to replace the time spent writing email with learning git's internals, I would be pulled away so frequently that I would end up not having learned anything. Therefore, I write emails now, and I leave learning git's internals until later, when I have the undisturbed time to devote.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Hi,

ENOUGH ALREADY!

Yes, you are flaming. You sent easily over 30 totally useless mails in this thread. Over a couple of days.

And Martin is right, in that same amount of time, you could have learnt the internals of git _easily_. Especially since I provided an own chapter in the manual for people like you.

So instead of _PESTERING_ us with things that we do _NOT CARE_ about, you could _DO SOMETHING USEFUL_ instead.

For example, read up that chapter, and not type any _POINTLESS_ mails in that _UTTERLY POINTLESS_ thread anymore.

I hoped that subtle _HINTS_ would give you an _IDEA_, but _EVIDENTLY_ I have to use _ALL-CAPS_ on you.

So _EITHER_ read up _OR_ go away, but _DO NOT BOTHER_ to post _ANY_ response without patches in this _DARNED_ thread.

Sheesh,
Dscho

And so has EVERYONE ELSE. You cannot hold me to a standard which you yourself do not apply.

As I said before, I've been responding to emails in the midst of doing other things. So no, I can't learn an entirely new system in my off-time between other tasks, but I can respond to emails.

You sure argue a lot for something you don't care about.

I'm especially annoyed since I have made MANY offerings to stop this argument and work towards a solution, but NOBODY ELSE seems to care enough to actually accept that offer.

And treating me like a moron (using all caps?) just shows how bad you are at actually reading my emails. I've been giving you and everyone else the courtesy of actually reading your emails, glad to know nobody else has bothered to read to the end of mine.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Hi Kevin,

not sure if you are just joking, but perhaps you have not noticed that in technical lists like these, you get karma to voice strong opinions once you've contributed lots of good code. *That* is the standard - and Johannes has earned his karma points, as Ted and Linus have, by learning the slow way and writing tons of code. Alas, you don't seem to have time for such a thing!

Strong opinions without working code tend to not get much respect. So that is the standard that is applied to all of us, including you.

cheers,

martin

But there is no point in responding to emails as long as you don't have a clue what you are actually talking about.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

Hi,

While I was playing chess, unsuspectingly, there were 40 mails in this useless thread. You sent 15 of them.

Now, just making a _conservative_ guess, [email protected] has about 50000 subscribers (I know for a fact this figure is too low, so it is conservative).

I estimate all except for 10 of them, let's be conservative, 100, did not care about your lognorrhea. They needed at least 0.5 seconds to delete those mails _you_ are responsible for.

Congratulations, you cost at least 500 seconds today. And many nerves.

The sad thing: I read about two people the other day, Singh and McKinstry, who I'd rather see writing mails here.

I don't care if you honour my contributions to git. I don't care about you.

You must get a kick out of annoying people. Why don't you try to play chicken with a train?

Ciao,
Dscho

On Jan 22, 2008 2:34 PM, Johannes Schindelin <[email protected]> wrote:

Hey guys. Let's stop right here -- Kevin has perhaps been annoying but this is a *technical* argument, so let's go back to working code.

Anyone send a patch to clear the air? *please*?

m

Hi,

Not me. This thread was not _my_ fault.

And I'll be _damned_ if I encourage people to be a pain in the rear end, in order to get other people to write code/patches for them.

I think it is clear who has the obligation of contributing some _code_, for a change.

Ciao,
Dscho

No reason? Don't say where he says that. No sane reason? Certainly. If the visibility of the upsides is not in the same order of magnitude as that of the downsides (and your "I trust they must have had good reason" is implicating exactly that), then yes, this appears like a misdesign, however well-intended. Because its cleverness hinges on an what amounts to an arbitrary historic point of stability with only fleeting convenience.

It reminds me of the self-defeatingproblem haunting the MIPS (microprocessor without interlocked pipeline stages) architecture: for pipelined processors, one has to add logic that prevents one command from working before the results from other commands arrive. Now the ingenious idea of the MIPS architecture was to move that logic into the compiler instead of the hardware. But then the implications of that idea got intermingled with binary compatibility and the result was that the advantages lasted for one processor generation, and afterwards, the m-stage pipelines needed logic that simulated the n-stage pipeline of the first MIPS processor rather then a comparatively simple 1-stage pipeline of a conceptual sequential processor. Rendering the whole original idea completely absurd and requiring rather more complicated rather than simpler hardware as originally envisioned.

The road to hell is paved with good intentions.

There is no shortage in short-sighted decisions to repeat. Some political parties bank on it.

Jumping to blind faith-based conclusions is never a good move. You don't end up improving the work of your predecessors that way.

If OS X has any interest in being a viable system, perios, it really should behave properly.

Then please understand that you have too much work on your plate right now to devote the time necessary to provide any constructive criticism. A smart person in this situation would shut up until he has the time.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

A smart person would not join the conversation late and respond to points that have already been exhausted ages ago.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Find somebody willing to explain to you the difference between Email and IRC, and how to read "Date:" headers. I have no doubt you'll be able to grasp the basic involved principles in less than a week.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

to treat as text could mean different for different people. Some may prefer to fi and fi_ligature to be treated as same in some context.

As matter of fact it does, otherwise characters would be the same and we would not have this conversation at all. String can be equivalent and not equivalent at the time, because there are different equivalent relations. Finally, what HFS+ does is even not normalization. In the technote, Apple explains that they decompose some characters but not others for better compatibility. So, you see, there is a PROBLEM here.

It is not about byte stream. After all, if it were UTF-16 instead of UTF-8, it would be one to one conversion for each character. So, what gets corrupted by HFS+ are Unicode *characters*.

I don't say they do that without *any* reason, but I suppose all Apple developers in the Copland project had some reasons for they did, but the outcome was not very good...

Not true. You lose the original sequence of *characters*.

Dmitry

Those people can use NFKC/NFKD (compatibility equivalence). As I've said before, I'm talking about canonical equivalence, because that doesn't lose information like compatibility equivalence does (ex. the fi ligature gets turned into fi in compatibility equivalence, but not canonical equivalence).

Again, I've specified many times that I'm talking about canonical equivalence.

And yes, HFS+ does normalization, it just doesn't use NFD. It uses a custom variant. I fail to see how this is a problem.

Stupid engineers don't get to work on developing new filesystems. And Copland didn't fail because of stupid engineers anyway. If I had to blame someone, I'd blame management.

Which is only a problem if you care about the byte sequence, which is kinda the whole point of my argument.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

If you think that HFS+ does normalization then you apparently have no idea of what the term "normalization" means. Have you? But if you don't know what is "normalization" then you cannot really know what canonical equivalence means.

Assigning someone to work on a new filesystem does not make him suddenly smart. As to that stupid engineers don't get to work, it is like saying there is no stupid engineers at all. There are plenty evidence to contrary. And when management is disastrous then most idiots with big mouth and little capacity to produce any useful does get assignment to develop new features, while those who can actually solve problems are assigned to fix the next build, because the only thing that this management worries about how to survive another year or another months...

But if the code was so good then why was most of that code thrown away later when management was changed? Still bad management?

Byte sequences are not an issue here. If the filesystem used UTF-16 to store filenames, that would NOT cause this problem, because characters would be the same even though bytes stored on the disk were different. So, what you actually lose here is the original sequence of *characters*.

Dmitry

I would go look up specifics to back me up, but my DNS is screwing up right now so I can't access most of the internet. In any case, there are 4 standard normalization forms - NFC, NFD, NFKC, NFKD. If there are others, they aren't notable enough to be listed in the resource I was reading. HFS+ uses a variant on NFD - it's a well-defined variant, and thus can safely be called its own normalization form. I fail to see how this means it's not "normalization".

I'm not talking about assigning engineers, I'm saying developing a new filesystem, especially one that's proven itself to be usable and extendable for the last decade, is something that only smart engineers would be capable of doing.

Yes. Even the best of engineers will produce crap code when overworked and required to implement new features instead of fixing bugs and stabilizing the system. Copland is well-known to have suffered from featuritis, to the extent that it was practically impossible to test in any sane fashion. Bad management can kill any project regardless of how good the engineers are.

I've already talked about that, but you are apparently incapable of understanding.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Then you are lucky that your mails reach this ML without problem.

The defining property of normalization is producing binary identitical strings for equivalent strings, IOW, normalization allows you to tell what strings are equivalent and what are not just by binary comparision. HFS+ decomposition lacks that property, because strings are not fully decomposed thus being comparision of equivalent strings may give false result.

You know, many people still use FAT, but somehow I don't think that FAT is good despite of it being extendable for more than a decade... Apparently, HFS+ was not worst part of the Copland project, but I see no evidence to think that it was developed by the best engineers.

I don't think that anyone asked them to implement so much new features. AFAIK, it was very difficult (nearly impossible) to get anyone to work on stabilizing existing software and fixing existing bugs in it.

Exactly. IMHO, both management and developers are equally responsible for that feature-mania.

Sure.

Well, it is *you* who is incapable of understanding anything, even basic terms as encoding and normalization...

Dmitry

Sure I do, because it matters a lot for things like - wait for it - things like checksumming it.

I've already told you the reason: they did the mistake of wanting to be case-independent, and a (bad) case compare is easier in NFD.

Once you give strings semantic meaning (and "case independent" implies that semantic meaning), suddenly normalization looks like a good idea, and since you're going to corrupt the data *anyway*, who cares? You just created a file like "Hello", and readdir() returns "hello" (because there was an old file under that name), and it's a lot more obviously corrupt than just due to normalization.

.. but you *have* to look at the octets at some point. They're kind of what the string is built up of. They never went away, even if you chose to ignore them. The encoding is really quite important, and is visible both in memory and on disk.

It's what shows up when you sha1sum, but it's also as simple as what shows up when you do an "ls -l" and look at a file size.

It doesn't matter if the text is "equivalent", when you then see the differences in all these small details.

You can shut your eyes as much as you want, and say that you don't care, but the differences are real, and they are visible.

You're right, I messed up. I used a non-combining diaeresis, and you're right, it doesn't get corrupted. And I think that means that if Apple had used NFC, we'd not have this problem with Latin1 systems (because then the UTF-8 representation would be the same).

So I still think that normalization is totally idiotic, but the thing that actually causes most problems for people on OS X is that they chose the really inconvenient one.

Linus

I believe I already responded to the issue of hashing. In summary, just re-define your hash function to convert the string to a specific encoding. Sure, you'll lose some speed, but we're already assuming that it's worth taking a speed hit in order to treat filenames as strings (please don't argue this point, it's an opinion, not a factual statement, and I'm not necessarily saying I agree with it, I'm just saying it's valid).

Perhaps that is the reason, I don't know (neither do you, you're just guessing). However, my point still stands - as long as the string stays canonically equivalent, it doesn't matter to me if the filesystem changes the encoding, since I'm working at the string level.

Someone has to look at the octets, but it doesn't have to be me. As long as I use unicode-aware libraries and such, I can let the underlying system care about the byte order and my code will be clean.

It does? Why on earth should it do that? Filename doesn't contribute to the listed filesize on OS X.

kevin@KBLAPTOP:~> echo foo > foo; echo foo > foobar kevin@KBLAPTOP:~> ls -l foo*

-rw-r--r--  1 kevin  kevin  4 Jan 21 14:50 foo
-rw-r--r--  1 kevin  kevin  4 Jan 21 14:50 foobar

It would be singularly stupid for the filesize to reflect the filename, especially since this means you would report different filesizes for hardlinks.

Visible at some level, sure, but not visible at the level my code works on. And thus, I don't have to care about it.

I'm not sure what you mean. The byte sequence is different from Latin1 to UTF-8 even if you use NFC, so I don't think, in this case, it makes any difference whether you use NFC or NFD. Yes, the codepoints are the same in Latin1 and UTF-8 if you use NFC, but that's hardly relevant. Please correct me if I'm wrong, but I believe Latin1->UTF-8->Latin1 conversion will always produce the same Latin1 text whether you use NFC or NFD.

The only reason it's particularly inconvenient is because it's different from what most other systems picked. And if you want to blame someone for that, blame Unicode for having so many different normalization forms.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Umm. What's this inability to see that data is data is data?

Why do you think Unicode has anything in particular to do with filenames?

Those same unicode strings are often part of the file data itself, and then that encoding damn well is visible in "ls -l".

Doing

echo ä > file
ls -l file

sure shows that "underlying octet" thing that you wanted to avoid so much. My point was that those underlying octets are always there, and they do matter. The fact that the differences may not be visible when you compare the normalized forms doesn't make it any less true.

You can choose to put blinders on and try to claim that normalization is invisible, but it's only invisible TO THOSE THINGS THAT DON'T WANT TO SEE IT.

But that doesn't change the fact that a lot of things *do* see it. There are very few things that are "Unicode specific", and a *lot* of tools that are just "general data tools".

And git tries to be a general data tool, not a Unicode-specific one.

The problem is that the UTF-8 form is different, so if you save things in UTF-8 (which we hopefully agree is a sane thing to do), then you should try to use a representation that people agree on.

And NFC is the more common normalization form by far, so by normalizing to something else, you actually de-normalize as far as those other people are concerned.

So if you have to normalize, at least use the normal form!

I blame them for encouraging normalization at all.

It's stupid.

You don't need it.

The people who care about "are these strings equivalent" shouldn't do a "memcmp()" on them in the first place. And if you don't do a memcmp() on things, then you don't need to normalize.

So you have two cases:
(a) the cases that care about *identity*. They don't want normalization
(b) the cases that care about *equivalence*. And they shouldn't do octet-by-octet comparison.

See? Either you want to see equivalence, or you don't. And in neither case is normalization the right thing to do (except as *possibly* an internal part of the comparison, but there are actually better ways to check for equivalence than the brute-force "normalize both and compare results bitwise").

Linus

I'm not sure what you mean. I stated a fact - at least on OS X, the filename does not contribute to the listed filesize, so changing the encoding of the filename doesn't change the filesize. This isn't a philosophical point, it's a factual statement.

I don't, but I do think this discussion revolves around filenames, therefore it should not surprise you when I talk about filenames.

Yes, I am well aware that the encoding of the *file contents* affects filesize. But when did I suggest changing the encoding of filenames inside file contents? If you treat filenames as strings, there's no requirement to change the encoding of filenames inside file contents. I'm talking specifically about the filenames, not about file contents, so stop trying to argue against that which is irrelevant.

Don't want to, or don't need to? It's not a matter of ignoring encoding because I don't want to deal with it, it's ignoring encoding because it's simply not relevant if I treat filenames as strings.

Yes, I realize that. See my previous message about discussing ideal vs practicality.

Was NFC the common normalization form back in 1998? My understanding is Unicode was still in the process of being adopted back then, so there was no one common standard that was obvious for everyone to use.

I could argue against this, but frankly, I'm really tired of arguing this same point. I suggest we simply agree to disagree, and move on to actually fixing the problem.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Kevin,

as you might know, Linus' "other hobby" is to write kernels ;-) From taht POV, a filename is as much data as the data in the file. Doing odd things like sorting it, searching through it, etc, is all work for code higher in the stack that is free to mangle the data in any way it wants, including creating nice case-insensitive indexes, and who-knows-what for ideogram-based languages. In contrast, the core OS treats user data a sacred stuff, and I'm thankful it does.

And from a kernel/filesystem POV, a directory is also a file. So if a filename has a different number of octets, the directory will be different.

For all the searching and matching, it really makes sense to have something like locate or SpotLight or whatever to index user files that should be easy to find and match, because all the locale rules for matching are hideously expensive to apply. Even today, most UTF-8 aware (and supposedly collation-smart) applications have trouble matching MARTÍN when asked for martín in a case-insensitive search. That pesky latin í trips them up everytime.

cheers,

martin

That's certainly a reasonable POV. However, it's not the only one. As evidenced by the Mac, treating filenames as strings rather than bytes is a viable alternative POV - you can't argue that it doesn't work, because OS X proves it does.

However, it is a trade-off.

Sure, that makes sense. That's why, if you are going to mangle filenames, you need to pick a stable form to always use, which HFS+ does.

Perhaps you should try OS X. Every single Cocoa app should do the search properly. In fact, I just checked using 3 different text engines (WebKit, Cocoa's text engine, and ATSUI) and all 3 did the case-insensitive search properly. That said, this isn't particularly relevant.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

With its own slew of bugs. See Ted's reply earlier for a mouthful of woe in HFS+ that is not easy to workaround.

OSX has given me enough grief with other filesystem and general OS problems that I have definitely abandoned it after 2 years trying to use it part-time. It has been back to linux for me ;-)

cheers,

martin

Changing the encoding of the file name most certainly changes the file size of the directory.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

And my point was that your *whole* argument boils down to "normalization is invisible".

When it isn't. It's not invisible for filenames, it's not invisible for file contents.

You're trying to claim that normalization cannot matter. I'm just pointing out that it sure as hell can. Exactly because lots of things don't actually look at data other than as just a Unicode string. They do look at the raw format.

And that's true both of file contents and file names.

I'm surprised that you make generalized sweeping statements about how it's ok to normalize because normalization is "invisible", and then when I point out that that isn't true, you try to limit it.

And no, that normalization is not invisible EVEN IN FILENAMES. If it was, git wouldn't ever have noticed it, would it?

I don't know which argument you're talking about. Git (and, btw, Linux) does the "ideal" thing (don't screw up peoples data), and it turns out to be the "practical" thing too (it can handle a wider range of cases than OS X can).

So no, this is not "ideal" vs "practical". They aren't in any conflict here.

.. and people have even suggested how. Hide the idiotic OS X choices by making a OS X-specific wrapper around readdir() that turns it into NFC.

That's just about the best we can do. We can't *fix* the thing that OS X loses information, but a least we can then show the lost information in the same form it _probably_ was in originally.

But no, it won't "fix" git on OS X.

Linus

I'm really surprised that, after all of this, you're still horribly misunderstanding my argument. I never said it was invisible. NEVER.

I'm also surprised that you seem to care more about this argument then my offer to stop arguing and work towards fixing the problem.

You misunderstand my point. In a previous email I specifically used the words "ideal" and "practical" to describe arguments, which is what I was referring to here.

And I've responded to that suggestion, multiple times, saying that this doesn't actually fix the problem, it only hides it.

Quite a while ago it was suggested that git uses a table that maps the original byte sequence as seen in the index to the form returned by readdir(). So far this has sounded like the best solution, but as I've said before I don't know git's internals enough (or, really, at all) to be able to work on this myself.

This solution should only "lose" information in the case where the index has 2 filenames that HFS+ treats as a single filename.

Is there some reason this won't work?

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

You said it was invisible when you treat things "as text". Here's the quote:

.. when you treat filenames as text, it DOESN'T MATTER if the string gets normalized ..

Without ever apparently realizing that "as text" is part of the problem in itself. What is "text" to one person is gibberish to another.

In particular, the biggest reason to not normalize is that you don't know it's text or Unicode in the first place. Which is why git doesn't do it.

And no, even with filenames you don't know that they are "text". People encode stuff in them. And people don't always use UTF-8.

Of course, you could ask everybody to create OS X-only programs that know that under OS X, you only have a subset of filenames. If so, you're complaining about the wrong tool. Especially when the whole point of the tool was to be distributed (not to mention coming from an environment that simply doesn't have the same silly limitations OS X has).

So here's a few clues:

- "as text" isn't "as unicode": it may well be Latin1 or EUC-JP or something. Yes, it's still used. Git doesn't care, and very consciously has avoided forcing character sets, even if the *default* (and notice how it's overridable) commit message encoding may be utf-8.

- In fact, even in unicode, the difference between "identical" and "equivalent" strings exists, and even in the standard, unicode strings are very much defined to be arbitrary codepoint sequences, not normalized.

So even for the very specific case of unicode text, it's simply not true that "it doesn't matter if the string gets normalized". The unicode spec itself talks about cases where even canonical normalization makes a difference.

Search for this quote:

"Not all processes are required to respect canonical equivalence. For
example:

* A function that collects a set of the General_Category values present in a string will and should produce a different value for <angstrom sign, semicolon> than for <A, combining ring above, greek question mark>, even though they are canonically equivalent.
* A function that does a binary comparison of strings will also find these two sequences different."

and notice that first case. Even things that are *very*much* aware of Unicode text do actually have cases where canonical equivalence doesn't mean crud.

Linus

Which is actually a good argument as to why filenames should be enforced as UTF-8.

Sure, I understand why git doesn't do it. I'm saying in a system which uses unicode top-to-bottom, which you can create if you're using HFS+ only, can do it. On HFS+ you know the filename is unicode.

Again, I was talking about a system that used unicode top-to-bottom. On HFS+ you have to use UTF-8 for your filename or it simply won't work.

I find it amusing that you keep arguing against having git treat filenames as unicode when, if you had actually taken my advice and read my previous email talking about "ideal" vs "practical", you'd realize that I was not suggesting git should. I was simply describing why having the filesystem specifically treat filenames as utf-8 isn't a problem when the entire system is unicode-aware, and thus showing how the problems that are cropping up in git aren't because the filesystem treats filenames as unicode, but rather because the filesystem treats filenames differently than other filesystems. In other words, I was trying to illustrate that HFS+ isn't wrong, it's just different, and the difference is causing the problem.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

If by "ideal" you mean a world where 100% of all computers were designed by Steve Jobs, you might have a point. But trying to argue for such a state of idealism seems to be stupid, and certainly a complete waste of everyone's time on the git mailing list. It's simply not reality. It's like with the infamous resource forks, which would have worked fine if all the world were MacOS, but which had a tendency to get stripped off whenver you used a program that wasn't resource fork aware, like zip, or a protocol that wasn't resource fork aware, like FTP. And so people had to put in all sorts of kludges like BinHex to work around MacOS's "if only the entire world was like *me*, no one would get hurt" attitude. In some ways, the MacOS designers are even worse than Microsoft in terms of having the "the world revolves around us" attitude.

And if you want to interoperate with the rest of the world, where at least count over 92% of computers are NOT running HFS+, then "Thinking Different" is indeed causing the problem, yes. And whose fault is that?

The whole point of interoperability is that when we communicate, we have to do so in a uniform and predictable way. If we can't, the next best thing is to have protocol translators; but in order to do that, we must avoid lossy transformations, such as HFS+'s pseudo-normalization. (Why, by the way, will not result in a "normal" form for any glyph which can be encoded with and without a combining character if said glyph was introduced into Unicode after 1988. So you can't even call it a "normalization" algorithm, but just a pseudo-normalization transformation which is lossy and which DESTROYS filename information in an irrecoverable way.)

- Ted

NO NO NO NO NO. READ MY EMAIL. STOP MAKING ASSUMPTIONS ABOUT WHAT I'M TALKING ABOUT.

The most frustrating thing about this thread is everybody keeps arguing about what they *assume* I'm talking about without actually bothering to read what I'm saying.

And if you want to interoperate with the rest of the world, where at least count over 92% of computers are running Windows, then using another OS is stupid, right? Right? I mean, if everyone else is doing it, we should too, shouldn't we?

Sure it's normalization, it's just not using one of the standard forms. But the form is well-defined.

And yes, protocol translators are a good idea. That's why I thought the original suggestion of using a table to map index filenames <-> HFS+ filenames sounded like it could work. The only time that should fail is if the index contains multiple filenames that HFS+ will treat as a single filename. Is there a problem with this approach?

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Hmmm. I m pretty sure HFS+ has a lot of problems if you run OSX as an NFS server with clients in different encodings. It would never work in real life. The "envelope" OSs have to work in is hugely varied -- much more so than any other apps. You should try writing one someday ;-)

Did you spot the rather nasty issues that Ted mentioned earlier in the thread? I would say HFS+ is a bit "special" rather than "different".

cheers,

m

Right. If you are defining the requirements for a new FS on a new OS, would you not include a requirement that says "must not add any funny rule that prevents clean interoperation with other filesystems or OSs"? Forgetting that requirement is... a big one! And if someone asks "how do we do nice user-friendly filename matching with these technical differences that users mostly don't care about"... wouldn't you say "do it in the GUI facilities, changing the FS to handle this is wrong because it will break the OS as a server, as a reliable file storage"?

FSs have pretty hard requirements these days -- all the modern FS you've heard about respect the requirement above, and a ton more that you have to be in the FS business to be aware of. Mostly anyway, wherever they don't, users have all sorts of trouble.

Well, Ted answered that part, noting that then the "normalisation" is patchy, and everyone else is left to guess what chars are normalised and what chars aren't so being HFS+ compatible becomes a very weird game indeed. You didn't reply to his explanation -- you called him arrogant instead. Did you manage to read to the end if his email?

The HFS+ designers mucked it up -- and then papered over it with the OSX libraries. But a good chunk of the world does not use them, they forgot about the little "interop" requirement.

cheers,

m

Sure, but you have to remember, HFS+ was developed back for Mac OS 8, which really wasn't a very good server machine.

I've read every single email in this thread, all the way through. Ted was arguing against calling it "normalization". If you want to argue that it's using a non-standard normal form, go ahead, but surely you can figure out that can you simply re-normalize it to whatever form you want.

Sure, maybe they did forget about interop. Or maybe they developed this back on Mac OS 8 where the only real competitor was Windows, and they didn't have to worry about the Mac being used as an NFS server, and thus interop wasn't even a requirement.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Yet another bounce.
Original CC: [email protected], [email protected], [email protected], [email protected], [email protected]

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

NO I DO NOT!

Dammit, stop this idiocy.

I think it's fine having git treat filenames "as unicode", as long as you don't do any munging on it.

Why? Because if it's utf-8, then treating them "as unicode" means exactly the same as treating them "as a user-specified string".

So stop lying about this whole thing. I have never *ever* argued against unicode per se.

All my complaints - every single one of them - comes down to making the idiotic choice of trying to munge those strings (not even strictly "normalize") into something they are not.

And what you don't seem to understand is that once you accept _unmodified_ raw UTF-8 as a good unicode transport mechanism, suddenly other encodings are possible. I'm not out to force my world-view on users. If they are using legacy encodings (whether in filenames *or* in commit texts or in their file contents), that's *their* choice.

I actually personally happen to use UTF-8-encoded unicode.

I'm just not stupid enough to think that (a) corrupting it is a good idea, *or* (b) that I should force every Asian installation of git to also force people to use unicode (or even having all the conversion libraries and overheads!)

So stop this idiotic "unicode == normalization" crap.

I'm a huge fan of UTF-8. But that does not mean that I think normalization is a good idea.

Linus

Please read to the bottom of this email. As near as I can figure out, you haven't done that on any of my previous emails.

When I say "treat filenames as unicode" I'm implying the equivalence comparisons and everything else that we've been talking about.

If that's what "as unicode" meant, then the phrase "as unicode" has zero meaning.

No, you've argued against unicode equivalency in filenames. Can't you figure out, when the entire time I've been talking about equivalency, that I'm *still* talking about equivalency?

Yes, I understand quite well that you are against munging strings.

You're not using raw UTF-8, you're just using raw bytes. Calling it UTF-8 doesn't mean anything, since you don't actually know that's what it is. But this is fairly irrelevant.

How many times must I say the same thing over and over? I'm not arguing that forced normalization is a good thing. I'm arguing that, in a system which is unicode-aware top to bottom, forced normalization is irrelevant to the user, since they don't care about the exact byte sequence. And I'm also arguing that git should have some solution to this problem. I find it interesting that you're perfectly happy to rant and rail against your misperception of my argument, and yet you consistently and repeatedly ignore my offers to stop this argument and work towards a solution, as well as my comments on existing proposed solutions.

Are you even reading to the end of my emails?

- Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Yes, because you're an idiot.

I've told you over and over again that equivalence is stupid.

It's stupid when it's "equivalent except for case", and it's stupid when it's "canonically equivalent".

I agree: normalization and equivalency is idiotic.

But the two actually go hand in hand:

You don't seem to.

The thing is, the two are inexorably intertwined. Any filename equivalence (except for the trivial "identity" equivalence) INVARIABLY means that filenames get munged.

Why?

Think about the file name "Abc", and think about what happens when you create it.

Now, think about what happens if that filename is considered equivalent in case..

See? The filesystem has to *corrupt* the filename.

Can you not UNDERSTAND this? Equivalence and normalization is STUPID. It's just two sides of the exact same coin. They both INVARIABLY cause the filename to be munged.

And changing user data is not acceptable.

Do you get it now?

Linus "probably not" Torvalds

Let me make this really clear, because I'm afraid that you won't get it when I leave out any steps of the way.

Let us say that there is a filename "xyz" that is equivalent to a filename "abc" in *any* way. It does not matter if xyz/abc is Hello/hello, or whether it's two canonically equivalent strings.

So now, do

	close(open(xyz, O_WRONLY | O_CREAT, 0666));
	close(open(abc, O_WRONLY | O_CREAT, 0666));

and then look at the directory contents afterwards.

There are two, and only two, choices here (*):
- the filesystem created both files, and they show up as created
- the filesystem decided they were equivalent, and munged one (or both) of them

Now, let's go back to my claim:
- munging user data is unacceptable
and realize that equivalence BY DEFINITION must do it.

So no, you do *not* get to have your cake and eat it too. You simply fundamentally *cannot* have both filename equivalence and a non-munging filesystem. See above why.

Linus

(*) Actually, there is third choice above, which is:

- the filesystem created the first file, and errored out on the second because it noticed it was equivalent - but not identical - to one it already had

This one is actually a perfectly fine choice, but it's not "your" kind of equivalence, since it actually makes a difference between two equivalent but non-identical names. So the filenames aren't actually interchangable, and this case is really more of a "the filesystem has some very specific limitations on what it allows".

Linus, have you even bothered to read my arguments, or do you just get a kick out of building these straw man arguments? You have consistently failed to actually address what I'm talking about, and instead persist in explaining stuff I already know, as if that was the answer to anything I've been talking about. You are clearly incapable of understanding my basic point, no matter how simple I break it down. I suspect it's because you've been working low-level so long you can't think high-level, and so you manage to misinterpret my high-level arguments as boneheaded low-level mistakes.

Anyway, please see my countless former emails where I ask to work towards a solution instead of just arguing.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

We know what the solution is:

- The OS X filesystem _is_ crap (and you seem to have almost admitted as much by your comment that the HFS+ designers did it back in the dark ages and didn't mean for it to ever be a server filesystem anyway)

- But we can at least make a wrapper around readdir() return the NFC form on OS X, and effectively hide much of the fallout from the crap.

There is no way around it. Your "solutions" all seem to boil down to asking git to do the same idiotic crap that OS X does, taking all the same performance hits, and just generally doing crap just to work around crap in your favourite OS.

And no, making git be stupid just to suit a stupid filesystem simply isn't going to happen.

So how about you see _my_ point instead: OS X may have an inferior filesystem, but we don't have to make git inferior just for that. The fact that OS X does case independence is *its* problem, not git's.

Linus

I agree that HFS+ isn't well suited for tasks which it is being asked to do. I was never arguing that it was the perfect filesystem. But that hardly matters now, I know nobody's going to bother understanding my argument so I may as well just stop trying.

Again, I don't think that's the correct solution. What about the translation table that was suggested back at the beginning of the thread? That would solve the case insensitivity issue as well, whereas this NFC "solution" does nothing for that.

No, I am not asking git to do the same thing HFS+ does. You just persist in misinterpreting my arguments, no matter how many times I protest that this is not what I am saying.

So, what, you're saying git shouldn't do any work at all to try and behave nicer on OS X? Because OS X sure as hell can't change to suit git.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I did some research on this point, since if we really are going to be compatible with MacOS X's crappy HFS+ system, we need to know what the decomposition algorithm actually is. Turns out, there are *two* of them. Kevin didn't know what he was talking about. In fact, different versions of Mac OS X use different normalization algorithms.

Mac OS X 8.1 through 10.2.x used decompositions based on Unicode 2.1. Mac OS X 10.3 and later use decompositions based on Unicode 3.2.[1]

As I correctly predicted, Apple is changing their normalization algorithm in different versions of Mac OS X. It is not static, which meands there will be compatibility problems when moving hard drives between Mac OS X versions. I don't know if they try to fix this in their fsck or not, when upgrading from 10.2 to 10.3, but if not, certain files could disappear as part of the Mac OS X upgrade. Fun fun fun.

And clearly Kevin didn't read the tech note very carefully, since it clearly admits why they did it. The Mac OS X developers were being cheasy with how they implemented their HFS B-tree algorithms, and took the cheap, easy way out. So yeah, "crappy" is the only word that can be used for what Mac OS X perpetuated on the world. Because of that, a quick Google search shows it causes problems all over the stack, for many different programs beyond just git, including limewire and gnutella[2][3], Slim[4], and no doubt others.

[1] http://developer.apple.com/technotes/tn/tn1150.html#UnicodeSubtleties
[2] http://lists.limewire.org/pipermail/gui-dev/2003-January/001110.html
[3] http://osdir.com/ml/network.gnutella.limewire.core.devel/2003-01/msg00000.html
[4] http://forums.slimdevices.com/showthread.php?t=40582

In any case, it seems pretty clear that by now everyone except Kevin has realized that HFS+ is crappy and causes Internet-wide interoperability problems. So I'll justify sending this note by pointing out the specific table of Mac OS's filesystem corruption algorithm can be found here:

http://developer.apple.com/technotes/tn/tn1150table.html

I'd also recommend that the Mac OS X code try to either figure out whether it is running on an HFS+ partition, or let the HFS+ workaround code be something that can be controlled via .git/config. It shouldn't be on unconditionally even on a Mac OS X system, since if the git repository is on a ZFS or NFS filesystem, there's no reason to pay the overhead of working around the HFS+ bugs.

- Ted

I just finished talking to one of the HFS+ developers, so I suspect I know a lot more on this subject now than you do. Here's some of the relevant information:

* Any new characters added to Unicode will only have one form (decomposed), so HFS+ will always accept new characters as they will be NFD. The only exception is case-sensitivity, as the case-folding tables in HFS+ are static, so new characters with case variants will be treated in a case-sensitive manner. However, as they are already decomposed, the NFD algorithm will not change their encoding. This means that no, there are zero problems moving HFS+ drives between versions of OS X.

* At the time HFS+ was developed, there was no one common standard for normalization. The HFS+ developers picked NFD because they thought it was "a more flexible, future-looking form", but Microsoft ended up picking the opposite just a short time later. Interestingly, NFC is a weird hybrid form which only has composed forms for pre-existing characters, and decomposed forms for all new characters (as they only have one form). So in a sense NFD is more sane then NFC.

* The core issue here, which is why you think HFS+ is so stupid, is that you guys see no problem with having 2 files "Märchen" (NFC) and "Märchen" (NFD), whereas the HFS+ developers don't consider it acceptable to have 2 visually identical names as independent files. Unfortunately, the only way to do this matching is to store the normalized form in the filesystem, because it would be a performance nightmare to try and do this matching any other way. The HFS+ developers considered it an acceptable trade-off, and as an application developer I tend to agree with them.

As I have stated in the past, this isn't a case of HFS+ being stupid and causing problems, it's a case of HFS+ being *different* and causing problems. But this difference is just as much your fault as it is HFS+'s fault.

* For detecting case-sensitive filesystems you can use pathconf(2): _PC_CASE_SENSITIVE (if unsupported, you can assume the filesystem is case-sensitive). There is also the getattrlist(2) attribute: VOL_CAP_FMT_CASE_SENSITIVE.

There appears to be no API for determining if normalization will be applied. However, any filesystem that uses UTF-8 explicitly as storage (unlike the Linux filesystems, which you claim use UTF-8 but is obviously you really use nothing at all) is pretty much guaranteed to have to normalize or it will have abysmal performance.

I must say it is shocking that someone as smart as you is still more interested in finding ways to prove me wrong then to actually address the problem. It's obvious that the only research you did was intended to find ways to call me stupid.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

Except there *are* problems, because this promise doesn't apply to Unicode 2.1 (Mac OS 10.2 and before) and Unicode 3.2 (Mac OS 10.3 and above). And there were changes between the normalization algorithm between Unicode 3.2 and the Unicode version 4.1. So taking a hard drive between Mac OS X 10.2 and 10.3 *will* cause problems. The guarantees of Unicode stability didn't come until well past Unicode 2.1.

Also, I know of no guarantee that there will be no more new compositions. According to Unicode Stnadard Annex #15
(http://unicode.org/reports/tr15/), new characters that can be decomposed are strongly discouraged, but "It would be possible to add more compositions in a future version of Unicode". Got a reference to back up your claim that there will never be any more?

NFC is better if you care about compatibility with existing legacy character sets, where you want round-trip conversions to be idempotent. On the other hand, given that Mac OS has historically never cared about being compatible with the rest of the world, it makes sense that it would choose NFD.

Yep. No problems to do that. You seem to think that supporting Unicode requires imposing this constraint, but that's simply not true, except maybe in some kind of religious sense.

Nope. They were just not clever enough. If they use a hashed key for their b-tree and used a hash which had the property that two strings that were equivalent in the Unicode sense have the same hash value, it's quite possible to do Unicode-equivalence lookups quickly. Yeah, calculating the hash algorithm takes a bit amount of time, but it gets called no more than the normalization routine, and its performance overhead is no worse than the normalizing a string.

I know how to do it in a Linux filesystem; it's just an insane thing to do, and so I choose not to do it. But it is doable; if you must persue the course of filesystem insanity, it's possible to do it in a performant way, without normalization; it's the same way that you can use b-tree lookups in a case insensitive way.

No, I did the research to try to find the HFS-specific filename mangling algorithm. And given that's based on an back-level, old version of Unicode, you can't just use NFD algorithm from the latest Unicode spec. As I did that research, I came across the evidence that claims you had made (i.e., that HFS had never changed the Unicode version for its Normalization algorithm), was directly contradicted by the Apple TechNote.

- Ted

Don't ruin it. You were silent for 12hs and lots of patches and research on the problem started flowing. If you keep making a nuisance of yourself, people will turn from helping you to beating you up for being so annoying.

Perhaps help prepare those tests you said it was a good idea to work on. If you manage to stay silent a bit, we'll need them soon ;-)

m

Uh, Ted is a filesystem developer. I can't count the hours I spent talking with my father, a theoretical physicist, but that does not make me qualified to consider myself a better authority on physics than a sub-average actual grad student of the matter.

If you don't manage to check your arrogance eventually, you'll be causing more damage to your cause than if you just shut up. You make abundantly clear that you don't understand the _implications_ of the details you may or not may happen to find out.

David Kastrup, Kriemhildstr. 15, 44793 Bochum

One thing I'd like somebody to check: what _does_ happen with OS X and NFS (OS X as a client, not server)? In particular:

- Is it suddenly sane and case-sensitive?

- Does the NFS client do any unicode conversion?

I tried to google for it, but didn't find the right keywords to get anything useful out of that modern-day internet oracle.

Linus

Using a Linux server, and a OS X client, over NFS, it is in case-sensitive. This is not unexpected, since you can mount UFS partitions on Mac OS X, or reformat HFS+ filesystems and make them be case-sensitive.

Nope:

# perl -CO -e 'print pack("U",0x00C4)."\n"'  | xargs touch
# ls -l | cat -v
total 0
0 -rw-r--r--   1 nobody  nobody  0 Jan 22 20:30 M-CM-^D

It's pretty clear the Unicode conversion is being done in HFS+, not in the VFS layer of Mac OS X.

So presumably if and when Mac OS adopts ZFS, they will be able to be free of this mess, at least if they care about being compatible with Solaris.

- Ted

There must be something at the VFS layer, or some other layer:
- IIRC, Joliet iso9660 volumes end up being mounted with files names in NFS when the real file names are NFC on the disk.
- Likewise for Samba shares.
- When I had my problems with iso9660 rockridge volumes using NFC (you can create that just fine with mkisofs), the volume is mounted without normalisation, i.e. if you get to a shell and want to access files, you must use NFC, but at least the Finder does transliteration at some stage, because going into the mount point and opening some files fail because it's trying to open the file with the name transliterated to NFD. I just hope the same doesn't happen with other filesystems.

Also, OSX using NFD widely, a file created from non Unix applications may end up being named in NFD on any file system. File contents, too, may end up being transliterated whenever a file is modified with non Unix applications, introducing unwanted changes.
Typing file names in the Terminal might also make them encoded in NFD, too.

Mike

I assume you mean NFD, not NFS, but here's what one of the HFS+ engineers had to say:

"In Mac OS X, SMB, MSDOS, UDF, ISO 9660 (Joliet), NTFS and ZFS file systems all store in one form -- NFC. We store in NFC since that what is expected for these files systems."

See above.

Can you produce a reproducible set of steps for this? Because the Finder shouldn't be doing any of this work on its own, all the normalization stuff happens directly in HFS+.

Entirely possible, though renormalizing file contents seems a bit less likely. I will point out that the text input system in OS X seems to default to producing NFC (at least, typing echo 'Märchen' | xxd in the Terminal shows that the input string there is NFC). So user input will most likely produce NFC, the only way you're probably going to end up with NFD is if you move a file from HFS+ to another filesystem.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I wonder what happens if you do this:

touch 'Märchen'
echo M*rchen | xxd -g1

Will that produce NFC or NFD?

Dmitry

0000000: 4d 61 cc 88 72 63 68 65 6e 0a Ma..rchen.

This is NFC! Did you do that on HFS+?

If so, it means that shell on Mac also converts filenames to NFC when it reads them from the disk.

Dmitry

NFD, you mean ?

Mike

Oops, you are right.

Dmitry

That's the point. It's stored in NFC, but what applications see is NFD.

Simple : on a Linux host, create files with NFC names, and create an iso image with mkisofs, with rockridge but no joliet. Burn this to a disc, and insert the disc in your OSX host, and try to open files from the finder. Interestingly, IIRC, Finder is able to copy the files, though.

As a bonus, try the same with an iso volume name in NFC, it's even better :
the created mount point is NFD, but it tries to mount on the name in NFC and fails. And then you just can't eject the CD anymore.

Mike

Here's a reliable test case to test filename normalization on Mac OS.

------ cut here -------
cat > test.pl << EOF
#!/usr/bin/perl -CO
print "M".pack("U",0x00E4)."rchen\n";
print "Ma".pack("U",0x0308)."rchen\n";
EOF
chmod +x test.pl
./test.pl | xargs touch
echo M* | xxd -g1
------ cut here -------

On an NFS mounted filesystem, what you will get is this:

0000000: 4d 61 cc 88 72 63 68 65 6e 20 4d c3 a4 72 63 68 Ma..rchen M..rch
0000010: 65 6e 0a en.

and on an HFS+ mounted filesystem, what you will get is this:

0000000: 4d 61 cc 88 72 63 68 65 6e 0a Ma..rchen.

So this demonstrates that on my MacOS 10.4.11 system, on NFS, MacOS is doing no normalization, as it is creating two files. On HFS+, MacOS is mapping both filenames to the same decomposed name.

More (or not) surprisingly, given Kevin Ballard's "reliable source":

"In Mac OS X, SMB, MSDOS, UDF, ISO 9660 (Joliet), NTFS and ZFS file systems all store in one form -- NFC. We store in NFC since that what is expected for these files systems."

Using a Sony Reader (which uses an internal FAT filesystem) hooked up to a MacOS 10.4.11 system:

% /fs/u1/tmp/test.pl | xargs touch
% echo M* | xxd -g1
0000000: 4d 61 cc 88 72 63 68 65 6e 0a Ma..rchen.

.. which is the decomposed form. So it looks like on FAT/MSDOS filesystems MacOS 10.4.11 normalizes files to NFD, which will *not* do the right thing as far as Windows compatibility is concerned on USB sticks, et. al. Mac OS users would be well advised not to use non-ASCII names in their filesystems if they care about interoperating with other systems. :-P

- Ted

Well, it demonstrates that (a) the OS and (b) _perl_ don't mangle filenames on non-HFS+ filesystems.

The problem is that since most native applications *expect* that name mangling, they'll probably do name mangling of their own (internally) just to compare the names!

So I would not be surprised if the globbing libraries, for example, will do NFD-mangling in order to glob "correctly", so even programs ported from real Unix might end up getting pathnames subtly changed into NFD as part of some hot library-on-library action with UTF hackery inside.

Things like the finder etc, which must be very aware of the fact that filenames get corrupted, would presumably internally always convert everything they get into NFD in order to compare names from different sources. And as part of that, programs may well corrupt the name before they then use it to create a pathname.

The fact that your perl program works under NFS, but creates NFD on a VFAT volume, does imply that they probably used at least some of the same routines they use in HFS+ for VFAT. Not entirely surprising: doing case insensitive stuff with Unicode is nasty code, so why not share it (even if it's then incorrect for FAT)..

Piece of crap it is, though. Apple has painted themselves into a nasty corner there.

Linus

Well "touch" actually since that was what was actually creating the files; I only used perl because it was easist way to gaurantee exactly how the filenames would be generated.

It's worse than that. You can specify at format time whether or not HFS+ does case-sensitivity or not, and of course, there is UFS, which I expect does no Unicode normalization at all, much like NFS. I suspect what you've pointed out is why certain MacOS programs break horribly when run on non-HFS+ filesystems, though. And if that is the case, then those same programs might not be reliable if the user's home directory is stored on NFS --- like they would be in an enteprise/corproate environment, if Apple ever wants to have any hope of penetrating that market.

Because of this, git code won't be able to just check for HFS+; it will probably have to do a run-time test to see whether or not the filesystem is doing case-folding or not, since that can be turned on or off on a per-filesystem basis. Also unknown, and which should be tested, is whether turning off case-folding also turns off Unicode normalization. It may be that they did this so that HFS+ could be UFS compatible, since Darwin *must* be built on a UFS filesystem, reflecting its Mach/BSD heritage. (I ran across this while doing my web research; apparently HFS+ has been causing Apple headaches internally. Heh. :-)

Well, hopefully not everyone inside Apple's OS groups are total morons, and actually use a utf8_str_equiv() routine instead of strcmp() to do their Unicode comparisons. But then again, maybe not...

No kidding!!

- Ted

Well yes, any context in which a string is treated as Unicode instead of an opaque sequence of bytes will probably lead to normalization at some point (e.g. when searching text, I'm going to want Märchen and Märchen to be treated as the same string). The Mac OS X APIs use NFD, and everybody else uses NFC, but either way it's still normalization.

Why would the globbing libraries have to do anything special to understand NFD? In fact, I prefer that they don't - it's very handy to be able to type Ma* and have that match Märchen, as the globbing library sees Ma??rchen and is happy to match the ??rchen against *. Were the filename in NFC, I couldn't do that. Similarly, Ma<tab> autocompletes the name Märchen for me. But the convenience is beside the point - what I'm trying to show here is that if the globbing library were NFD-aware, it probably would decide Ma* shouldn't match Märchen, right?

I assume globbing libraries et al don't do UTF-8 hackery in Linux, right? And yet using NFC-encoded filenames is fairly common? So why should it be any different on OS X, especially since HFS+ isn't the only option here (and thus doing NFD conversion in the library would mess up other filesystems)?

In fact, probably the biggest reason the NFD-encoding was done at the HFS+ level is because they simply couldn't trust user-level libraries to always do the NFD conversion for pathnames. And I quote:

"I would prefer that case sensitivity and unicode normalization were not the responsibility of the file system -- but I realize that we cannot just ignore the problem and let the other layers sort it all out."

I don't get why you're still calling it corruption when, on an HFS+ system, NFD-encoding is correct. It would be corruption for HFS+ to write anything else but NFD.

There's no reason to assume that OS X is actually storing the NFD on the volume. In fact, it's quite explicitly not:

"As far as storing exactly what was passed in, its not just HFS that's involved her. In Mac OS X, SMB, MSDOS, UDF, ISO 9660 (Joliet), NTFS and ZFS file systems all store in one form -- NFC. We store in NFC since that what is expected for these files systems. If we were to allow KFD to pass through, it would cause problems when these names were accessed outside of Mac OS X. So this is not just an HFS issue but an interchange issue for Mac OS X. We have the legacy NFD use/expectation in our applications and we chose not to ignore the problem but make a conscience effort to have the appropriate form used (NFD in Mac OS X APIs, NFC elsewhere). Its not perfect but neither is the agnostic approach where both forms can be used and you can have duplicate filenames in your file system."

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

How about this: it's lossy. It's lossy in a similar sense that TIFF -> JPEG -> TIFF doesn't give you back exactly the same bytes, even though (modulo the compression level) the two TIFFs might be visually indistinguishable.

You seem to have an issue with calling this "corruption", but to most of us, if you have a system where you don't get back *exactly the same data* that you put in, then the data has been corrupted.

Now, please stop trolling this point, agree to disagree, and either contribute some code or be quiet and allow others to make progress.

j.

I hope you don't mind that I'm redirecting this back onto the list.

I don't understand how you can possibly think that disagreeing == trolling. Similarly, just because I'm the only person *on this list* who holds my viewpoint doesn't in any way mean I should abandon it. In fact, it makes it much more important that I continue to stand up for what I believe, The whole notion of democracy is based on the fact that every person is important, and that every person has the right to their own opinion. I realize this is a mailing list, not a democratic body, but the same principles should still apply. If your criteria for judging any viewpoint is purely how many people hold that viewpoint, then you end up ignoring things just because they are different or new.

I may be the single person defending this behavior on this list, but if you were to leave your comfortable linux community and talk to people elsewhere, you might find yourself in the minority opinion.

Why do you persist in thinking of this as right vs. wrong? I've tried to emphasize, many times, that HFS+ behaves this way not because it's "right" and ext4 is "wrong", but because HFS+ has a different set of values. The developers of HFS+ believed that, for a consumer OS like OS X, it made much more sense to treat visually indistinguishable filenames as the same file. I, and I'm sure the vast majority of OS X users, agree. Unfortunately this decision had some drawbacks, but they felt the trade-off was worth it. I'm well aware that you all don't think the trade-off was worth it, but like I said, this is a matter of behaving differently due to a different set of values, not behaving "right" or "wrong". I've been making an attempt to agree to disagree, but it seems that you would rather just squash dissent instead of accepting it.

I did let the point drop. Then you guys resurrected it. You can't pin this one on me.

Don't you think I'm frustrated at the behavior of everyone else here? But you don't see me flinging insults.

Collecting facts yourself is fine, but insulting anybody with a dissenting *opinion* simply because it's different is just plain wrong.

For the majority of this thread, nobody was making any indication that they cared at all about fixing this problem - that was my primary motivation to continue. If I had dropped this the first time someone told me to, do you think anybody would be working on the problem now?

As for dropping this conversation now, I'd love to. If you really want to drop it, I urge you to do just that - don't respond to this message. Read it, digest it, and then just let it sit. If this is the last message on the subject, that would be *wonderful*. But if you respond to this message then you have absolutely no ground to accuse me of refusing to drop it. So please, don't.

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

I would not have said that if I were you. That makes you look very bad. The impression I get after reading the above is that the only thing you care about is to have the last word in the thread.

People with opinions different from you could tone their message down and stick to a more neutral sounding statement, "This patch works around the issue X on HFS+", but not everybody is always nice-and-calm. But _you_ do not have to counter fire with fire, especially if your goal isn't to flame but is to resolve technical issues with cool head. As long as you do not get upset and start the flamewar every time whenever somebody says "This patch works around the issue only that broken crap HFS+ has due to its stupid filename corruption choice it made", when he could just have said it in a more neutral way, we can keep the conversation constructive and civilized.

Let me suggest an alternative, as I think this thread raged on long enough. When you read somebody says "HFS+ corrupts", "HFS+ is broken", "this works around the stupidity of HFS+", just take a deep breath, pretend that you did not hear these words that make you feel insulted. Instead pretend that you heard "HFS+ normalizes", "HFS+ is different", and "fixes problem on HFS+". Do not respond with "No it is not a corruption", "No, HFS+ is not broken" and "No, that is not a work around, but is a fix" with another long thread.

I can imagine a civilized conversation to go this way:

Linus: This patch would hopefully work around the stupid and broken normalization choice HFS+ people made years ago.

You: Ok, I tested that patch, and it does fix the issue for me on HFS+ for most cases, but I still have issues if I use character X, Y and Z.

Linus: Yeah, that is another direct consequence of the stupidity of HFS+. At this point I think the previous patch bends git backwards enough and I do not know if it is worth addressing by bending further...

You: How about introducing this new structure so that these cases can be handled in a way more friendly to HFS+, like this patch?

Linus: Yeah, I can buy that, it looks ugly but it would not hurt people on other systems.

Hmm?

Indeed. And it'd be good if Kevin could consider that this forum is for technical discussion - not democracy but meritocracy, best-solution-cracy and perhaps "fix-patch-ocracy". And that people that have written good code in the past, posted amazing patches, and wondrous test cases can sometimes get a bit more opinionated. But newcomers needs to earn a bit of respect before lecturing people.

Kevin, other people have already started posting nice nuggets of test cases. Where are *your* test cases? That would be a nice way to "have the last word" on this ;-)

cheers,

martin

I'm planning on devoting time this weekend to learning enough about git to be able to start hacking. I'm just too busy during the week to be able to devote the dedicated time necessary to this stuff. Hopefully I'll actually be able to start producing stuff this weekend.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

You do not need to learn much about git to post a test case. Only a few lines of shell code that demonstrate how git fails to handle a specific situation is needed. To do this, knowledge about the internals of git does not necessary help. It should be sufficient to know how to use git.

You may start with a simple shell script and send it to the list. Though, a real patch would be the preferred way. For this, you should have a quick look into the t/ subdirectory. Just open any of the tNUMBER*.sh files. It should be quite obvious how your sequence of shell commands could be cast into a git test script.

Steffen

Here is a start maybe, I was just testing all of the HFS variants for fun. I will write up a test case later tonight when I am out done with work.

#!/bin/sh
#
# Test git behavior on OSX's multitudes of HFS types # So far UFS, HFS, HFS+, HFSX, HFS+J, UFS is the only sane FS available...
#

#cloneurl="git://git.kernel.org/pub/scm/git/git.git" cloneurl="/Volumes/gitufs/git" ramdiskdir="/private/tmp/gitramdisk" results="/private/tmp/fsresults"

echo "Creating 100M UFS ramdisk for clone operation." rawdev=hdid -nomount ram://102400
newfs $rawdev > /dev/null 2>&1
mkdir $ramdiskdir > /dev/null 2>&1
mount -t ufs $rawdev $ramdiskdir > /dev/null 2>&1 cd $ramdiskdir && git clone $cloneurl > /dev/null 2>&1 cd $ramdiskdir/git
if [ -f $results ] ; then
echo "Removing old results."
rm $results
fi

echo "Creating HFS image"
hdiutil create -size 50m -fs HFS -attach -volname "hfs" /tmp/hfs.dmg

echo "Creating HFS+ image"
hdiutil create -size 50m -fs HFS+ -attach -volname "hfsplus" /tmp/ hfsplus.d > /dev/null 2>&1
echo "Creating HFS+J image"
hdiutil create -size 50m -fs HFS+J -attach -volname "hfsplusJ" /tmp/ hfsplusjournal.dmg > /dev/null 2>&1
echo "Creating HFSX image"
hdiutil create -size 50m -fs HFSX -attach -volname "hfsx" /tmp/ hfsx.dmg > /dev/null 2>&1
echo "Creating UFS image"
hdiutil create -size 50m -fs UFS -attach -volname "hfsu" /tmp/hfsu.dmg

for x in ls -d /Volumes/hfs*;do
echo "Testing $x clone."
echo "Results for $x:" >> $results
(echo "-- git clone results --" && cd ${x} && /usr/bin/time git
clone $ramdiskdir/git) >> $results 2>&1
(echo "-- git status results --" && cd ${x}/git && /usr/bin/time
git status) >> $results 2>&1
cd ${x} && perl -CO -e 'print pack("U",0x00E4)."\n"' | xargs touch

# umlauted a
   cd ${x} && perl -CO -e 'print pack("U",0x0061).pack("U", 
0x0308)."\n"' | xargs touch # umlauted a by combining diareses
   ls -d ${x}/* | xxd >> $results
   cd && hdiutil eject ${x} > /dev/null 2>&1
done

# cleanup
cd $HOME
umount -f $ramdiskdir > /dev/null 2>&1
hdiutil detach $rawdev > /dev/null 2>&1
rm -Rf $ramdiskdir /tmp/hfs*.dmg
more $results

My results on leopard:

$ cat /tmp/fsresults
Results for /Volumes/hfs:
-- git clone results --
Initialized empty Git repository in /Volumes/hfs/git/.git/
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-06100ef5fbd98d07358505696e2e0c5600a9b279.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-06100ef5fbd98d07358505696e2e0c5600a9b279.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-401a5ae571eb23ec896d7e441deae4e313d0de9c.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/pack- 
ab8844b63fcb4fc5896e9d75b0d10c566d5ce5bb.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/pack- 
ab8844b63fcb4fc5896e9d75b0d10c566d5ce5bb.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-9ffbf58084280a496aef6849fe3effe742a99d77.pack: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-9ffbf58084280a496aef6849fe3effe742a99d77.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/pack/ 
pack-401a5ae571eb23ec896d7e441deae4e313d0de9c.idx: Invalid argument
cpio: Unable to create /Volumes/hfs/git/.git/objects/ 
03/4ee24912da0a700ba27109825710fd84d64591: Invalid argument
         0.20 real         0.02 user         0.05 sys
-- git status results --
git_fs.sh: line 39: cd: /Volumes/hfs/git: No such file or directory
0000000: 2f56 6f6c 756d 6573 2f68 6673 2f61 cc88  /Volumes/hfs/a..
0000010: 0a                                       .
Results for /Volumes/hfsplus:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsplus/git/.git/
         2.56 real         0.47 user         0.94 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.46 real         0.27 user         0.08 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 706c 7573  /Volumes/hfsplus
0000010: 2f61 cc88 0a2f 566f 6c75 6d65 732f 6866  /a.../Volumes/hf
0000020: 7370 6c75 732f 6769 740a                 splus/git.
Results for /Volumes/hfsplusJ:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsplusJ/git/.git/
         2.29 real         0.45 user         0.91 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.57 real         0.31 user         0.10 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 706c 7573  /Volumes/hfsplus
0000010: 4a2f 61cc 880a 2f56 6f6c 756d 6573 2f68  J/a.../Volumes/h
0000020: 6673 706c 7573 4a2f 6769 740a            fsplusJ/git.
Results for /Volumes/hfsu:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsu/git/.git/
         5.08 real         0.48 user         0.94 sys
-- git status results --
# On branch master
nothing to commit (working directory clean)
         0.26 real         0.20 user         0.05 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 752f 61cc  /Volumes/hfsu/a.
0000010: 880a 2f56 6f6c 756d 6573 2f68 6673 752f  ../Volumes/hfsu/
0000020: 6769 740a 2f56 6f6c 756d 6573 2f68 6673  git./Volumes/hfs
0000030: 752f c3a4 0a                             u/...
Results for /Volumes/hfsx:
-- git clone results --
Initialized empty Git repository in /Volumes/hfsx/git/.git/
         2.49 real         0.46 user         0.88 sys
-- git status results --
# On branch master
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	gitweb/test/Märchen
nothing added to commit but untracked files present (use "git add" to  
track)
         0.25 real         0.20 user         0.04 sys
0000000: 2f56 6f6c 756d 6573 2f68 6673 782f 61cc  /Volumes/hfsx/a.
0000010: 880a 2f56 6f6c 756d 6573 2f68 6673 782f  ../Volumes/hfsx/
0000020: 6769 740a                                git.

Apologies Steffen, I grabbed your CamelCase test and did a search/ replace, wasn't sure what to call it though... But I am on lunch and wanted to be useful. Rip it apart all you want.

Fails on hfs* on OSX, works on ufs. I will bother with zfs when it can be used again.

On UFS:

$ /bin/sh ./t0060-normalization.sh
*   ok 1: setup
*   ok 2: rename (silent normalization)
*   ok 3: merge (silent normalization)
* passed all 3 test(s)

On HFS:

$ /bin/sh t0060-normalization.sh
*   ok 1: setup
* FAIL 2: rename (silent normalization)

	 git mv ä ä &&
	 git commit -m "rename"

* FAIL 3: merge (silent normalization)

	 git reset --hard initial &&
	 git merge topic

* failed 2 among 3 test(s)

The test case, it uses perl, assuming only 5.6.1+ will work with this:

new file mode 100755
index 0000000..e012c02
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Test for silent normalization issues'
+
+. ./test-lib.sh
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success setup "
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_success 'rename (silent normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_success 'merge (silent normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done

1.5.3

Well, I was planning on writing my own test case today, but you seem to have beaten me to the punch. I just tested your script and it does indeed fail as expected on HFS+. Thank you for producing this test case.

-Kevin Ballard

Kevin Ballard
http://kevin.sb.org
[email protected]
http://www.tildesoft.com

As pointed out (multiple times), this is only true if the programmer is a moron.

You do not need to - and *should* not - convert to a common normalization in order to compare to Uncode strings. You should just compare them with a Unicode-aware comparison routine. It will be faster, and it will avoid corrupting the input.

Sadly, stupid people are much too common.

Linus

One of Linus's recent patch introduces an index hashtable so that we can later hash "equivalent" names into the same bucket to allow us non-byte-by-byte comparison.

Before going further, I needed to formalize what we are trying to achieve. I learned a few things from the long flamewar thread, but it is very inefficient to go back to the thread to pick only the useful pieces. The whole flamewar simply did not fit a small Panda brain.

That was the reason for this write-up.

Design constraints. In the following, I'll use two names $A and $B as an example. They are a pair of names that are considered equivalent in some contexts, such as:

A=xt_connmark.c B=xt_CONNMARK.c

(1) Some filesystems prevent you from having these two (confusing) paths in a directory at the same time. Some do not implement this confusion prevention, and allows both names to exist at the same time.

Let's call the former "case insensitive", and the latter "case sensitive".

(2) readdir(3) on some "case insensitive" filesystems returns $A, after a successful creat(2) of $B. Others remember which one of the two "equivalent" names were used in creat(2).

Let's call the former "case folding", and the latter "case preserving".

We assume open(2) or lstat(2) of $A or $B will succeed after allowing creat(2) of $B if a case folding filesystem returns $A from readdir(3).

(3) Among the "case folding" ones, some filesystems fold the pathname to a form that is less interoperable with other systems, and/or the form that is likely to be different from what the end-user usually enters.

Such filesystems are "inconveniently case folding".

The last one is not quite apparent with the "xt_connmark.c" example, but if you replace $A and $B in the above description with:

A=Ma"rchen B=Märchen

it would hopefully become more clear.

For example, vfat is generally "case preserving". In that long flamewar thread, I think we learned that HFS+ is in general "inconveniently case folding" with respect to Unicode, by always folding to $A but the keyboard/IM input is more likely to come as $B, which happens to be the more interoperable form with other systems.

Issues with case insensitive filesystems
----------------------------------------

At the data structure level, a pathname to git is a sequence of bytes terminated with NUL. This will _not_ change.

By the way, at the data structure level, a tree entry in git can represent a blob that is a symbolic link. A tree entry in git can also represent a blob that is a regular file, and in that case, it can represent if it is executable or not. These will also not change.

Now, let's think about how we allow use of git on a filesystem that is incapable of symbolic links, and/or a filesystem that does not have trustable executable bit.

We do not say "Symlinks are evil and not supported everywhere, so let's introduce a project configuration to disallow addition of symlinks". We do not say that to the executable bit, either.

Instead, we have fallback methods to allow manipulating symlinks and executable bit on such a filesystem that is incapable of handling them natively.

We should be able to do the same for this "case sensitivity" issue. A tree that has xt_connmark.c and xt_CONNMARK.c at the same time cannot be checked out on a case insensitive filesystem.

The filesystem is simply incapable of it (please just calmly rephrase it in your head as "does not allow such confusing craziness" instead of starting another flamewar, if you feel the expression "incapable of" insults your favorite filesystem).

That may mean the project should avoid such equivalent names in its trees (and having a project wide configuration could be a technical means to help enforcing that policy), but it does not mean the core level of git should prevent them to be created on such systems. It just means that there should be a way, that could (and sometimes has to) be different from the "natural" way, to manipulate such tree entries even on a case insensitive filesystem.

For example, if I find that RelNotes symlink incorrectly points at Documentation/RelNotes-1.5.44.txt and want to fix it and push it out immediately, but if I am on the road and the only environment I can borrow is a git installation on a filesystem that is symlink-challenged, I can still do the fix. On such a filesystem, a symlink is checked out as a regular file but is still marked as a symlink in the index. The only thing I need to do is to edit the file (making sure not to add an extra LF at the end) and add it to the index. That's certainly different from the "natural" way to do that on a filesystem with symlinks, which is "ln -fs Documentation/RelNotse-1.5.4.txt RelNotes", but the point is that we make it possible.

The same thing should apply to two files that cannot be checked out at the same time on case insensitive filesystems. Perhaps we could have something like:

	$ git show :xt_CONNMARK.c >xt_connmark-1.c
        $ edit xt_connmark-1.c
	$ git add --as xt_CONNMARK.c xt_connmark-1.c

Issues with case folding filesystems
------------------------------------

In addition to the above, case folding filesystems additionally have an issue even when there is no "confusing" names in the tree. The project may want to have "Märchen" (but not "Ma"rchen"), but a checkout (which is creat(2) of "Märchen" -- because that is the byte sequence recorded in tree objects and the index) will result in "Ma"rchen" and no "Märchen" (hence readdir(3) returns "Ma"rchen").

Linus's patch to use a hashtable that links "equivalent" names together is a step in the right direction to address this. The tree (and the index) has name $B, we check out and the filesystem folds it to $A. When we get the name $A back from the filesystem (via readdir(3)), we hash the name using a hash function that would drop names $A and $B into the same bucket, and compare that name $A with each hash entry using a comparison that considers $A and $B are equivalent. If we find one, then we keep the name $B we have already.

If it is a new file, we won't find any name that is equivalent to $A in the index, and we use the name $A obtained from readdir(3).

BUT with a twist.

If the filesystem is known to be inconveniently case folding, we are better off registering $B instead of $A (assuming we can convert from $A to $B).

One bad issue during development is that we cannot sanely emulate case folding behaviour on non case-folding filesystems without wrapping open(2), lstat(2), and friends, because of the assumption we made above in (2) where we defined the term "case folding". This means that the codepath to deal with case folding filesystems inevitably are harder to debug.

Tasks
-----

- Identify which case folding filesystems need to be supported, and make sure somebody understands its folding logic;

- For each supported case folding logic, these are needed:

- a hash function that throws "equivalent" names in the same

     bucket, to be used in Linus's patch;

- a compare function to determine equivalent names;

- a convert function that takes a possibly inconvenient form of equivalent name (i.e. $A above) as input and returns more convenient form (i.e. $B above)

- Identify places that we use the names obtained from places other than the index and tree. From these places, we would need to call the convert function to (de)mangle the name before they hit the index.

Because we may be getting driven by something like:

	$ find | xargs git-foo

handling readdir(3) we do ourselves any specially does not make much sense. Any path from the user is suspect.

- Identify places that we look for a name in the index, and perform equivalent comparison instead of memcmp(3) we traditionally did. Linus's patch gives scaffolding for this.

All true. Though Linux support for creating and using HFS+ volumes seems like it may be helpful. Trying the test case patch[*] posted by Mitch Tishmack showed the problem here. The only slightly strange thing was that there didn't seem to be an issue with the gitweb/test/Märchen file after cloning to the HFS volume.

Sean.

[*]

$ dd bs=1M count=250 < /dev/zero > hfs_vol
  262144000 bytes (262 MB) copied, 6.12703 s, 42.8 MB/s

$ /sbin/mkfs.hfsplus -v Test -n c=4096,e=1024 hfs_vol
  Initialized hfs_vol as a 250 MB HFS Plus volume

$ mkdir hfs
$ sudo mount -t hfsplus -o loop hfs_vol hfs
$ sudo chmod a+rwx hfs
$ cd hfs
$ git clone ~/local/sources/git
  Initialized empty Git repository in ~/hfs/git/.git/
  49486 blocks

$ cd git
$ make
   ...

$ cd t
$ git apply ~/Mitch_Tishmack.patch
$ ./t0060-normalization.sh
  * FAIL 1: setup

	  touch ä &&
	  git add ä &&
	  git commit -m "initial"
	  git tag initial &&
	  git checkout -b topic &&
	  git mv ä tmp &&
	  git mv tmp ä &&
	  git commit -m "rename" &&
	  git checkout -f master

* FAIL 2: rename (silent normalization)

	 git mv ä ä &&
	 git commit -m "rename"

* FAIL 3: merge (silent normalization)

	 git reset --hard initial &&
	 git merge topic

* failed 3 among 3 test(s)

Sean.

Why?

If you have no other representation for the file name than $A already, then I don't see why Git would have to play similar evil games and corru^H^H^Hnvert $A into $B. Just store $A in the index and tree objects and be done with it.

Nicolas

Because if you happen to be on a case-challenged filesystem, and you need to add a new file - say xt_CONNBARK.c, you can, even if your FS only has xt_connbark.c to offer to git. Granted, it's harder, but it can be done, and it means that powerusers and wrappers around git have a hope of dealing with it.

I think this is an excellent plan.

There is one thing that I don't see in Junio's plan -- which is excellent -- and is

- a warning during checkout if the index contains "equivalent" paths
that will clobber eachother during checkout.
- an optional warning/error during add, to be raised if I am adding a
path that is equivalent to an already-existing path in the index

The second one is to support project that are known to be developed on these non-filename-preserving platforms. So if I am on a linux host, and I add Readme where README already exists, a warning can save me and the project a bit of grief - and possibly catch an unintended mistake! Because while I agree that users should be able to store any file in git, in practice most instances of Ma"rchen/Märchen case will be due to user error (or editor/gui error).

cheers,

m

Thanks. I think these and many other issues need to be worked out.

In my message, I did not even try to be exhaustive. I outlined the parts that would most deeply affect the parts I care more deeply about, which is the plumbing. As I said many times, I do not do Porcelains ;-).

Because this "conversion" is limited to the case where the filesystem is known to be inconveniently case folding, I personally do not care about this part of the outline that deeply. It would not bite _me_ or my friends either way.

But I would imagine that a person who has to work on HFS+ would appreciate it if these two sequences behaved the same way:

    $ edit Märchen ;# assume this is a new file
    $ git add Märchen ;# we were told that IM gives $B (aka NFC)

vs

    $ edit Märchen ;# assume this is a new file
    $ git add M*en ;# now readdir(3) gives $A (aka NFD)

If we always convert $A (less interoperable form) to $B (more interoperable form) on inconveniently case folding filesystems, the new index entry will always be in form $B. Without the conversion, the former will give form $B while the latter will give form $A. It is, as you said, "similar evil game to corrupt", but it is not even a corruption at that point, because the inconveniently case folding filesystem already corrupted the pathname before we get our hands on it, and it won't make a difference for HFS+ only people anyway.

However, if the resulting tree that adds a new file is prepared on an inconveniently case folding filesystem, the conversion process, by definition, would make the resulting tree more interoperable with other systems than without.

So I do not see any downside of doing the conversion on such a filesystem but there is this "interoperability" upside.