Skip to contentSkip to conversation

mingw, windows, crlf/lf, and git

DISCUSSION Mark Levedahl 15 participants 83 messages Jun 15, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

I am NOT intending to start a flamewar O:-) , so please don't turn this into one.

The recent threads on a mingw git port are explicit in the intent to provide a Windows native git. I believe there is a fundamental conflict here with the position, clearly stated by Linus, that git does not alter content in any way. Windows suffers the curse of DOS line endings (\r\n vs \n), and a true port to Windows *must* allow for \r\n and \n to be semantically the same thing as most large projects end up with a mixture of such files and/or are targeting cross-platform capabilities. The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations. To be relevant on native Windows, git must do the same. Otherwise, git will be deemed "too wierd" and dismissed in favor of a tool "that works."

There is no use to debating the technical merits of \r\n vs \n vs \r vs whatever, nor of not converting. Really. Just accept that there is a fundamental requirement that any version control tool on Windows be able to silently convert between \r\n and \n. To believe otherwise is to expect that the conversion be pushed elsewhere into the tool chain in use, and that won't happen as the competition already provide this conversion capability.

So, I think the git project needs to come to an explicit position on this, basically being:

1) git is a POSIX only tool (i.e., there will be no \r\n munging), or
2) a Windows port of git will handle and mung \r\n and \n line endings.

If the answer is 1, the mingw port is a waste of time as it simply won't be usable by its target audience. If the answer is 2, then I think a very careful design of this capability is in order.

Comments?

BTW, I have addressed this in my own world using a pre-commit script that converts textfile line endings into \n, recognizing that our Windows tool chain handles such files perfectly well, while our Linux toolchain requires it.

Mark Levedahl

#1 lore raw

måndag 12 februari 2007 00:13 skrev Mark Levedahl:

Quoted 5 lines

The recent threads on a mingw git port are explicit in the intent to provide a Windows native git. I believe there is a fundamental conflict here with the position, clearly stated by Linus, that git does not alter content in any way. Windows suffers the curse of DOS line endings (\r\n vs \n), and a true port to Windows *must* allow for \r\n and \n to be

Quoted 9 lines

semantically the same thing as most large projects end up with a mixture of such files and/or are targeting cross-platform capabilities. The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations. To be relevant on native Windows, git must do the same. Otherwise, git will be deemed "too wierd" and dismissed in favor of a tool "that works."

As of today git is a posix tool simply because it's not fully ported to other enviromnents. I brought this up quite a time ago, and didn't face heavy artillery then, and wouldn't today either. The code is still missing though. I didn't write it then, because it's my #1 priority and nobody else did. Linus even did a rough scetch, but that's it.

I guess git will get this feature when someone does the code for it.

-- robin

#2 lore raw
Quoted 3 lines · Robin Rosenberg

As of today git is a posix tool simply because it's not fully ported to
other enviromnents. I brought this up quite a time ago, and didn't face heavy artillery
then, and wouldn't today either. The code is still missing though. I didn't write it then, because it's my #1 priority and nobody else did. Linus even did a rough scetch, but that's it.

So, the basic design for this feature exists where? I would assume this would include a file mode indicator set in the blob or tree designating the blob is "text", along with mechanism to specify for a project what files are "text", along with some safety valve to check and not do transformation when the file does not look text-ish.

Mark

#3 lore raw
In reply to Mark Levedahl
Quoted 16 lines · Mark Levedahl

I am NOT intending to start a flamewar O:-) , so please don't turn this into one.

The recent threads on a mingw git port are explicit in the intent to provide a Windows native git. I believe there is a fundamental conflict here with the position, clearly stated by Linus, that git does not alter content in any way. Windows suffers the curse of DOS line endings (\r\n vs \n), and a true port to Windows *must* allow for \r\n and \n to be semantically the same thing as most large projects end up with a mixture of such files and/or are targeting cross-platform capabilities. The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations. To be relevant on native Windows, git must do the same. Otherwise, git will be deemed "too wierd" and dismissed in favor of a tool "that works."

So this is something that I've tried proposing to the Mercurial developers, but it's never been implemented in hg. It'll be interesting to see what the git community thinks. :-)

My proposal does require adding a file type to each file, as tracked metadata, which may doom it from the start. If you add a file type, then you have to support mutating the file type, and some way of handling merge conflicts (generally, picking one type or another).

Then for each file type, we implement a set of interfaces (perhaps as simple as a series of executables named git-<type>-<operation>) which if present, transforms the file from its live format to the canonical format which is actually checked in and back again. Besides using this for the DOS CR/LF problem, it also allows for an efficient storage of things like OpenOffice files which are a zipped set of .xml files. By decompressing them before pushing them into the SCM, it means that if the user makes a tiny spelling correction in their OpenOffice file, the delta stored in the git repository can be much more efficiently stored (since the diff of the .xml tree will be small, where as the diff of the entire compressed file is likely going to be close to the entire size of the .odt file).

Another nice thing to provide for each file type would be a pretty-printer for the diffs, so it becomes easier to see the delta between two versions of an OpenOffice file in a textual window.

So, is this idea sane or completely insane? Hopefully it passes Linus's it-solves-multiple-problems-at-once test, at least. :-)

- Ted

#4 lore raw
Quoted 13 lines · Theodore Tso

Then for each file type, we implement a set of interfaces (perhaps as simple as a series of executables named git-<type>-<operation>) which if present, transforms the file from its live format to the canonical format which is actually checked in and back again. Besides using this for the DOS CR/LF problem, it also allows for an efficient storage of things like OpenOffice files which are a zipped set of .xml files. By decompressing them before pushing them into the SCM, it means that if the user makes a tiny spelling correction in their OpenOffice file, the delta stored in the git repository can be much more efficiently stored (since the diff of the .xml tree will be small, where as the diff of the entire compressed file is likely going to be close to the entire size of the .odt file).

Quoted 6 lines

Another nice thing to provide for each file type would be a pretty-printer for the diffs, so it becomes easier to see the delta between two versions of an OpenOffice file in a textual window.

So, is this idea sane or completely insane? Hopefully it passes Linus's it-solves-multiple-problems-at-once test, at least. :-)

there have been other things discussed that could use the 'do this on checkout' hooks, specificly on the issue of useing git to manage /etc the need to save/restore permissions requires a hook on checkout that doesn't exist yet. this sounds like it would solve that problem as well.

David Lang

#5 lore raw
In reply to Theodore Tso

Hi,

Quoted 2 lines · Theodore Tso

My proposal does require adding a file type to each file, as tracked metadata, which may doom it from the start.

I'd rather do that a la .gitignore, i.e. make this handling dependent on file name patterns. It is not only backwards compatible (from the viewpoint of the repository format), it also avoids having to specify over and over again that yes, this new .odt file _is_ an OpenOffice document.

Quoted 4 lines

Then for each file type, we implement a set of interfaces (perhaps as simple as a series of executables named git-<type>-<operation>) which if present, transforms the file from its live format to the canonical format which is actually checked in and back again.

Again, I propose a slight change: Let's add a transformation driver like the merge driver: this allows inlining common operations like unzipping, CRLF->LF conversion, etc.

Ciao,
Dscho

#6 lore raw
In reply to Theodore Tso
Quoted 9 lines · Theodore Tso

So this is something that I've tried proposing to the Mercurial developers, but it's never been implemented in hg. It'll be interesting to see what the git community thinks. :-)

My proposal does require adding a file type to each file, as tracked metadata, which may doom it from the start. If you add a file type, then you have to support mutating the file type, and some way of handling merge conflicts (generally, picking one type or another).

I agree that a file-type approch would work, but I personally think it's too inflexible (just cr/lf vs lf? There are tons of other interesting issues that are valid). I also think it falls down on another (and in some ways much more fundamental problem): these things exist EVEN WHEN THE FILE ITSELF DOES NOT EXIST!

In other words, a policy about cr/lf is *not* a policy about actual content. It's something much more: it's a policy about representation in general, which includes *potential* content. It should obviously take effect on "git add" even with content that didn't exist before, and to work well, it should do so without the user having to think about it.

Equally importantly, this happens with content that was added by people who simply DO NOT CARE. In other words, I think a "file type" thing fundamentally cannot work, because under UNIX, it would be stupid and pointless, so any project that is maintained under UNIX might _add_ the file types, but since they won't matter, they'll inevitably be wrong (ie people forgot to mark a binary thing binary, or a text thing as text).

So: file types or attributes are broken. They cannot work well.

But enough on the negative rambling, I do have a positive and constructive suggestion, because I actually think I have a great model for it. But I've never cared enough (and since the main target would be some windows issue, I suspect I never really _will_ care enough) to really worry about it.

Anyway, if somebody really wants to look at this, and wants to create something that is actually _usable_, my suggestion is to simply extend on the ".gitignore" file approach. The great thing about .gitignore is that

(a) you can track it like you track any other file

This makes merges a *lot* easier. You see it as conflicts, you can fix it up, and in general, you can use all the same tools with it as you use with anything else. In contrast, explicit per-file filetypes are _horrible_ for maintenance.

(b) you can add to it with *patterns*, which is exactly what you want for file types.

You can do things like

*.bin: binary
*: text

to say "everythgn that matches *.bin is binary, the rest is text", and solves the maintenance issue trivially. Everybody will like it. For the kernel, for example, we'd have a really easy

Documentation/logo.gif: binary
*: text

and that would probably take care of it.

You can also have a few default file patterns built in, which would take care of it for 99% of all projects without anybody ever having to even think about it - even under DOS.

(c) it doesn't actually affect database representation, it only changes behaviour for programs, which is also exactly what you want (if you have per-file "file types", you end up having serious problems at merge time: when I say "affect database representation", I don't mean that I think git cannot change its database, I literally mean at a "higher" level: represening per-file attributes is a DISASTER from a merge situation)

So not only is it backwards-compatible with traditional git usage, it's much more fundamentally simple: it doesn't add any new core data structures or rules. All the core stays exactly as it is, and it just affects higher-level behaviour. And that's important: one reason git has been so stable is that the really core data structures are really really stable and simple.

Even when we did *really* core changes like the whole packfile thing, the fundamental data structures didn't change at all *conceptually*.

(d) it's actually a lot more flexible than file types.

Merge stategies, anybody? We can easily have the default merge strategy be the normal three-way merge (which is obviously the right thing for almost anything), but how about something like

*.doc: binary,merge=doc-merge

which tells git that it should use a separate "doc-merge" program to merge those kinds of files when it needs to do a nontrivial merge..

(e) exactly like ".gitignore", you should also be able to have a ".git/info/exclude" file that is your _private_ rules, and per-directory ".gitignore" files that are the _hierarchical_ rules.

This just makes maintenance much simpler. Not one big file that has everything, and that clashes. Make the top-level one contain all the generic default rules, and then lower down we can have more specific rules for very specific things, exactly like the kernel .gitignore files do. The top-level file should *not* have to know all the details of some architecture- or sub-project specific file behaviour.

Similarly, having an untracked file (.git/info/exclude) allows people to have rules that make sense for *them*, but that might not make sense for the upstream developers (say, somebody crazy enough to develop Linux under Windows). So people can have their purely local rules without forcing them on others.

Anyway, that would be my suggestion. Call it ".gitattributes" or something. Make it a nice ASCII format, exactly like .gitignore, and make all the rules exactly the same, except it has a ": <attributelist>" at the end for each line.

Start off supporting just "binary" and "text", but keep in mind that people may want other things. Individualized merge strategies etc.

Also, keep in mind that a *lot* of git operations will work purely on a SHA1 level, and those operations fundamentally *will*not*care* about file types. So when you merge a file, for example, the initial merge will be done purely on SHA1's, and git would do all the normal "if it didn't change in branch 1, take the branch 2 version directly" without ever even *looking* at any file rules.

This is important, because this is what makes git efficient for large projects, and which would allow git to _remain_ efficient even in the face of having to read all those comples .gitattributes files. When we merge two repositories with 20,000+ files, we usually really only "merge" a couple of the files.

Same goes for "text" mode. The "text" thing would only affect things like "git add" etc that use "git-update-index" to calculate the new SHA1. We'd never use it "normally". "git diff" would still be instantaneous, because the git index shows the file still matches, and that is all done on a SHA1 only level. So only when you do a "git add" or when it needs to refresh the index because the file changed, and it reads in the file, will it actually care about whether it's a text or a binary file.

This is actually *exactly* what you want. Not just for performance, but simply because this is also how you can take something like the Linux archive, and "just use it" under Windows, even if your editor adds (or wants) CR/LF.

Btw, how would I implement this? If I really were energetic enough to implement it, I would do:

(a) Add a flag to "git-ls-files" logic to add "type information" in front.

Not only do you want this *anyway* for other reasons, but for binary/text, the thing you actually care most about is "git add", and it already basically just does "take this file pattern, feed it through git-ls-files, and add those files". So you'd get it basically for free.

It is also fairly easy to add at this stage, because you can simply look for all the places that work with "info/exclude" and ".gitignore", and you know that "Ahh, I need to teach these exact places to understand about attributes". So you'd add an "add_attributes_from_file()" function etc etc.

Quite straightforward. In fact, you might be able to use the gitignore parsing *as*is*, and just teach it about more flags that just "ignore": both in "struct dir_entry" and in "struct exclude".

(b) Teach the git-update-index logic about hashing text blobs.

(c) Profit!

It really should be fairly straightforward. I'm sure it wouldn't be *entirely* trivial, but I'm also fairly sure that somebody reasonably competent could do it in a couple of days (with testing) if they were just sufficiently motivated to get started.

Anybody?

Linus

#7 lore raw
Quoted 18 lines · Linus Torvalds

Btw, how would I implement this? If I really were energetic enough to implement it, I would do:

(a) Add a flag to "git-ls-files" logic to add "type information" in front.

Not only do you want this *anyway* for other reasons, but for binary/text, the thing you actually care most about is "git add", and it already basically just does "take this file pattern, feed it through git-ls-files, and add those files". So you'd get it basically for free.

It is also fairly easy to add at this stage, because you can simply look for all the places that work with "info/exclude" and ".gitignore", and you know that "Ahh, I need to teach these exact places to understand about attributes". So you'd add an "add_attributes_from_file()" function etc etc.

Quoted 5 lines

Quite straightforward. In fact, you might be able to use the gitignore parsing *as*is*, and just teach it about more flags that just "ignore": both in "struct dir_entry" and in "struct exclude".

(b) Teach the git-update-index logic about hashing text blobs.

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

Textual diff generation needs to learn the CRLF-to-LF conversion in diff_populate_filespec(); this needs to be done even when the caller wants size_only.

Oops.

Not so fast. What's your plan for st_size?

Quoted 8 lines

(c) Profit!

It really should be fairly straightforward. I'm sure it wouldn't be *entirely* trivial, but I'm also fairly sure that somebody reasonably competent could do it in a couple of days (with testing) if they were just sufficiently motivated to get started.

Anybody?

Not me.

#8 lore raw
Quoted 11 lines · Junio C Hamano
Quoted 4 lines · Linus Torvalds

Btw, how would I implement this? If I really were energetic enough to implement it, I would do:
...
(b) Teach the git-update-index logic about hashing text blobs.

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

Quoted 7 lines

Textual diff generation needs to learn the CRLF-to-LF conversion in diff_populate_filespec(); this needs to be done even when the caller wants size_only.

Oops.

Not so fast. What's your plan for st_size?

If I were to do this, I would say the cache should store the size on the filesystem in stat fields. Which means that the object name recorded is text blob _after_ line endings are normalized to LF, and its exploded size does not necessarily match the cached size.

So this means that whoever does the diff_populate_filespec() change needs to be careful, but it is not such a big deal.

#9 lore raw
In reply to Junio C Hamano

Hi,

Quoted 3 lines · Junio C Hamano

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely on Windows' editors not to fsck up with line endings. The worst case is if the file contains _some_ CRLF and _some _LF_. Almost always I had the problem that it now converted _all_ LFs to CRLFs. Even those which already were converted.

So, if we are to support text mode, it is not one-way. If we do one-way, we really do _not_ support text mode, but pre-commit conversion to LF style text. And in this case, core git does not need _any_ change.

Ciao,
Dscho

#10 lore raw
Quoted 8 lines · Johannes Schindelin

Hi,

Quoted 2 lines · Junio C Hamano

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

Quoted 8 lines

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely on Windows' editors not to fsck up with line endings. The worst case is if the file contains _some_ CRLF and _some _LF_. Almost always I had the problem that it now converted _all_ LFs to CRLFs. Even those which already were converted.

So, if we are to support text mode, it is not one-way. If we do one-way, we really do _not_ support text mode, but pre-commit conversion to LF style text. And in this case, core git does not need _any_ change.

Ciao,
Dscho

In my work flow, I am using a pre-commit script that (among other things) rewrites all text files to have \n endings. This is a one-way conversion, and does work well for the set of tools I am using. The converters I use I wrote years ago, and are smart enough to deal with mixtures of \n, \r\n, and \r line endings in one file, transforming all into one unified form. d2u / u2d were not that robust when I last tried them (years ago), but this is an absolute necessity.

However, I don't think the one-way conversion is acceptable across the board. While the only Windows editor I am aware of that doesn't grok \n is Notepad (the moral equivalent of edlin), I suspect that undo reliance upon this will still lead to grief. If nothing else, someone, somewhere will find that their beloved crlf's are missing and will complain. Loudly. And in the lore, git will become known for being "wierd." So, I suspect a checkout script is necessary.

Mark

#11 lore raw
In reply to Johannes Schindelin
Quoted 16 lines · Johannes Schindelin

Hi,

Quoted 3 lines · Junio C Hamano

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely on Windows' editors not to fsck up with line endings. The worst case is if the file contains _some_ CRLF and _some _LF_. Almost always I had the problem that it now converted _all_ LFs to CRLFs. Even those which already were converted.

Quoted 3 lines

So, if we are to support text mode, it is not one-way. If we do one-way, we really do _not_ support text mode, but pre-commit conversion to LF style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

- I do not see how you propose to solve some CRLF and some LF case with both-ways conversion.

- Pre-commit hook would not be sufficient. In a edit, diff, test and then commit cycle, diff and test step needs to look at whatever the editor left on the filesystem, so the changes to populate-filespec is needed to make diff part work.

#12 lore raw
Quoted 16 lines · Junio C Hamano
Quoted 16 lines

Hi,

Quoted 3 lines · Junio C Hamano

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely on Windows' editors not to fsck up with line endings. The worst case is if the file contains _some_ CRLF and _some _LF_. Almost always I had the problem that it now converted _all_ LFs to CRLFs. Even those which already were converted.

Quoted 8 lines
Quoted 3 lines

So, if we are to support text mode, it is not one-way. If we do one-way, we really do _not_ support text mode, but pre-commit conversion to LF style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

- I do not see how you propose to solve some CRLF and some LF case with both-ways conversion.

the expectation is that the some-of-each situation is unlikly to happen if you convert all the time.

and if you do end up with a mixed ending file, the next time you check it in from a windows box it should clean it up.

David Lang

#13 lore raw
In reply to Junio C Hamano

Hi,

Quoted 18 lines · Junio C Hamano
Quoted 16 lines · Johannes Schindelin

Hi,

Quoted 3 lines · Junio C Hamano

I agree that we can assume editors can grok files with LF end-of-line just fine and we would not need to do the reverse conversion on checkout paths (e.g. "read-tree -u", "checkout-index").

In that case, a simple pre-commit hook would suffice.

No, the problem mentioned by Mark was a very real one: you _cannot_ rely on Windows' editors not to fsck up with line endings. The worst case is if the file contains _some_ CRLF and _some _LF_. Almost always I had the problem that it now converted _all_ LFs to CRLFs. Even those which already were converted.

Quoted 8 lines
Quoted 3 lines

So, if we are to support text mode, it is not one-way. If we do one-way, we really do _not_ support text mode, but pre-commit conversion to LF style text. And in this case, core git does not need _any_ change.

Well I disagree in two counts.

- I do not see how you propose to solve some CRLF and some LF case with both-ways conversion.

Very easy. Forward: s/\r\n/\n/. Backward: s/\(^\|[^\r]\)\n/\r\n/.

Quoted 4 lines

- Pre-commit hook would not be sufficient. In a edit, diff, test and then commit cycle, diff and test step needs to look at whatever the editor left on the filesystem, so the changes to populate-filespec is needed to make diff part work.

Yes, you are right.

However, since this is all post-1.5.0 (right? Right?) why not go with more of Ted's proposal, and make this whole mess also usable for other things than just crlf issues?

And I _really_ think that you do not help Windows people by doing this one-way thing.

Ciao,
Dscho

#14 lore raw
Quoted 1 line · Johannes Schindelin

However, since this is all post-1.5.0 (right? Right?) why not go with more of Ted's proposal, and make this whole mess also usable for other things than just crlf issues

Whatever is done, it needs to be robust to the notion that people will fail to set the correct file type somewhere. Current cvsnt is fairly good at autodetecting and setting text vs binary file type, and enforces this across all platforms, so things don't go awry too often. It is in my experience more reliable than subversion, which basically relies upon file extensions mapping to mime types to identify content. All of which is a very much too low standard of accuracy for a version control system: I lost many files per year due to the above nonsense, so I worry about trying to create a very general transform solution and not making it really, really failsafe. Having projects define individual globbing patterns is good, double checking the content for sanity is an absolute must, but I don't think that is enough. I suspect the solution should include round-trip conversion when creating blobs to assure that the input can be exactly reconstructed by the inverse transformation (and therefore possibly rejecting input with mixed line endings). A similar check could be applied on checkout.

Perhaps I'm too paranoid, but I've been burnt way too many times by text/binary mode stuff to let this part be trivialized. Maybe it only gets enabled by core.ImReallyParanoid, but I want that option.

Mark

#15 lore raw
Quoted 3 lines · Mark Levedahl

fail to set the correct file type somewhere. Current cvsnt is fairly good at autodetecting and setting text vs binary file type, and enforces this across all platforms, so things don't go awry too often. It is in

There is obviously much sentiment that this should _not_ be the default (and I agree). But if arbitrary filters are possible, then you can theoretically write an 'autocrlf' filter which will try to do the right thing, and you could set it for some or all files:

echo '*: autocrlf' >.gitattributes

but it would be off by default. If we implement this, everyone has to "pay" for .gitattributes (even if you don't use it, we have to look it up to make sure you're not using it!), but nobody has to pay for any filters they don't use.

-Peff

#16 lore raw
In reply to Mark Levedahl

Hi,

Quoted 3 lines · Mark Levedahl

Perhaps I'm too paranoid, but I've been burnt way too many times by text/binary mode stuff to let this part be trivialized. Maybe it only gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally opposed to enabling that option by default on all platforms. I am okay with .gitattributes (but I would call it .gitfiletypes), but I am _not_ okay with git being _too much_ fscked up by Windows. Microsoft has done enough harm already.

Ciao,
Dscho

#17 lore raw
Quoted 11 lines · Johannes Schindelin
Quoted 3 lines · Mark Levedahl

Perhaps I'm too paranoid, but I've been burnt way too many times by text/binary mode stuff to let this part be trivialized. Maybe it only gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally opposed to enabling that option by default on all platforms. I am okay with .gitattributes (but I would call it .gitfiletypes), but I am _not_ okay with git being _too much_ fscked up by Windows. Microsoft has done enough harm already.

Indeed; this type of checking should only occur if there is a filter applied to a file. Most files in most projects would hopefully just be considered to be byte streams to Git, like they are today, and thus not incur any additional overhead, beyond matching their type to determine they are in fact just a byte stream.

The type could be cached in the index; or at least a single bit which says "I'm just a byte stream, thanks" so that the matching only needs to occur during an initial read-tree.

Signature · 1 line
Shawn.
#18 lore raw
Quoted 19 lines · Shawn O. Pearce
Quoted 11 lines · Johannes Schindelin
Quoted 3 lines · Mark Levedahl

Perhaps I'm too paranoid, but I've been burnt way too many times by text/binary mode stuff to let this part be trivialized. Maybe it only gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally opposed to enabling that option by default on all platforms. I am okay with .gitattributes (but I would call it .gitfiletypes), but I am _not_ okay with git being _too much_ fscked up by Windows. Microsoft has done enough harm already.

Indeed; this type of checking should only occur if there is a filter applied to a file. Most files in most projects would hopefully just be considered to be byte streams to Git, like they are today, and thus not incur any additional overhead, beyond matching their type to determine they are in fact just a byte stream.

Quoted 3 lines

The type could be cached in the index; or at least a single bit which says "I'm just a byte stream, thanks" so that the matching only needs to occur during an initial read-tree.

for the limited case of line endings it may be reasonable to define the internal git format to be lf, and if you are running on a platform that uses this nativly no transition is needed

one possible way to make this be a general feture is to have the helper script have a --needed flag that tells git if it would do anything on the current platform or not. this way you don't need to run it (and sanity check it) if it's not needed.

David Lang

#19 lore raw
In reply to Johannes Schindelin
Quoted 7 lines · Johannes Schindelin

Hi,

Quoted 1 line · Mark Levedahl

Perhaps I'm too paranoid, but I've been burnt way too many times by text/binary mode stuff to let this part be trivialized. Maybe it only gets enabled by core.ImReallyParanoid, but I want that option.

Be aware that what you proposed costs many CPU cycles. I am totally opposed to enabling that option by default on all platforms. I am okay with .gitattributes (but I would call it .gitfiletypes), but I am _not_ okay with git being _too much_ fscked up by Windows. Microsoft has done enough harm already.

I would assume that none of this crlf stuff exists at all on Linux / Unix / Posix, so if done right has zero impact outside of the Windows nuthouse. Inside that, folks are already so used to incredible slowness in file I/O that I'm not sure the round tripping I suggest as a check would be very noticeable, but in any case I fully agree it should be optional even there. However, if git could support something that never screws up, absolutely guaranteeing data integrity in the presence of these transforms, that would be a first in this arena and I believe a significant selling point.

Mark

#20 lore raw
In reply to Junio C Hamano
Quoted 2 lines · Junio C Hamano

Not so fast. What's your plan for st_size?

Umm. There's two (very distinct) uses for st_size.

The one that we actually use to validate the current index obviously must match the "OS returned value". It contains all the CR/LF stuff.

The one where we actually read the file and run SHA1 on the result must obviously be the post-conversion one.

But it shouldn't be a problem. We'll always know which one matters: the index case is always about pure stat information (and has no meaning outside of that, really - after all, it's no different from st_mode etc, and we actually keep it in a special binary format that is endian-safe!) and the "real object" case is always about the *data* we use to compare with.

I don't think we ever mix the two anyway.

Linus

#21 lore raw
Quoted 7 lines · Linus Torvalds

But it shouldn't be a problem. We'll always know which one matters: the index case is always about pure stat information (and has no meaning outside of that, really - after all, it's no different from st_mode etc, and we actually keep it in a special binary format that is endian-safe!) and the "real object" case is always about the *data* we use to compare with.

In fact, for git-update-index, I think it's *literally* as easy as just changing "index_fd()" to convert the buffer on-the-fly as needed, before we actually call "write_sha1_file()" or "hash_sha1_file()".

So we'd just need to pass in the information about whether it's binary or not, and then do something like

	@@ -2091,6 +2091,10 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, con
	 
	 	if (!type)
	 		type = blob_type;
	+#ifndef __UNIX__
	+	if (text && !strcmp(type, blob_type))
	+		convert_crlf_to_lf(&buf, &size);
	+#endif
	 	if (write_object)
	 		ret = write_sha1_file(buf, size, type, sha1);
	 	else

and that would take care of a lot of things (yeah, I'd not do it that way in practice, but really doesn't look that nasty - it's actually much nastier to have to look up the text/binary type in the first place).

Something similar looks to be true in diff generation. The core "compare two SHA1's at a time" doesn't need any changes, but the code that actually reads in the temporary file from disk obviously does. But even that is just _one_ point, afaik - diff_populate_filespec()":

	@@ -1362,6 +1362,10 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
	 		if (fd < 0)
	 			goto err_empty;
	 		s->data = xmmap(NULL, s->size, PROT_READ, MAP_PRIVATE, fd, 0);
	+#ifndef __UNIX__
	+		if (text)
	+			convert_crlf_to_lf(&s->data, &s->size);
	+#endif
	 		close(fd);
	 		s->should_munmap = 1;
	 	}

(and again, that's not real code, it would also need to change the "should_munmap" flag to indicate the state of the _new_ "data" thing.

Linus

#22 lore raw
Quoted 15 lines · Linus Torvalds

So we'd just need to pass in the information about whether it's binary or not, and then do something like

	@@ -2091,6 +2091,10 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, con

	 	if (!type)
	 		type = blob_type;
	+#ifndef __UNIX__
	+	if (text && !strcmp(type, blob_type))
	+		convert_crlf_to_lf(&buf, &size);
	+#endif
	 	if (write_object)
	 		ret = write_sha1_file(buf, size, type, sha1);
	 	else
Quoted 3 lines

and that would take care of a lot of things (yeah, I'd not do it that way in practice, but really doesn't look that nasty - it's actually much nastier to have to look up the text/binary type in the first place).

you could do something like this and it would deal with the srlf/lf problem, but if you instead put in the conversion hooks like Ted suggested then you can actually gain a LOT more.

his example of openoffice documents that are gziped xml files is a very good one. if the 'conversion' is to gunzip on checkin and gzip on checkout then the core git logic will work on the nice diffable xml instead of the compressed binary blob.

if this is extensable to arbatrary helper functions to do the conversions I'll bet that there are many other cases that can use this.

I think the big questions needs to be, is this helper app a filter, or can it be passed a filename as the destination (which would let it do things like set permissions on the files it creates), or should it be both?

David Lang

#23 lore raw
In reply to Linus Torvalds

Hi,

[I agree on the .gitignore approach; see my other mail in this thread]

Quoted 18 lines · Linus Torvalds

Btw, how would I implement this? If I really were energetic enough to implement it, I would do:

(a) Add a flag to "git-ls-files" logic to add "type information" in front.

Not only do you want this *anyway* for other reasons, but for binary/text, the thing you actually care most about is "git add", and it already basically just does "take this file pattern, feed it through git-ls-files, and add those files". So you'd get it basically for free.

It is also fairly easy to add at this stage, because you can simply look for all the places that work with "info/exclude" and ".gitignore", and you know that "Ahh, I need to teach these exact places to understand about attributes". So you'd add an "add_attributes_from_file()" function etc etc.

Quoted 7 lines

Quite straightforward. In fact, you might be able to use the gitignore parsing *as*is*, and just teach it about more flags that just "ignore": both in "struct dir_entry" and in "struct exclude".

(b) Teach the git-update-index logic about hashing text blobs.

(c) Profit!

Not so fast.

In order for this to be _useful_, you also have to have a way to _extract_ the text blobs. Not only for read-tree, but _also_ for diff. It makes no sense at all to have this transformation one-way. For diff, you _might_ want to have a diff beautifier (for example the .odt thing), but read-tree is _really_ important.

Ciao,
Dscho

#24 lore raw
Quoted 4 lines · Johannes Schindelin
Quoted 2 lines

(c) Profit!

Not so fast.

Aww! And just when I _finally_ had a "step 2".

Quoted 2 lines

In order for this to be _useful_, you also have to have a way to _extract_ the text blobs. Not only for read-tree, but _also_ for diff.

Actually, my argument is that we don't need it all that much.

For example, your "read-tree" argument is actually wrong. Anything that is in a tree is _already_ fixed to be '\n'. So as long as we keep to things like

git diff version1..version2

we'll actually always get the right version.

Also, the index will make sure that we don't even *try* to diff normal checked out files.

So the only time you actually really need to test the .gitattributes file is when you do an "open blob in working tree". And once you do that function right, and just make sure both git-update-index and yes, the "diff against working tree" cases use it, you really should be mostly done.

Both git-update-index and git-diff-files want basically the same interface:

	struct file_buf {
		const char *buf;
		unsigned long size;
		int flags;
	}

	int read_file(const char *path, struct file_buf *);
	close_file(struct file_buf *);

and we should use that instead of the current "open + stat + mmap/read + close" sequences.

It really shouldn't be too nasty.

Linus

#25 lore raw
In reply to Mark Levedahl

Hi,

Quoted 4 lines · Mark Levedahl

The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations.

Agree with transformations on input and output; disagree on diff.

The problem is that it really is a transformtion. Since most Windows tools (at least those used in portable software) handle \n without \r quite well, thank you, I'd tend towards the view point: do not mess with line endings pre-commit/post-checkout.

Even MacOSX uses \n now, instead of \r.

Of course, for those projects which _use_ CRLF: they can continue with it. Git has no problem with those line endings.

The only problem CVS tried to solve (badly) was to be able to checkout text files on DOS, Unix _and_ MacOS. In practice, though, this use case does not matter anymore IMHO.

Ciao,
Dscho

#26 lore raw
Quoted 8 lines · Johannes Schindelin
Quoted 4 lines · Mark Levedahl

The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations.

Agree with transformations on input and output; disagree on diff.

I wonder if this could/should be solved with adding some option to git-diff, similar to --ignore-space-change and --ignore-all-space...

Just a [idle] thought.

Signature · 3 lines
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
#27 lore raw
Quoted 7 lines · Jakub Narebski
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Mark Levedahl

The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations.

Quoted 8 lines
Quoted 2 lines

Agree with transformations on input and output; disagree on diff.

I wonder if this could/should be solved with adding some option to git-diff, similar to --ignore-space-change and --ignore-all-space...

Just a [idle] thought.

That would work. Assuming blobs are stored in with \n, diff just has to open files in 'rt' mode rather than just 'r' and the \r\n are transformed on read so are never seen by git code. That is basically what Windows native tools do, but they also write files opened in 'wt' mode so \n become \r\n on output. Of course, if this were an option, users could look for line ending differences if they cared.

Mark

#28 lore raw
In reply to Jakub Narebski

Hi,

[Cc'ing git list, which I sometimes have to do when Jakub replies]

Quoted 8 lines · Jakub Narebski
Quoted 7 lines · Johannes Schindelin
Quoted 4 lines · Mark Levedahl

The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations.

Quoted 4 lines
Quoted 1 line

Agree with transformations on input and output; disagree on diff.

I wonder if this could/should be solved with adding some option to git-diff, similar to --ignore-space-change and --ignore-all-space...

It could be done, but those options were introduced for CRLF breakage in the first place.

You need --ignore-crlf-breakage? Just holler.

Ciao,
Dscho

#29 lore raw
In reply to Mark Levedahl

В сообщении от Monday 12 February 2007 05:13 Mark Levedahl написал(a):

Quoted 6 lines

1) git is a POSIX only tool (i.e., there will be no \r\n munging), or
2) a Windows port of git will handle and mung \r\n and \n line endings.

If the answer is 1, the mingw port is a waste of time as it simply won't be usable by its target audience. If the answer is 2, then I think a very careful design of this capability is in order.

I am strongly object this statement. I develop one project under Windows and use Cygwin git for this. Yes, I have a problem with git's thinking line ending is a \n but most of troubles are diff and rebase. In general git works well with \r\n line endings.

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

#30 lore raw

Hi,

Quoted 5 lines · Alexander Litvinov

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

That's awful!

Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

Ciao,
Dscho

#31 lore raw

В сообщении от Tuesday 13 February 2007 16:06 Johannes Schindelin написал(a):

Quoted 10 lines

Hi,

Quoted 5 lines · Alexander Litvinov

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

That's awful!

If you are tring to build history that looks good - you are right this is a terrible workflow.

Quoted 6 lines
Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

I still wish to have working rebase so if git will hanle somehow \r\n it would be nice. But please do not produce the same behavior as cvs does: under cygwin it still use \n !

By the way, most windows programmers I work with says 'git is cool but is there gui like tortoise or wincvs ?' :-)

#32 lore raw

Hi,

Quoted 7 lines · Alexander Litvinov

Tuesday 13 February 2007 16:06 Johannes Schindelin:

Quoted 6 lines

At some stage, Alexander wrote this:

Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Quoted 6 lines
Quoted 2 lines

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

I still wish to have working rebase so if git will hanle somehow \r\n it would be nice. But please do not produce the same behavior as cvs does:
under cygwin it still use \n !

You really should teach format-patch to output \n patches, and keep all your blobs CR free.

Quoted 2 lines

By the way, most windows programmers I work with says 'git is cool but is there gui like tortoise or wincvs ?' :-)

Some time ago, I started playing with a shell extension. Now that MinGW git is almost there, I might clean it up... Would you be interested in working on it, or is this just wishtalk?

Ciao,
Dscho

#33 lore raw
In reply to Alexander Litvinov
Quoted 19 lines · Alexander Litvinov

? ????????? ?? Tuesday 13 February 2007 16:06 Johannes Schindelin ???????(a):

Quoted 10 lines

Hi,

Quoted 5 lines · Alexander Litvinov

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

That's awful!

If you are tring to build history that looks good - you are right this is a terrible workflow.

Quoted 4 lines
Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Quoted 5 lines
Quoted 2 lines

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

I still wish to have working rebase so if git will hanle somehow \r\n it would be nice. But please do not produce the same behavior as cvs does:
under cygwin it still use \n !

Cygwin != Windows, Cygwin is a POSIX emulation layer with the explicit goal of providing user tools behaving exactly as they do under Linux, and this includes line ending style.

So, the Cygwin ports of various Linux tools are not expected to satisfy users who want native Win32 behavior. This is where the mingw port of git fits in. Yes, under Cygwin git can track files with \r\n endings, but:
1) Those projects are not portable to non-windows platforms, and
2) As you noted, git will have trouble with rebase, merge, etc. as there is an assumption of \n endings throughout.

A proper win32 port will accept any of \n, \r\n as valid line endings (add \r to support Mac pre-OSX if anyone cares, I still occasionally see such files), treat any of them as semantically equal, and enforce the user's chosen style (\n or \r\n) on output. cvsnt and svn under Windows do this today, serving up "text" files from the same repository with \n endings or \r\n endings depending upon the client, and is what we need a win32 git to do as well.

Mark

#34 lore raw

tisdag 13 februari 2007 20:36 skrev Mark Levedahl:

Quoted 23 lines
Quoted 21 lines · Alexander Litvinov

? ????????? ?? Tuesday 13 February 2007 16:06 Johannes Schindelin ???????(a):

Quoted 10 lines

Hi,

Quoted 5 lines · Alexander Litvinov

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

That's awful!

If you are tring to build history that looks good - you are right this is a terrible workflow.

Quoted 6 lines
Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

Quoted 7 lines
Quoted 3 lines

I still wish to have working rebase so if git will hanle somehow \r\n it would be nice. But please do not produce the same behavior as cvs does:
under cygwin it still use \n !

Cygwin != Windows, Cygwin is a POSIX emulation layer with the explicit goal of providing user tools behaving exactly as they do under Linux, and this includes line ending style.

Line ending style is selectable in cygwin, both on a global level and path level (cygwin mounts). If you use CVS for windows development using CRLF works well and
is the only option if you want to use the same working are with both native CVS clients like TortoiseCVS and the cygwin client. I use the CRLF style by default and LF only for selected directories. The only annoying thing I see is that files transformed by patch end up with LF-only line endings.

Quoted 6 lines

So, the Cygwin ports of various Linux tools are not expected to satisfy users who want native Win32 behavior. This is where the mingw port of git fits in. Yes, under Cygwin git can track files with \r\n endings, but:
1) Those projects are not portable to non-windows platforms, and
2) As you noted, git will have trouble with rebase, merge, etc. as there is an assumption of \n endings throughout.

Even if there is a native port, I'm inclined to want to use the cygwin version anyway because of the nice shell and scripting capabilities and large selection of packages that match what I'm used to in Linux. Git under cygwin should do CRLF transformations according to the same rules that apply to text files in cygwin.

-- robin

#35 lore raw
Quoted 5 lines · Robin Rosenberg

Even if there is a native port, I'm inclined to want to use the cygwin version anyway because of the nice shell and scripting capabilities and large selection of packages that match what I'm used to in Linux. Git under cygwin should do CRLF transformations according to the same rules that apply to text files in cygwin.

-- robin

The cygwin project is explicitly trying to bury the "text" mount option and drive towards binary (= \n line endings) only. They once had a rule that all cygwin programs fully grok \r\n, but that ethic disappeared a couple of years ago, it was just too hard. The cygwin git port itself will not operate on a text mount, it requires a binary mount, so crlf translations are simply not available with git under cygwin.

Mark

#36 lore raw
In reply to Mark Levedahl
Quoted 5 lines · Mark Levedahl

A proper win32 port will accept any of \n, \r\n as valid line endings (add \r to support Mac pre-OSX if anyone cares, I still occasionally see such files), treat any of them as semantically equal, and enforce the user's chosen style (\n or \r\n) on output.

The patch I sent out does that, except right now the "autocrlf" flag is just a pure boolean.

I could easily make it take a ternary value:
- off (normal UNIX semantics - never change anything)
- on (turn CRLF->LF on input, turn LF->CRLF on output)
- input-only (turn CRLF->LF on input, leave LF alone on output)

that would be just a couple of extra lines (almost all of them in the config file parsing logic).

[ The "output-only" case is obviously possible, but insane. It would turn
a LF-only file into CRLF on output, and then not turn it back on input, so doing any "git commit -a" would basically turn every single lines into CRLF, which you do NOT want.

So hopefully that explains the three - not four - cases ]

And the patch already leaves files that the user doesn't touch alone (ie if you check something out with CRLF turned off, and then turn it on in the config, nobody will care - the checked-out copy will have LF-only even if explicitly re-checking it out would turn it into CRLF, but that's fine.

It would be interesting to hear if the patch works for the MinGW people in particular. People using git with a Cygnus environment are probably used to try to keep files with just LF, since they are really trying to do a UNIX environment on top of Windows. But I suspect that WinGW people are more likely to use native Windows tools for things, and then perhaps just a smattering of UNIXy tools..

Linus

#37 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Mark Levedahl

A proper win32 port will accept any of \n, \r\n as valid line endings (add \r to support Mac pre-OSX if anyone cares, I still occasionally see such files), treat any of them as semantically equal, and enforce the user's chosen style (\n or \r\n) on output.

The patch I sent out does that, except right now the "autocrlf" flag is just a pure boolean.

Quoted 7 lines

I could easily make it take a ternary value:
- off (normal UNIX semantics - never change anything)
- on (turn CRLF->LF on input, turn LF->CRLF on output)
- input-only (turn CRLF->LF on input, leave LF alone on output)

Linus

Wow, this is an incredible response: I expected I was going to be studying git internals for a while to get to this point. Thank you!

The ternary value is definitely useful. As noted elsewhere, most tools on windows are very happy with \n ending, few honor those line endings when files are modified, and fewer still allow the user to specify use of \n for new files. However, cygwin tools in particular are not tolerant of crlf, so for that environment it makes sense to banish crlf and the input-only option is most likely the best default setting there.

Mark

#38 lore raw
Quoted 7 lines · Mark Levedahl

The ternary value is definitely useful. As noted elsewhere, most tools on windows are very happy with \n ending, few honor those line endings when files are modified, and fewer still allow the user to specify use of \n for new files. However, cygwin tools in particular are not tolerant of crlf, so for that environment it makes sense to banish crlf and the input-only option is most likely the best default setting there.

Here's a UNTESTED patch on top of the patch I already sent, which allows you to do

	[core]
		AutoCRLF = input

and it should do only the CRLF->LF translation (ie it simplifies CRLF only when reading working tree files, but when checking out files, it leaves the LF alone, and doesn't turn it into a CRLF).

And by "untested" I mean that it looks ok and seems to compile, but I really didn't do anything else.

Linus
---

config.c +4 -0
index ffe0212..e8ae919 100644
@@ -325,6 +325,10 @@ int git_default_config(const char *var, const char *value)
 	}
 
 	if (!strcmp(var, "core.autocrlf")) {
+		if (value && !strcasecmp(value, "input")) {
+			auto_crlf = -1;
+			return 0;
+		}
 		auto_crlf = git_config_bool(var, value);
 		return 0;
 	}
convert.c +1 -1
index c04b6c2..b5a47c2 100644
@@ -133,7 +133,7 @@ int convert_to_working_tree(const char *path, char **bufp, unsigned long *sizep)
 	 * based on filename patterns. Right now we just do the
 	 * stupid auto-CRLF one.
 	 */
-	if (!auto_crlf)
+	if (auto_crlf <= 0)
 		return 0;
 
 	size = *sizep;
environment.c +1 -1
index 2fa0960..570e32a 100644
@@ -28,7 +28,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 int pager_in_use;
 int pager_use_color = 1;
-int auto_crlf = 0;
+int auto_crlf = 0;	/* 1: both ways, -1: only when adding git objects */
 
 static const char *git_dir;
 static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;
#39 lore raw
In reply to Johannes Schindelin
Quoted 9 lines · Johannes Schindelin
Quoted 5 lines · Alexander Litvinov

When I have file that was converted from dos to unix format (or from unix to dos) git genereta big diff. But anyway, c++ compiler works well with both formats and in this case I simply convert file to dos format and git shows again nice diff. If unix format was commited to git I simply change the format and commit that file again.

Quoted 8 lines

That's awful!

Quoted 3 lines

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

No no no.

It's going to be _horrible_ if people start interesting projects in Windows, and there are files in a git repository that are encoded with CRLF.

I'd much rather just get this right, and that means "no hooks". If people start using commit hooks etc, that will just mean that they won't use them for all-windows environments (why use it? Everybody hass CRLF, and everybody _wants_ CRLF), or it will just be relatively expensive to have a complex hook anyway.

So I think we should plan on something like .gitattributes or similar, so that we _can_ handle mixed environments well, without any real setup or any real costs.

The costs really shouldn't be too high - we tend to avoid doing any expensive working tree changes *anyway*. For example, even "git checkout" has a huge optimization to avoid rewriting files that are already ok, so doing things like switching whole branches usually wouldn't even need any conversion for most files - even on platforms like Windows that need the conversion in the first place.

So considering that it looks _trivial_ for git-update-index, fairly easy for diff generation, and I doubt "git checkout" is really likely to be any worse either, this should just be somethign we do.

The *ONLY* case where we may not be able to do things automatically is actually a much more subtle one: "git cat-file". If we just get a SHA1, we don't know what the path to look it up was like, and thus we can never know whether it's a binary or a text object. With "-p" we can trivially guess, of course, but "git cat-file blob" simply must not do that!

But that really doesn't sound like a big problem to me ;)

Linus

#40 lore raw

Hi,

Quoted 11 lines · Linus Torvalds
Quoted 9 lines · Johannes Schindelin
Quoted 3 lines · Alexander Litvinov

The only trouble is the rebase, it does not like \r\n ending and othen produce unexpected merge conflict. But I don't use rebse to othen to realy investigate and try to solve the problem.

Well, if everybody thinks like you, maybe we do not have to change anything for Windows after all?

Quoted 7 lines

No no no.

It's going to be _horrible_ if people start interesting projects in Windows, and there are files in a git repository that are encoded with CRLF.

I'd much rather just get this right, and that means "no hooks".

No hooks means something like cvsnt does, and that means no .gitattributes either. (BTW I really hate .gitattributes, as it does not at all say what this is about; it's about file _conversions_, not attributes).

CVSNT analyzes the files, and guesses if they are text, and only then activates the text mode.

I am strongly opposed to including something like that. (It was already proposed, and your "no hooks" suggests the same.)

However, I am slightly positive about the .gitfiletypes approach, _iff_ we think about more than just text/binary from the start. If we do it right, it will buy us more.

Ciao,
Dscho

#41 lore raw
Quoted 7 lines · Johannes Schindelin

No hooks means something like cvsnt does, and that means no .gitattributes either. (BTW I really hate .gitattributes, as it does not at all say what this is about; it's about file _conversions_, not attributes).

However, I am slightly positive about the .gitfiletypes approach, _iff_ we think about more than just text/binary from the start. If we do it right, it will buy us more.

We might start with only binary/text attributes, but we may add more later, e.g. chmod=o-rwx. I do not see much differnece between attributes vs filetypes.

#42 lore raw
In reply to Johannes Schindelin
Quoted 4 lines · Johannes Schindelin

No hooks means something like cvsnt does, and that means no .gitattributes either. (BTW I really hate .gitattributes, as it does not at all say what this is about; it's about file _conversions_, not attributes).

No, it *is* about attributes.

In order to know how to convert, you need to know the attributes of the file.

So it's not about conversion: we would ALWAYS do conversion. It's about the fact that in order to do the conversion, we need to know what the attributes of the file is - is it text, or what.

And the equal point is that there are _other_ attributes that git might care about. The "merge strategy" attribute, for example. Or "owner" attributes for files etc.

Linus

#43 lore raw

Hi,

Quoted 11 lines · Linus Torvalds
Quoted 4 lines · Johannes Schindelin

No hooks means something like cvsnt does, and that means no .gitattributes either. (BTW I really hate .gitattributes, as it does not at all say what this is about; it's about file _conversions_, not attributes).

No, it *is* about attributes.

In order to know how to convert, you need to know the attributes of the file.

Quoted 7 lines

So it's not about conversion: we would ALWAYS do conversion. It's about the fact that in order to do the conversion, we need to know what the attributes of the file is - is it text, or what.

And the equal point is that there are _other_ attributes that git might care about. The "merge strategy" attribute, for example. Or "owner" attributes for files etc.

Yes, you're right. Colour me converted (pun intended).

Ciao,
Dscho

#44 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds

I'd much rather just get this right, and that means "no hooks". If people start using commit hooks etc, that will just mean that they won't use them for all-windows environments (why use it? Everybody hass CRLF, and everybody _wants_ CRLF), or it will just be relatively expensive to have a complex hook anyway.

So I think we should plan on something like .gitattributes or similar, so that we _can_ handle mixed environments well, without any real setup or any real costs.

Here's a patch that I think we can merge right now. There may be other places that need this, but this at least points out the three places that read/write working tree files for git update-index, checkout and diff respectively. That should cover a lot of it.

Some day we can actually implement it. In the meantime, this points out a place for people to start. We *can* even start with a really simple "we do CRLF conversion automatically, regardless of filename" kind of approach, that just look at the data (all three cases have the _full_ file data already in memory) and says "ok, this is text, so let's convert to/from DOS format directly".

THAT somebody can write in ten minutes, and it would already make git much nicer on a DOS/Windows platform, I suspect.

And it would be totally zero-cost if you just make it a config option (but please make it dynamic with the _default_ just being 0/1 depending on whether it's UNIX/Windows, just so that UNIX people can _test_ it easily).

Linus

#45 lore raw
Quoted 4 lines · Linus Torvalds

Here's a patch that I think we can merge right now. There may be other places that need this, but this at least points out the three places that read/write working tree files for git update-index, checkout and diff respectively. That should cover a lot of it.

Thanks, applied. I think git-apply has separate codepaths for both reading and writing; I won't look into them before 1.5.0 but people are welcome to help advancing the cause before I get to it ;-).

#46 lore raw
Quoted 5 lines · Junio C Hamano

Thanks, applied. I think git-apply has separate codepaths for both reading and writing; I won't look into them before 1.5.0 but people are welcome to help advancing the cause before I get to it ;-).

Actually, I did it myself.

This is a "lazy man's auto-CRLF", and it really is pretty simple.

It currently does NOT know about file attributes, so it does its conversion purely based on content. Maybe that is more in the "git philosophy" anyway, since content is king, but I think we should try to do the file attributes to turn it off on demand.

Anyway, BY DEFAULT it is off regardless, because it requires a

	[core]
		AutoCRLF = true

in your config file to be enabled. We could make that the default for Windows, of course, the same way we do some other things (filemode etc).

But you can actually enable it on UNIX, and it will cause:

- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF

and things work fine.

Funnily, it actually shows an odd file in git itself:

git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff

shows a diff for "Documentation/docbook-xsl.css". Why? Because we have actually checked in that file *with* CRLF! So when "core.autocrlf" is true, we'll always generate a *different* hash for it in the index, because the index hash will be for the content _without_ CRLF.

Is this complete? I dunno. It seems to work for me. It doesn't use the filename at all right now, and that's probably a deficiency (we could certainly make the "is_binary()" heuristics also take standard filename heuristics into account).

I don't pass in the filename at all for the "index_fd()" case (git-update-index), so that would need to be passed around, but this actually works fine.

NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours truly. I will not guarantee that they work at all reasonable. Caveat emptor. But it _is_ simple, and it _is_ safe, since it's all off by default.

The patch is pretty simple - the biggest part is the new "convert.c" file, but even that is really just basic stuff that anybody can write in "Teaching C 101" as a final project for their first class in programming. Not to say that it's bug-free, of course - but at least we're not talking about rocket surgery here.

Linus

---
commit f0731319497ac8121bd901a91fc33d715745d3af

Author: Linus Torvalds <[email protected]>
Date:   Tue Feb 13 10:56:50 2007 -0800

Add "auto-CRLF" conversion logic

It's simple and it's stupid. But it actually seems to work. What more can you want?

It's not enabled by default: you need to add a

[core]
AutoCRLF = true

to your .git/config file to enable it universally.

---
 Makefile      |    3 +-
 cache.h       |    5 ++
 config.c      |    5 ++
 convert.c     |  179 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 diff.c        |   16 +++++
 entry.c       |   15 +++++
 environment.c |    1 +
 sha1_file.c   |   22 +++++++-
 8 files changed, 244 insertions(+), 2 deletions(-)
 create mode 100644 convert.c
Makefile +2 -1
index 40bdcff..60496ff 100644
@@ -262,7 +262,8 @@ LIB_OBJS = \
 	revision.o pager.o tree-walk.o xdiff-interface.o \
 	write_or_die.o trace.o list-objects.o grep.o \
 	alloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \
-	color.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o
+	color.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o \
+	convert.o
 
 BUILTIN_OBJS = \
 	builtin-add.o \
cache.h +5 -0
index c62b0b0..9c019e8 100644
@@ -201,6 +201,7 @@ extern const char *apply_default_whitespace;
 extern int zlib_compression_level;
 extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
+extern int auto_crlf;
 
 #define GIT_REPO_VERSION 0
 extern int repository_format_version;
@@ -468,4 +469,8 @@ extern int nfvasprintf(char **str, const char *fmt, va_list va);
 extern void trace_printf(const char *format, ...);
 extern void trace_argv_printf(const char **argv, int count, const char *format, ...);
 
+/* convert.c */
+extern int convert_to_git(const char *path, char **bufp, unsigned long *sizep);
+extern int convert_to_working_tree(const char *path, char **bufp, unsigned long *sizep);
+
 #endif /* CACHE_H */
config.c +5 -0
index d821071..ffe0212 100644
@@ -324,6 +324,11 @@ int git_default_config(const char *var, const char *value)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.autocrlf")) {
+		auto_crlf = git_config_bool(var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "user.name")) {
 		strlcpy(git_default_name, value, sizeof(git_default_name));
 		return 0;
convert.c new file +179 -0

179 lines not on this page · raw message

diff.c +16 -0
index aaab309..561587c 100644
@@ -1332,6 +1332,9 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
 	    reuse_worktree_file(s->path, s->sha1, 0)) {
 		struct stat st;
 		int fd;
+		char *buf;
+		unsigned long size;
+
 		if (lstat(s->path, &st) < 0) {
 			if (errno == ENOENT) {
 			err_empty:
@@ -1364,6 +1367,19 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
 		s->data = xmmap(NULL, s->size, PROT_READ, MAP_PRIVATE, fd, 0);
 		close(fd);
 		s->should_munmap = 1;
+
+		/*
+		 * Convert from working tree format to canonical git format
+		 */
+		buf = s->data;
+		size = s->size;
+		if (convert_to_git(s->path, &buf, &size)) {
+			munmap(s->data, s->size);
+			s->should_munmap = 0;
+			s->data = buf;
+			s->size = size;
+			s->should_free = 1;
+		}
 	}
 	else {
 		char type[20];
entry.c +15 -0
index 0ebf0f0..472a9ef 100644
@@ -78,6 +78,9 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 			path, sha1_to_hex(ce->sha1));
 	}
 	switch (ntohl(ce->ce_mode) & S_IFMT) {
+		char *buf;
+		unsigned long nsize;
+
 	case S_IFREG:
 		if (to_tempfile) {
 			strcpy(path, ".merge_file_XXXXXX");
@@ -89,6 +92,18 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 			return error("git-checkout-index: unable to create file %s (%s)",
 				path, strerror(errno));
 		}
+
+		/*
+		 * Convert from git internal format to working tree format
+		 */
+		buf = new;
+		nsize = size;
+		if (convert_to_working_tree(ce->name, &buf, &nsize)) {
+			free(new);
+			new = buf;
+			size = nsize;
+		}
+
 		wrote = write_in_full(fd, new, size);
 		close(fd);
 		free(new);
environment.c +1 -0
index 54c22f8..2fa0960 100644
@@ -28,6 +28,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 int pager_in_use;
 int pager_use_color = 1;
+int auto_crlf = 0;
 
 static const char *git_dir;
 static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;
sha1_file.c +21 -1
index 0d4bf80..6ec67b2 100644
@@ -2082,7 +2082,7 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, con
 {
 	unsigned long size = st->st_size;
 	void *buf;
-	int ret;
+	int ret, re_allocated = 0;
 
 	buf = "";
 	if (size)
@@ -2091,10 +2091,30 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, con
 
 	if (!type)
 		type = blob_type;
+
+	/*
+	 * Convert blobs to git internal format
+	 */
+	if (!strcmp(type, blob_type)) {
+		unsigned long nsize = size;
+		char *nbuf = buf;
+		if (convert_to_git(NULL, &nbuf, &nsize)) {
+			if (size)
+				munmap(buf, size);
+			size = nsize;
+			buf = nbuf;
+			re_allocated = 1;
+		}
+	}
+
 	if (write_object)
 		ret = write_sha1_file(buf, size, type, sha1);
 	else
 		ret = hash_sha1_file(buf, size, type, sha1);
+	if (re_allocated) {
+		free(buf);
+		return ret;
+	}
 	if (size)
 		munmap(buf, size);
 	return ret;
#47 lore raw
Quoted 7 lines

Anyway, BY DEFAULT it is off regardless, because it requires a

	[core]
		AutoCRLF = true

in your config file to be enabled. We could make that the default for Windows, of course, the same way we do some other things (filemode etc).

This whole auto CRLF things seems to deal with DOS issues that I personally have not encountered since looong time ago.
Granted notepad in Windows does not understand UNIX files but that a bug in notepad and everyone knows that wordpad can be used.

I wonder what we are really trying to address here. Or in other words could the original poster maybe tell what Windows IDE's that does not handle UNIX files properly?

core git today should not care about CRLF as opposed to LF end-of-line as long as the end-of-line is consistent - correct?

So defaulting to autoCRLF in Windows/DOS environments was maybe sane 10 years ago but today that seems to be the wrong thing to do. For certain project the option could be useful if the tool-set in the project *requires* CRLF, but if the toolset like all modern toolset supports both CRLF and LF then git better avoid changing end-of-line marker.

Sam

#48 lore raw

В сообщении от Wednesday 14 February 2007 02:42 Sam Ravnborg написал(a):

Quoted 3 lines

I wonder what we are really trying to address here. Or in other words could the original poster maybe tell what Windows IDE's that does not handle UNIX files properly?

MS VC has text file for project file but don't like \n line endings, only \r\n.

#49 lore raw
In reply to Sam Ravnborg
Quoted 5 lines · Sam Ravnborg

I wonder what we are really trying to address here. Or in other words could the original poster maybe tell what Windows IDE's that does not handle UNIX files properly?

Quoted 8 lines

core git today should not care about CRLF as opposed to LF end-of-line as long as the end-of-line is consistent - correct?

So defaulting to autoCRLF in Windows/DOS environments was maybe sane 10 years ago but today that seems to be the wrong thing to do. For certain project the option could be useful if the tool-set in the project *requires* CRLF, but if the toolset like all modern toolset supports both CRLF and LF then git better avoid changing end-of-line marker.

I've actually run into grief on this subject with perl scripts within the last year (files from windows systems with crlf not working cleanly on a linux system with just lf)

this is real, not just historic

David Lang

#50 lore raw
In reply to Sam Ravnborg
Quoted 3 lines · Sam Ravnborg

This whole auto CRLF things seems to deal with DOS issues that I personally have not encountered since looong time ago.

Maybe you stopped using DOS a loong time ago ;)

It's definitely an issue. Yes, all windows programs basically *understand* files that have just LF. But almost all of them will *write* files with CRLF.

(Which means that I suspect I made the default for "auto_crlf" be wrong in my patch: I probably should not default to checking out with CRLF, but checking out with just LF, and only do the CRLF->LF conversion on input).

Anybody who has ever worked with _any_ Windows people have long since learnt that they always end up having to convert CRLF to just LF when they get files. Even _I_ know it, and I seldom have to work with people who use Windows ;)

So it's a good idea to try to make sure that Windows users don't corrupt files by adding CRLF where there is no need for them into a git archive. We hope to convert those people to a real OS some day ("here's a nickel, boy"), and to make it easier for them to do it, making sure that their projects in -git are already in a sane format is probably a good idea.

Linus

#51 lore raw
Quoted 4 lines
Quoted 2 lines

This whole auto CRLF things seems to deal with DOS issues that I personally have not encountered since looong time ago.

Maybe you stopped using DOS a loong time ago ;)

Unfortunately not. (Sitting with a Windows 2000 laptop atm but saved by ssh).

Quoted 4 lines

It's definitely an issue. Yes, all windows programs basically *understand* files that have just LF. But almost all of them will *write* files with CRLF.

So the issue with git supporting CRLF -> LF is to make interoperability between UNIX* programs and Windows programs which is anohter domain.

My main objective is the proposal to make a conversion default when many users do not need it. For the UNIX* compatibility thing having conversion at lowest layer make sense.

Quoted 3 lines

(Which means that I suspect I made the default for "auto_crlf" be wrong in my patch: I probably should not default to checking out with CRLF, but checking out with just LF, and only do the CRLF->LF conversion on input).

Expect that it seems a few br0ken programs yet does not support LF as end-of-line marker - so .gitattriutes make take special care here.

Sam

#52 lore raw
Quoted 7 lines · Sam Ravnborg
Quoted 4 lines

(Which means that I suspect I made the default for "auto_crlf" be wrong in my patch: I probably should not default to checking out with CRLF, but checking out with just LF, and only do the CRLF->LF conversion on input).

Expect that it seems a few br0ken programs yet does not support LF as end-of-line marker - so .gitattriutes make take special care here.

Yes, but I also think that even without .gitattributes, you just want to have a default for what "text" actually means, and it's entirely possible that the default should be: "check out with just LF, and on check-in turn CRLF into LF".

But exactly because _some_ programs might want to always see CRLF on input too, it should be overridable.

Or maybe the default should be "turn into CRLF", and there should just be an option to make it check out as LF-only.

Regardless, I think that is independent of ".gitattributes". The _attribute_ should be "text", but what it then means in practice is a separate flag.

And yes, we *could* have a per-file attribute ("text,crlf-checkout") which could be used to say "I want to always check out as crlf regardless of any other policy") and the same for lf-only, but I seriously doubt that anybody really needs that kind of knob-tweaking. At some point it's just fine to say "you're crazy".

Linus

#53 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Sam Ravnborg
Quoted 4 lines

(Which means that I suspect I made the default for "auto_crlf" be wrong in my patch: I probably should not default to checking out with CRLF, but checking out with just LF, and only do the CRLF->LF conversion on input).

Quoted 7 lines
Quoted 2 lines

Expect that it seems a few br0ken programs yet does not support LF as end-of-line marker - so .gitattriutes make take special care here.

Yes, but I also think that even without .gitattributes, you just want to have a default for what "text" actually means, and it's entirely possible that the default should be: "check out with just LF, and on check-in turn CRLF into LF".

The definition of what is "text" and what action to take upon check-in / check-out of text is two sepearate things.

I could see it as beneficial as a per-project or even as an overall git-policy to say "checkin-as-LF" - "checkout-as-LF" to overcome interoperability issues when more tools gets UNIX* based.

Quoted 3 lines

But exactly because _some_ programs might want to always see CRLF on input too, it should be overridable.

Which is where I see .gitattributes come into play.
-> A rule that says files with extension .prj and of type "text" shall not see any conversion.

In this way almost all "text" over time get a proper format and the remaining brain-dead tools that continue to save in CRLF format will not destroy the sane LF format.

If anything gets defualt I would vote for LF. But overrideable.

My editor-of-choice does eol auto-sense. If I recall correct it scans the first 200 lines and counts number of CR,LF,CRLF and based on this judge the actual eol character used. But not all editors are that sensible :-(

Sam

#54 lore raw
In reply to Sam Ravnborg
Quoted 2 lines · Sam Ravnborg

This whole auto CRLF things seems to deal with DOS issues that I personally have not encountered since looong time ago.

Maybe you didn't share a work environment with Windows users since looong time ago.

Quoted 6 lines

Granted notepad in Windows does not understand UNIX files but that a bug in notepad and everyone knows that wordpad can be used.

I wonder what we are really trying to address here. Or in other words could the original poster maybe tell what Windows IDE's that does not handle UNIX files properly?

Windows IDE's can _create_files. Those files will be CRLF infected.

Also some of them read UNIX files just fine but they will use CRLF to end new added lines despite the rest of the file using only LF.

Quoted 2 lines

core git today should not care about CRLF as opposed to LF end-of-line as long as the end-of-line is consistent - correct?

Consistency won't come alone if not enforced in some way.

Quoted 5 lines

So defaulting to autoCRLF in Windows/DOS environments was maybe sane 10 years ago but today that seems to be the wrong thing to do. For certain project the option could be useful if the tool-set in the project *requires* CRLF, but if the toolset like all modern toolset supports both CRLF and LF then git better avoid changing end-of-line marker.

Rather git better enforce consistency otherwise it'll be only a mix of possible combination as soon as Windows and UNIX users work on the same project.

Nicolas

#55 lore raw
In reply to Linus Torvalds

В сообщении от Wednesday 14 February 2007 01:07 Linus Torvalds написал:

Quoted 3 lines

Actually, I did it myself.

This is a "lazy man's auto-CRLF", and it really is pretty simple.

Wow ! Thanks.

I just tried this patch and it works! From now I can use git-cvsimport under Linux and then clone it to cygwin and work there with full history. Nice, very nice. In my case text file detection work well as far most of our files are .cpp and .h

#56 lore raw
Quoted 4 lines · Alexander Litvinov

I just tried this patch and it works! From now I can use git-cvsimport under Linux and then clone it to cygwin and work there with full history. Nice, very nice.

Btw, it didn't do any commit message conversion etc, so you'll still always see commit messages with LF-only, and if you _create_ commits, you need to make sure that whatever program you use will do the right thing.

Quoted 2 lines

In my case text file detection work well as far most of our files are .cpp and .h

Yeah, considering that it worked in my testing for "git" itself, I'm not surprised. Source code tends to look the same..

Linus

#57 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Alexander Litvinov

I just tried this patch and it works! From now I can use git-cvsimport under Linux and then clone it to cygwin and work there with full history. Nice, very nice.

Btw, it didn't do any commit message conversion etc, so you'll still always see commit messages with LF-only, and if you _create_ commits, you need to make sure that whatever program you use will do the right thing.

I think stripspace removes CR so we should be Ok.

#58 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Junio C Hamano

Thanks, applied. I think git-apply has separate codepaths for both reading and writing; I won't look into them before 1.5.0 but people are welcome to help advancing the cause before I get to it ;-).

Actually, I did it myself.

This is a "lazy man's auto-CRLF", and it really is pretty simple.

Thanks a lot, busy beaver! I gave this a quick spin with a few interesting operations: merges and rebase. Merges leave the merge results with only LFs behind. Rebasing seems to work as expected (working files have CRLFs), except when merges are needed.

Doesn't git-unpack-file also need to call into the converter?

-- Hannes

#59 lore raw
Quoted 4 lines · Johannes Sixt

Thanks a lot, busy beaver! I gave this a quick spin with a few interesting operations: merges and rebase. Merges leave the merge results with only LFs behind.

Yes. Merge uses "git-cat-file" (well, it historically did, now that it's built-in it still does the equivalent operation).

I already talked about how git-cat-file was special ;)

Quoted 2 lines

Rebasing seems to work as expected (working files have CRLFs), except when merges are needed.

Well, it always "merges", but yes, you mean three-way data merges. The normal SHA1-direct merges will just use the normal git-read-tree thing which is the same as checkout.

Quoted 1 line

Doesn't git-unpack-file also need to call into the converter?

See earlier discussions. git-cat-file (and git-unpack-file, which is just a version of it, really) don't have the original filename, so we'll need to extend on it some way in order to support file attributes even in theory. So before we do that, I'd hate to do any format conversion there.

Yes, yes, right now it ignores the filename *anyway*, but the point is, right now that's a "small implementation detail". I would NOT want to do this if I couldn't know the filename at all!

The merge algorithms actually obviously *do* know the filename fo the things that they are going to merge, so the filename information does exists. It's just not passed on far enough.

Finally, one comment: if you use "autocrlf = input" (my second patch), all of this works even now, since the default is to just leave things as LF-only anyway. In fact, even with "autocrlf = on", nothing should really *break* except for silly editors that actuall *require* CRLF.

IOW, it's more important to do the CRLF->LF conversion than it is to do the LF->CRLF one ;)

Linus

#60 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours truly. I will not guarantee that they work at all reasonable. Caveat emptor. But it _is_ simple, and it _is_ safe, since it's all off by default.

It might be safe for some definition of safe, but it is very Asian unfriendly.

I'd probably suggest replacing it with what GNU diff uses, which we stolen and implemented in diff.c::mmfile_is_binary().

#61 lore raw
Quoted 6 lines · Junio C Hamano

It might be safe for some definition of safe, but it is very Asian unfriendly.

I'd probably suggest replacing it with what GNU diff uses, which we stolen and implemented in diff.c::mmfile_is_binary().

Well, the thing is, mmfile_is_binary() doesn't really have a big downside if it's wrong one way or the other.

In contrast CR->CRLF conversion, if wrong, actually corrupts binary files. So I felt it was better to be really safe than sorry. It's *much* better to miss some CRLF translation than to do too much of it.

That said, I'm sure it could be improved a lot. In particular, characters in the range 0x00 - 0x1f are clearly "more binary" than the 0x7f+ range, with the obvious exceptions (tab, cr, lf).

0x00 - which is the only one mmfile_is_binart() uses - is arguably the "most binary" one, of course, but it might be interesting to give different weights to the whole range.. In particular, especially for small files, the fact that there is no 0x00 byte in no way indicates that it's not "binary".

This whole issue is obviously one reason I'd like to involve the filename itself, and make it use a ".gitattributes" file - exactly because that allows you to be much more aggressive and more precise.

(0x00 may be one of the more _common_ characters in many binary files, which makes it a good character to search for too, so I don't really have any hugely strong opinions here. After all, the whole heuristic is off by default anyway, so it's "really safe" ;^)

Linus

#62 lore raw

Hi,

Quoted 5 lines · Linus Torvalds

0x00 - which is the only one mmfile_is_binart() uses - is arguably the "most binary" one, of course, but it might be interesting to give different weights to the whole range.. In particular, especially for small files, the fact that there is no 0x00 byte in no way indicates that it's not "binary".

Last time I checked, the text files never had lines longer than 200 characters (I chose this intentionally large). So, it might be a good heuristic to check the maximal line length, and refuse to believe that it's text once a certain (configurable) threshold is reached.

Ciao,
Dscho

#63 lore raw
Quoted 4 lines · Johannes Schindelin

Last time I checked, the text files never had lines longer than 200 characters (I chose this intentionally large). So, it might be a good heuristic to check the maximal line length, and refuse to believe that it's text once a certain (configurable) threshold is reached.

Ciao,
Dsch

Unfortunately, on my program we have folks using text files with single lines over 60,000 characters long, these are data files. Think for example of a comma or tab separated data file saved from a spreadsheet. In this case, the files are pure ascii. So, the line length could be something else to take into account, but is not decisive by itself.

To recap, we have the following various suggestions to determine textness:

1) ratio of ascii to non-ascii characters, possibly weighting some chars more than others
2) line length
3) existence of a null (\0)
4) file name globbing
5) roundtrip ( lf(crlf(file) ) == file

I don't think any one suggestion is completely adequate for all uses, all need to be available, somehow configurable. This suggests to me a core.AutoCRLFstrategy variable that is a comma separated list of methods to use (set to a reasonable default of course that does not cause runtime headaches on Unix): a file would be deemed binary unless all listed methods declare the file as text (with an empty list disabling AutoCRLF detection).

Mark

#64 lore raw

Hi,

Quoted 5 lines · Mark Levedahl

This suggests to me a core.AutoCRLFstrategy variable that is a comma separated list of methods to use (set to a reasonable default of course that does not cause runtime headaches on Unix): a file would be deemed binary unless all listed methods declare the file as text (with an empty list disabling AutoCRLF detection).

This sounds regretfully complex. Somebody (you?) mentioned that cvsnt does a kick-ass job here. Does cvsnt need strategies? I don't think so. Neither do we. Someone who cares enough should just rip^H^H^Hlook at cvsnt's text detection.

Ciao,
Dscho

#65 lore raw
Quoted 9 lines · Johannes Schindelin

Hi,

On Wed, 14 Feb 2007, Mark Levedahl wrote:

This sounds regretfully complex. Somebody (you?) mentioned that cvsnt does a kick-ass job here. Does cvsnt need strategies? I don't think so. Neither do we. Someone who cares enough should just rip^H^H^Hlook at cvsnt's text detection.

Ciao,
Dscho

I agree that is complex, I started thinking of PAM when I wrote that, leading to, "this aint gonna work." But in the modern day let's all feel good spirit of "there are no stupid ideas, just some are better" I threw it out anyway.

As to cvsnt, my actual feeling is I'd like to kick it in the ass, it has destroyed too many files for me over the years, binary and text, so I don't think its strategies are very good. That is why I'm kicking these ideas around, if I thought I knew the "right" way I would have written it already.

Mark

#66 lore raw

onsdag 14 februari 2007 18:28 skrev Mark Levedahl:

Quoted 5 lines

As to cvsnt, my actual feeling is I'd like to kick it in the ass, it has destroyed too many files for me over the years, binary and text, so I don't think its strategies are very good. That is why I'm kicking these ideas around, if I thought I knew the "right" way I would have written it already.

That may be why an excellent piece of software, TortoiseCVS, doesn't trust cvs or cvsnt to do the job. Here is how they do the binary detection (and some more):

http://tortoisecvs.cvs.sourceforge.net/tortoisecvs/TortoiseCVS/src/CVSGlue/CVSStatus.cpp?revision=1.172&view=markup

-- robin

#67 lore raw
Quoted 6 lines · Robin Rosenberg

That may be why an excellent piece of software, TortoiseCVS, doesn't trust cvs or cvsnt to do the job. Here is how they do the binary detection (and some more):

http://tortoisecvs.cvs.sourceforge.net/tortoisecvs/TortoiseCVS/src/CVSGlue/CVSStatus.cpp?revision=1.172&view=markup

Well, it does seem to boil down to what Junio already got to:

- 0-31 and 127 are never in text, except for BEL, BS, HT, LF, FF, CR and ESC.
- 128-255 can all be in either iso-8859 or extended ascii (or they explicitly add NEL but not 128+27 to "normal ASCII", which is strange)

So they've effectively added BEL and ESC to the listof characters that Junio has now. But they also make it an absolute error to have anything else (no "1% rule").

But they also do the filename tests, and I think that's more important in many ways.

Linus

#68 lore raw

onsdag 14 februari 2007 19:31 skrev Linus Torvalds:

Quoted 9 lines · Robin Rosenberg
Quoted 6 lines

That may be why an excellent piece of software, TortoiseCVS, doesn't trust cvs or cvsnt to do the job. Here is how they do the binary detection (and some more):

http://tortoisecvs.cvs.sourceforge.net/tortoisecvs/TortoiseCVS/src/CVSGlue/CVSStatus.cpp?revision=1.172&view=markup

Quoted 8 lines

Well, it does seem to boil down to what Junio already got to:

- 0-31 and 127 are never in text, except for BEL, BS, HT, LF, FF, CR and ESC.
- 128-255 can all be in either iso-8859 or extended ascii (or they explicitly add NEL but not 128+27 to "normal ASCII", which is strange)

So they've effectively added BEL and ESC to the listof characters that

Especially ESC used to be common in DOS/Windows and quite a few hang around in older code.

Quoted 2 lines

Junio has now. But they also make it an absolute error to have anything else (no "1% rule").

Can this 1%-rule be motivated from real cases, rather that hypotetical ones? It makes it harder to understand why the tools makes a particular decision.

Quoted 2 lines

But they also do the filename tests, and I think that's more important in many ways.

A unixy tool like git should maybe use magic too :).

Btw the filename (like .gitignore or similar) test in practice would give us
the binary flag. Just list a filename instead of a pattern.

-- robin

#69 lore raw
In reply to Johannes Schindelin
Quoted 5 lines · Johannes Schindelin

This sounds regretfully complex. Somebody (you?) mentioned that cvsnt does a kick-ass job here. Does cvsnt need strategies? I don't think so. Neither do we. Someone who cares enough should just rip^H^H^Hlook at cvsnt's text detection.

Well, one thing to keep in mind is that for source code in particular, this really very seldom is an issue.

So you can do a really *bad* job in theory, and in practice it really works very very well.

Very few people keep binary blobs in any SCM archive _anyway_, partly because they've always been told that it's unsafe (and with a lot of SCM's it is), but even more because binary blobs are almost always generated by some build method, so normally you'd never version them in the first place, or versioning isn't all that helpful.

And most binary blobs are so *obviously* binary that even the stupidest algorithm on earth will get it right. The only hard cases actually tend to be really tiny files, or literally test-sequences.

Tiny files are hard because:

- they (by being tiny) have so few characters that they can easily lack a "fingerprint" character (eg a NUL character or similar).

- tiny files are a lot more likely than bigger files to have strange statistics that throw some more "sophisticated" rule off the scent. Something like a "10% rule" tends to work fine if you have a big text, and ten percent is still a reasonable number to average things out over, but what if you only had ten characters to begin with?

The good news is that tiny files can usually be considered text, since you'd seldom use a binary format for something really small anyway.

So I suspect that IN PRACTICE, especially if you come as a CVS replacement (where binary files are just damn hard to get right even under the best of circumstances!), you can do just about anything, including just saying "everything is text", and you'd be fine.

It's entirely possible that that is exactly what CVSNT does ;)

Linus

#70 lore raw
In reply to Mark Levedahl
Quoted 9 lines · Mark Levedahl

To recap, we have the following various suggestions to determine textness:

1) ratio of ascii to non-ascii characters, possibly weighting some chars more than others
2) line length
3) existence of a null (\0)
4) file name globbing
5) roundtrip ( lf(crlf(file) ) == file

Actually, my patch already had one that you didn't mention:
6) CR never shows up alone.

So the patch I sent out basicallyhad the following rules:
- no more than ~10% of all characters being other than regular printable ASCII (where any control character except for newline/cr/tab was deemed nonprintable)
- any "lonely" CR automatically means it's binary, and I would refuse to convert that to a LF (the test in the code is that CRLF count must match CR count)

but the "roundtrip" rule is much too strict (it's actually perfectly possible for an editor to add CRLF characters only to new _lines_, leaving old lines with just LF - or the other way around. In fact, the editor I use under Linux does exactly that in reverse - if I add new lines, it will add those without CR, but will leave old lines with CRLF alone).

I think that to help asian languages (or strange text-files in utf8 or Latin1 too, for that matter: test-files with _just_ special characters), I should probably make the rule be that only the 0-31 range is special.

Linus

#71 lore raw
Quoted 2 lines · Linus Torvalds

Actually, my patch already had one that you didn't mention:
6) CR never shows up alone.

Older Macs ;-)?

Quoted 11 lines

So the patch I sent out basicallyhad the following rules:
- no more than ~10% of all characters being other than regular printable ASCII (where any control character except for newline/cr/tab was deemed nonprintable)
- any "lonely" CR automatically means it's binary, and I would refuse to convert that to a LF (the test in the code is that CRLF count must match CR count)
...
I think that to help asian languages (or strange text-files in utf8 or Latin1 too, for that matter: test-files with _just_ special characters), I should probably make the rule be that only the 0-31 range is special.

I would agree. 0-31 except HT, CR, LF and ESC would be a good idea; that would not harm text in UTF-8, EUC based various locales nor ISO 2022.

Patch is relative to 'pu'.
-- >8 --

convert.c +5 -7
index ebcf717..b6b7c66 100644
@@ -13,7 +13,7 @@ struct text_stat {
 	unsigned cr, lf, crlf;
 
 	/* These are just approximations! */
-	unsigned printable, nonprintable, nul;
+	unsigned printable, nonprintable;
 };
 
 static void gather_stats(const char *buf, unsigned long size, struct text_stat *stats)
@@ -34,13 +34,11 @@ static void gather_stats(const char *buf, unsigned long size, struct text_stat *
 			stats->lf++;
 			continue;
 		}
-		if (c == '\t' || (c >= 32 && c < 127)) {
-			stats->printable++;
+		if ((c < 32) && (c != '\t' && c != '\033')) {
+			stats->nonprintable++;
 			continue;
 		}
-		if (!c)
-			stats->nul++;
-		stats->nonprintable++;
+		stats->printable++;
 	}
 }
 
@@ -50,7 +48,7 @@ static void gather_stats(const char *buf, unsigned long size, struct text_stat *
 static int is_binary(unsigned long size, struct text_stat *stats)
 {
 
-	if (stats->nul)
+	if (stats->nonprintable)
 		return 1;
 	/*
 	 * Other heuristics? Average line length might be relevant,
#72 lore raw
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Linus Torvalds

Actually, my patch already had one that you didn't mention:
6) CR never shows up alone.

Older Macs ;-)?

Yeah, I think we can ignore them..

Let's see if anybody ever complains ;)

Quoted 3 lines

I would agree. 0-31 except HT, CR, LF and ESC would be a good idea; that would not harm text in UTF-8, EUC based various locales nor ISO 2022.

You could possibly add 127 to the list too (it's ascii DEL, I don't know if you should ever see it in anything that has anything to do with text).

convert.c static void gather_stats(const char *buf, unsigned long size, struct text_stat * · line 51

Quoted 2 lines
convert.c +1 -1
-	if (stats->nul)
+	if (stats->nonprintable)

But this is too harsh.

It's quite common to have the occasional FF character. Some things really do use it for page breaks. So saying that *any* nonprintable character is bad is not a good idea.

Same goes for BS (some programs use it to show bold and underlined text:
man-pages, for example).

Linus

#73 lore raw
Quoted 11 lines · Linus Torvalds

Review

Quoted 2 lines
(quoted diff) +1 -1
-	if (stats->nul)
+	if (stats->nonprintable)

But this is too harsh.

It's quite common to have the occasional FF character. Some things really do use it for page breaks. So saying that *any* nonprintable character is bad is not a good idea.

Same goes for BS (some programs use it to show bold and underlined text:
man-pages, for example).

Ok. How about adding BS and FF to the Ok set, and checking if bad ones are less than 1% of the good ones?

convert.c +15 -4
index b6b7c66..b0c7641 100644
@@ -34,11 +34,22 @@ static void gather_stats(const char *buf, unsigned long size, struct text_stat *
 			stats->lf++;
 			continue;
 		}
-		if ((c < 32) && (c != '\t' && c != '\033')) {
+		if (c == 127)
+			/* DEL */
 			stats->nonprintable++;
-			continue;
+		else if (c < 32) {
+			switch (c) {
+				/* BS, HT, ESC and FF */
+			case '\b': case '\t': case '\033': case '\014':
+				stats->printable++;
+				break;
+			default:
+				stats->nonprintable++;
+			}
+			
 		}
-		stats->printable++;
+		else
+			stats->printable++;
 	}
 }
 
@@ -48,7 +59,7 @@ static void gather_stats(const char *buf, unsigned long size, struct text_stat *
 static int is_binary(unsigned long size, struct text_stat *stats)
 {
 
-	if (stats->nonprintable)
+	if ((stats->printable >> 7) < stats->nonprintable)
 		return 1;
 	/*
 	 * Other heuristics? Average line length might be relevant,
#74 lore raw
Quoted 3 lines · Junio C Hamano

Ok. How about adding BS and FF to the Ok set, and checking if bad ones are less than 1% of the good ones?

I think that looks fine.

Linus

#75 lore raw
In reply to Johannes Schindelin
Quoted 4 lines · Johannes Schindelin

Last time I checked, the text files never had lines longer than 200 characters (I chose this intentionally large). So, it might be a good heuristic to check the maximal line length,

No, some broken editor programs and people use "flowing text" files, where a newline is actually a _paragraph_ end. You have lines in the hundreds (and thousands) of characters, and the program will just flow the text for you.

Ugh. Horrible, I know.

Linus

#76 lore raw

Hi,

Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Johannes Schindelin

Last time I checked, the text files never had lines longer than 200 characters (I chose this intentionally large). So, it might be a good heuristic to check the maximal line length,

No, some broken editor programs and people use "flowing text" files, where a newline is actually a _paragraph_ end.

Good point.

Ciao,
Dscho

#77 lore raw
In reply to Linus Torvalds

Hi,

Quoted 1 line · Linus Torvalds

Here's a patch that I think we can merge right now.

Why the haste all of a sudden? Your patch is easily applyable for anyone who wants to work on text/binary or arbitrary file types. No need to rush a developers-only patch into git.git.

Ciao,
Dscho

#78 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Here's a patch [...]

No. HERE's the trivial stupid patch that just marks the core places.

Linus
---

diff.c +1 -0
index aaab309..13b9b6c 100644
@@ -1364,6 +1364,7 @@ int diff_populate_filespec(struct diff_filespec *s, int size_only)
 		s->data = xmmap(NULL, s->size, PROT_READ, MAP_PRIVATE, fd, 0);
 		close(fd);
 		s->should_munmap = 1;
+		/* FIXME! CRLF -> LF conversion goes here, based on "s->path" */
 	}
 	else {
 		char type[20];
entry.c +1 -0
index 0ebf0f0..c2641dd 100644
@@ -89,6 +89,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct checkout *stat
 			return error("git-checkout-index: unable to create file %s (%s)",
 				path, strerror(errno));
 		}
+		/* FIXME: LF -> CRLF conversion goes here, based on "ce->name" */
 		wrote = write_in_full(fd, new, size);
 		close(fd);
 		free(new);
sha1_file.c +1 -0
index 0d4bf80..8ad7fad 100644
@@ -2091,6 +2091,7 @@ int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, con
 
 	if (!type)
 		type = blob_type;
+	/* FIXME: CRLF -> LF conversion here for blobs! We'll need the path! */
 	if (write_object)
 		ret = write_sha1_file(buf, size, type, sha1);
 	else
#79 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

The *ONLY* case where we may not be able to do things automatically is actually a much more subtle one: "git cat-file". If we just get a SHA1, we don't know what the path to look it up was like, and thus we can never know whether it's a binary or a text object. With "-p" we can trivially guess, of course, but "git cat-file blob" simply must not do that!

git-cat-file, and its counter part git-hash-object, are fairly low level plumbing. Anyone using them should be aware of the issue and apply the needed conversion. And actually, since we're going to have the conversion routines in the core, we'd only need to add a --crlf argument to both of them to optionally perform the conversion since the user of those commands is more likely to know if the conversion is needed.

Nicolas

#80 lore raw
In reply to Mark Levedahl
Quoted 2 lines · Mark Levedahl

I am NOT intending to start a flamewar O:-) , so please don't turn this into one.

Heh, a lofty goal. And I am glad to see that a thread full of constructive suggestion is already going on.

So now I do not have to fear starting a flamewar; I can safely vent.

Quoted 14 lines

The recent threads on a mingw git port are explicit in the intent to provide a Windows native git. I believe there is a fundamental conflict here with the position, clearly stated by Linus, that git does not alter content in any way. Windows suffers the curse of DOS line endings (\r\n vs \n), and a true port to Windows *must* allow for \r\n and \n to be semantically the same thing as most large projects end up with a mixture of such files and/or are targeting cross-platform capabilities. The major competing solutions git seeks to supplant (cvs, cvsnt, svn, hg) have capability to recognize "text" files and transparently replace \r\n with \n on input, the reverse on output, and ignore all such differences on diff operations. To be relevant on native Windows, git must do the same. Otherwise, git will be deemed "too wierd" and dismissed in favor of a tool "that works."

Quoted 7 lines

There is no use to debating the technical merits of \r\n vs \n vs \r vs whatever, nor of not converting. Really. Just accept that there is a fundamental requirement that any version control tool on Windows be able to silently convert between \r\n and \n. To believe otherwise is to expect that the conversion be pushed elsewhere into the tool chain in use, and that won't happen as the competition already provide this conversion capability.

I think there is a fundamental misconception in the above. I do not know about others, but to me personally, I do not see any "seeking to supplant", nor "competition". It's not like I or people who raised git into the current shape are begging to windows users to consider using git and bending backwards to please them. You should hone your diplomacy.

Current git may or may not match what they need, and if it does not match what they need, making it match what they need is primarily the responsibility of them. If Windows users find something in git that is interesting and useful, but if they find something else lacking in it to be truly useful for them, they can submit patches, or if they cannot implement the changes themselves but only have wishlist items, then _they_ can do the begging.

People in git community are certainly friendly and helpful bunch, and some (including me) are unfortunate enough that sometimes they have to touch Windows, so some degree of need is felt to support Windows better even within the community, but it has never been high priority. Making it higher priority by bringing in better ideas and starting the fire must come from people who care more about Windows than me and Linus.

Quoted 5 lines

So, I think the git project needs to come to an explicit position on this, basically being:

1) git is a POSIX only tool (i.e., there will be no \r\n munging), or
2) a Windows port of git will handle and mung \r\n and \n line endings.

I do not think git project needs to do any such thing. The project evolves reflecting the needs of its users, and the design is not decided upfront without doing any feasibility study. I would certainly not say our position is (1), IOW, I would not say we will rule out Windows support. If it can be reasonably done without harming the code, why not?

Depending on how cleanly a change Windows users want is done without negatively affecting the existing users, it may or may not be judged acceptable. We will know only when we see at least the design and preferably the code. I feel no need to decide between (1) and (2) upfront before that happens.

Quoted 5 lines

If the answer is 1, the mingw port is a waste of time as it simply won't be usable by its target audience. If the answer is 2, then I think a very careful design of this capability is in order.

Comments?

This is not just you, and fortunately it does not happen very often in git community, but I find it _very_ irritating when somebody says: "here is a patch, I'll do the doc, test, and tidying up if this patch is accepted". I usually pretend to be a nice person and accept the patch when it is obviously good, or pretend that I was too busy and did not notice such a message, but I feel _very_ tempted to say: "if you care deeply enough that what you did is useful, I expect you'd perfect it whether or not I apply your patch to my tree right now. If even the original author, you, do not find it worth perfecting, then I am not interested at all."

Even if all existing git community members felt (1) above and were unwilling to accept line-end conversions (which by now you already know is not the case -- and that is why I waited until now to address this as a separate "attitude" issue), if somebody who works on Windows is motivated enough to make git work better for him, he can fork (and forking is very easy with git). If the forked git works well both on Windows and on non Windows, people who initially felt (1) will realize that they were wrong and then the codebase can be merged back together (and merging the forked projects is very easy with git).

It's open source. People shouldn't worry too much about what they have done "wasted". You are not even talking about what you've already done -- you are talking about what you _might_ do.

And your saying "If 2, then we need to think carefully" was VERY good. My point is that you did not have to say "Is it 1, or is it 2, and if 2 then" part.

#81 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 2 lines · Mark Levedahl

I am NOT intending to start a flamewar O:-) , so please don't turn this into one.

Heh, a lofty goal. And I am glad to see that a thread full of constructive suggestion is already going on.

So now I do not have to fear starting a flamewar; I can safely vent.

Junio,

I meant absolutely no offense in anything I wrote, and sincerely apologize if any was taken. My past experiences caused me to be skeptical that a significant change to accommodate a very bad design of Windows would be accepted here. Happily, that skepticism was misplaced. I am much heartened by the responses, and am optimistic a good solution will be found that is acceptable to all. It is very clear that the group is open and supportive of working through the issues to help this, and I intend to contribute to that solution. (If nothing else, I would like to be known for something besides some modest ability to hack around Tk bugs).

So, I trust the flamethrowers can remain buried.

Mark

#82 lore raw
Quoted 2 lines · Mark Levedahl

I meant absolutely no offense in anything I wrote, and sincerely apologize if any was taken.

None taken, although I admit that I was somewhat annoyed, having to write the first part of my response.