Skip to contentSkip to conversation

First cut at git port to Cygwin

PATCH In review H. Peter Anvin 14 participants 62 messages Jun 15, 2016 1 message not in the archive Kbuild

lore
Share & tools
lore
#0 lore raw

I have made a first cut at a git port to Cygwin. It looks like the "git-diff-tree -p" problem has been resolved independently, or at least I can't reproduce it on a fresh Cygwin install (running on XP Home), but I have added support for running without the IPv6 and the getaddrinfo() API.

There are still funnies. In particular, Cygwin and Samba handle symlinks differently, so you can't trivially share a repository via Samba. Linus' "symbolic refs" changes should eventually take care of that.

Another funny which I haven't been able to figure out yet is that 'gitk' scrunches all its output up into a few pixels at the top of the window.
If I maximize the window, I can manually resize most of the panes and
the output looks correct, but the highlighted text in the top panes show up in black on a really really dark blue background and is thus illegible.

I have set up a git-on-Cygwin temporary tree at:

http://www.kernel.org/pub/scm/git/git-cygwin.git

-hpa

#1 lore raw
Quoted 4 lines · H. Peter Anvin

Another funny which I haven't been able to figure out yet is that 'gitk' scrunches all its output up into a few pixels at the top of the window.
If I maximize the window, I can manually resize most of the panes and
the output looks correct

This is visible on OSX too, and someone's mentioned it's a Tk oddity with rootless X. Does it do the same if you run X with a root window?

Quoted 3 lines

I have set up a git-on-Cygwin temporary tree at:

http://www.kernel.org/pub/scm/git/git-cygwin.git

Getting a 404 on that. Doesn't show up on gitweb either. I guess I have to wait...

Is there a way to get gitweb to "compare" branches using git-cherry? I often look at branches via git-web and it's impossible to tell what makes them unique...

cheers,

martin

#3 lore raw
In reply to Martin Langhoff
Quoted 3 lines · Martin Langhoff

Is there a way to get gitweb to "compare" branches using git-cherry? I often look at branches via git-web and it's impossible to tell what makes them unique...

Now that you mention it, I felt that too. Maybe git-show-branch output could help somehow?

#4 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

I have set up a git-on-Cygwin temporary tree at:

http://www.kernel.org/pub/scm/git/git-cygwin.git

: siamese; git clone http://kernel.org/pub/scm/git/git-cygwin.git/ git-cygwin defaulting to local storage area
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?

Could you do update-server-info there, please?

hera$ cd /pub/scm/git/git-cygwin.git
hera$ GIT_DIR=. git-update-server-info

Knowing nothing about Cygwin environment, here are some comments.

+# Define NO_IPV6 if you lack IPv6 support and getaddrinfo().

This part probably is applicable outside Cygwin. At some point, can we have it in the mainline please?

         # The ones that do not have to link with lcrypto nor lz.
         SIMPLE_PROGRAMS = \
        -	git-get-tar-commit-id git-mailinfo git-mailsplit git-stripspace \
        -	git-daemon git-var
        +	git-get-tar-commit-id$(X) git-mailinfo$(X) git-mailsplit$(X) \
        +	git-stripspace$(X) git-var$(X) git-daemon$(X)

I have seen these $(X) in other programs' ports and found them quite distasteful. Since I do not have immediate suggestions for improvements, I do not have rights to complain, though.

Spelling it $X is a bit less distracting but not that much better. Maybe "SIMPLE_PROGRAM_NAMES = git-foo git-bar" and "SIMPLE_PROGRAMS = $(patsubst %,%$X,$(SIMPLE_PROGRAM_NAMES))"... but that would not help bits like this:

        -	PROGRAMS += git-http-fetch
        +	PROGRAMS += git-http-fetch$(X)

or this:

        -git-%: %.o $(LIB_FILE)
        +git-%$(X): %.o $(LIB_FILE)

... so I'd shut up about this part.

        diff --git a/daemon.c b/daemon.c
        --- a/daemon.c
        +++ b/daemon.c
        @@ -1,9 +1,11 @@
         #include "cache.h"
         #include "pkt-line.h"
        +#include <alloca.h>

Why? I do not see any use of alloca in the added code...

        +#include <sys/poll.h>

Is poll preferrable over select in general? Some may have only select available and others may have only poll available, perhaps? In any case, this is probably relevant to wider audience than just Cygwin; please give it to mainline at some point, perhaps conditionally allowing either/both.

        +	*socklist_p = malloc(sizeof(int));
        +	pfd = calloc(socknum, sizeof(struct pollfd));

Please use xmalloc and xcalloc just for consistency.

                test -x $path/git-$cmd && exec $path/git-$cmd "$@" ;;
        +
        +	# In case we're running on Cygwin...
        +	test -x $path/git-$cmd.exe && exec $path/git-$cmd.exe "$@" ;;
         esac

Hmph, I think you forgot to drop double semicolon there.

The git.sh script is munged by Makefile so presumably we could fix this part up there, like:

        git: git.sh Makefile
                rm -f $@+ $@
                sed -e '1s|#!.*/sh|#!$(SHELL_PATH)|' \
                    -e 's/@@GIT_VERSION@@/$(GIT_VERSION)/g' \
                    -e 's/@@X@@/$X/g' <[email protected] >$@+
                chmod +x $@+
                mv $@+ $@

And then (a patch on top of your "master"):

git.sh +7 -3
@@ -12,10 +12,14 @@ case "$#" in
 		exit 0 ;;
 	esac
 
-	test -x $path/git-$cmd && exec $path/git-$cmd "$@" ;;
+	test -x $path/git-$cmd && exec $path/git-$cmd "$@"
 
-	# In case we're running on Cygwin...
-	test -x $path/git-$cmd.exe && exec $path/git-$cmd.exe "$@" ;;
+	case '@@X@@' in
+	'')
+		;;
+	*)
+		test -x $path/git-$cmd@@X@@ && exec $path/git-$cmd@@X@@ "$@" ;;
+	esac		
 esac
 
 echo "Usage: git COMMAND [OPTIONS] [TARGET]"
#5 lore raw
Quoted 3 lines · Junio C Hamano

Could you do update-server-info there, please?

Done...

Quoted 9 lines

Knowing nothing about Cygwin environment, here are some comments.

+# Define NO_IPV6 if you lack IPv6 support and getaddrinfo().

This part probably is applicable outside Cygwin. At some point, can we have it in the mainline please?

Well, I would hope that all the changes could eventually be merged.

Quoted 19 lines
         # The ones that do not have to link with lcrypto nor lz.
         SIMPLE_PROGRAMS = \
        -	git-get-tar-commit-id git-mailinfo git-mailsplit git-stripspace \
        -	git-daemon git-var
        +	git-get-tar-commit-id$(X) git-mailinfo$(X) git-mailsplit$(X) \
        +	git-stripspace$(X) git-var$(X) git-daemon$(X)

I have seen these $(X) in other programs' ports and found them quite distasteful. Since I not have immediate suggestions for improvements, I do not have rights to complain, though.

Spelling it $X is a bit less distracting but not that much better. Maybe "SIMPLE_PROGRAM_NAMES = git-foo git-bar" and "SIMPLE_PROGRAMS = $(patsubst %,%$X,$(SIMPLE_PROGRAM_NAMES))"... but that would not help bits like this:

        -	PROGRAMS += git-http-fetch
        +	PROGRAMS += git-http-fetch$(X)
Quoted 6 lines

or this:

        -git-%: %.o $(LIB_FILE)
        +git-%$(X): %.o $(LIB_FILE)

... so I'd shut up about this part.

My first cut had PROGRAMS_X and SIMPLE_PROGRAMS_X being patsubst of the original versions, but in the end I decided it was even uglier, because these patterns were needed elsewhere. I'll change them to $X except where the parens are needed.

Quoted 9 lines
        diff --git a/daemon.c b/daemon.c
        --- a/daemon.c
        +++ b/daemon.c
        @@ -1,9 +1,11 @@
         #include "cache.h"
         #include "pkt-line.h"
        +#include <alloca.h>

Why? I do not see any use of alloca in the added code...

I originally used alloca() before changing my mind and using calloc(); I think there might be platforms without alloca out there.

Quoted 7 lines
        +#include <sys/poll.h>

Is poll preferrable over select in general? Some may have only select available and others may have only poll available, perhaps? In any case, this is probably relevant to wider audience than just Cygwin; please give it to mainline at some point, perhaps conditionally allowing either/both.

The main reason I switched to poll() is that I believe all platforms that are even remotely relevant have both these days, and forming a poll list is so much cleaner than forming a select set. What makes forming a select set even remotely bearable is the invalid assumption that the number of file descriptors is bounded at compile time and therefore that fdset_t can be statically allocated. We've had problems in the past with that assumption on Linux, and I've tried to avoid select since then.

Quoted 4 lines
        +	*socklist_p = malloc(sizeof(int));
        +	pfd = calloc(socknum, sizeof(struct pollfd));

Please use xmalloc and xcalloc just for consistency.

Check.

Quoted 7 lines
                test -x $path/git-$cmd && exec $path/git-$cmd "$@" ;;
        +
        +	# In case we're running on Cygwin...
        +	test -x $path/git-$cmd.exe && exec $path/git-$cmd.exe "$@" ;;
         esac

Hmph, I think you forgot to drop double semicolon there.

D'oh!

Quoted 24 lines

The git.sh script is munged by Makefile so presumably we could fix this part up there, like:

        git: git.sh Makefile
                rm -f $@+ $@
                sed -e '1s|#!.*/sh|#!$(SHELL_PATH)|' \
                    -e 's/@@GIT_VERSION@@/$(GIT_VERSION)/g' \
                    -e 's/@@X@@/$X/g' <[email protected] >$@+
                chmod +x $@+
                mv $@+ $@

And then (a patch on top of your "master"):

git.sh +1 -2
@@ -12,10 +12,14 @@ case "$#" in
 		exit 0 ;;
 	esac
 
-	test -x $path/git-$cmd && exec $path/git-$cmd "$@" ;;
+	test -x $path/git-$cmd && exec $path/git-$cmd "$@"
 
-	# In case we're running on Cygwin...

git.sh case "$#" in · line 25

Quoted 10 lines
git.sh +6 -1
-	test -x $path/git-$cmd.exe && exec $path/git-$cmd.exe "$@" ;;
+	case '@@X@@' in
+	'')
+		;;
+	*)
+		test -x $path/git-$cmd@@X@@ && exec $path/git-$cmd@@X@@ "$@" ;;
+	esac		
 esac
 
 echo "Usage: git COMMAND [OPTIONS] [TARGET]"

That wouldn't work, because the shell scripts don't get the .exe extension. However, I can figure out something equivalent.

-hpa

#6 lore raw

Okay, I have updated the git-cygwin repository with the changes proposed, and they seem to work.

-hpa

#7 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

There are still funnies. In particular, Cygwin and Samba handle symlinks differently, so you can't trivially share a repository via Samba. Linus' "symbolic refs" changes should eventually take care of that.

I just sent out "The other side of Linus' symbolic refs" patch, saying that Cygwin capable of doing symlink would probably made it irrelevant. But it may not be a waste after all, considering what you said above.

#8 lore raw
Quoted 11 lines · Junio C Hamano
Quoted 1 line · H. Peter Anvin

There are still funnies. In particular, Cygwin and Samba handle symlinks differently, so you can't trivially share a repository via Samba. Linus' "symbolic refs" changes should eventually take care of that.

I just sent out "The other side of Linus' symbolic refs" patch, saying that Cygwin capable of doing symlink would probably made it irrelevant. But it may not be a waste after all, considering what you said above.

After looking at it some more, what Samba does when talking to a host that doesn't support Unix extensions is that it simply resolves the symlink, in effect turning it into a hard link. That might be all git needs. The reverse still doesn't work, though.

-hpa

#9 lore raw
In reply to H. Peter Anvin

Hi,

Quoted 2 lines · H. Peter Anvin

Another funny which I haven't been able to figure out yet is that 'gitk' scrunches all its output up into a few pixels at the top of the window.

See my mail about rootless X11. I went about working around that particular Tk bug by specifying the dimensions of the panes explicitely. However, I was not especially happy with my workaround, since it did not reproduce the layout exactly after a restart. Maybe you can figure it out how to do that.

Ciao,
Dscho

#10 lore raw
Quoted 9 lines · Johannes Schindelin

Hi,

Quoted 2 lines · H. Peter Anvin

Another funny which I haven't been able to figure out yet is that 'gitk' scrunches all its output up into a few pixels at the top of the window.

See my mail about rootless X11. I went about working around that particular Tk bug by specifying the dimensions of the panes explicitely. However, I was not especially happy with my workaround, since it did not reproduce the layout exactly after a restart. Maybe you can figure it out how to do that.

Unlikely, since I'm a complete Tcl/Tk illiterate.

-hpa

#11 lore raw
In reply to Johannes Schindelin
Quoted 9 lines · Johannes Schindelin

Hi,

Quoted 2 lines · H. Peter Anvin

Another funny which I haven't been able to figure out yet is that 'gitk' scrunches all its output up into a few pixels at the top of the window.

See my mail about rootless X11. I went about working around that particular Tk bug by specifying the dimensions of the panes explicitely. However, I was not especially happy with my workaround, since it did not reproduce the layout exactly after a restart. Maybe you can figure it out how to do that.

It looks like this isn't a rootless *X* thing; it looks like the wish that is included with Cygwin actually opens native Win32 windows; even when run from inside a rooted X session it still opens an external window. I also tried using the wish from the latest ActiveState distribution; it exhibits the same problem although with slightly different geometries.

-hpa

#12 lore raw
In reply to H. Peter Anvin

I have a few things I experienced with the merged cygwin stuff. Sorry I haven't investigated it further, but there should be enough for a few fixes.

When I ...

user@machine /usr/local/dev/git/git

  $ make prefix=/usr/local install
  install -d -m755 /usr/local/bin
  install git-apply.exe [...]
  sh ./cmd-rename.sh /usr/local/bin
  ln: creating symbolic link `/usr/local/bin/git-http-pull.exe' to `git-http-fetch.exe': File exists
  make: *** [install] Error 1

Can be fixed by the patch below. I don't know if it would be cleaner to pass cmd-rename.sh "$X" as a second argument from the Makefile.

cmd-rename.sh-orig renamed from cmd-rename.sh +1 -1
@@ -3,7 +3,7 @@
 test -d "$d" || exit
 while read old new
 do
-	rm -f "$d/$old" "$d/$old.exe" 
+	rm -f "$d/$old"
 	if test -f "$d/$new"
 	then
 		ln -s "$new" "$d/$old" || exit

Some other obscurities ...

user@machine /usr/local/dev/git/git

  $ git-log
  fatal: Not a git repository

user@machine /usr/local/dev/git/git

  $ GIT_DIR=.git git-log | wc -l
  26094

and I cannot rebuild the index file with git-reset. First time I run it it creates the index.lock file and errors out when writing. The second time it errors out because the lock file was not removed in the first case.

user@machine /usr/local/dev/git/git

  $ GIT_DIR=.git git-reset
  fatal: unable to write new index file

user@machine /usr/local/dev/git/git

  $ GIT_DIR=.git git-reset
  fatal: unable to create new cachefile

user@machine /usr/local/dev/git/git

  $ uname -a
  CYGWIN_NT-5.1 antimatter 1.5.18(0.132/4/2) 2005-07-02 20:30 i686 unknown unknown Cygwin
Signature · 1 line
Jonas Fonseca
#13 lore raw

Hi,

Quoted 7 lines · Jonas Fonseca

user@machine /usr/local/dev/git/git

  $ git-log
  fatal: Not a git repository

user@machine /usr/local/dev/git/git

  $ GIT_DIR=.git git-log | wc -l
  26094

That could have its cause in your .git/HEAD being no symlink. That happens when rsync´ing the .git directory.

The other errors could also stem from the fact that quite a few places expect HEAD to be a symlink.

Ciao,
Dscho

#14 lore raw

PATCH 1/1 Fix symbolic ref validation

Johannes Schindelin <[email protected]> wrote Wed, Oct 05, 2005:

Quoted 1 line

Hi,

Hello,

Quoted 6 lines · Jonas Fonseca
Quoted 5 lines

user@machine /usr/local/dev/git/git

  $ git-log
  fatal: Not a git repository
Quoted 6 lines
Quoted 3 lines

user@machine /usr/local/dev/git/git

  $ GIT_DIR=.git git-log | wc -l
  26094

That could have its cause in your .git/HEAD being no symlink. That happens when rsync´ing the .git directory.

Yes, used rsync when I cloned. Seems validate_symref() was buggy.

Quoted 2 lines

The other errors could also stem from the fact that quite a few places expect HEAD to be a symlink.

git-reset still error out ...

---

Use the correct buffer when validating 'ref: refs/...'

---

refs.c +1 -1
@@ -46,7 +46,7 @@ int validate_symref(const char *path)
 	len -= 4;
 	while (len && isspace(*buf))
 		buf++, len--;
-	if (len >= 5 && !memcmp("refs/", buffer, 5))
+	if (len >= 5 && !memcmp("refs/", buf, 5))
 		return 0;
 	return -1;
 }
Signature · 1 line
Jonas Fonseca
#15 lore raw

Re: [PATCH] Fix symbolic ref validation

refs.c

Quoted 15 lines · Jonas Fonseca

Yes, used rsync when I cloned. Seems validate_symref() was buggy.

Quoted 2 lines

The other errors could also stem from the fact that quite a few places expect HEAD to be a symlink.

git-reset still error out ...

---

Use the correct buffer when validating 'ref: refs/...'

---

refs.c +0 -0

Thanks.

One request, not just to Jonas. Please do not use '^---$' to separate the introductory discussion and the real commit log message.

Linus style (recently the kernel list had a thread on this as well) is to have the commit log upfront with signoff, three-dash line, optional discussion and diffstat, and then diff.

I do not mind seeing discussion upfront personally [*1*], but the thing is the tool treats everything after the first '^---$' something to be fed to patch, and does not treat it as the commit log message.

[Footnote]

*1* ...but remember, Linus does.

#16 lore raw
In reply to H. Peter Anvin
Quoted 8 lines · H. Peter Anvin

I have made a first cut at a git port to Cygwin. It looks like the "git-diff-tree -p" problem has been resolved independently, or at least I can't reproduce it on a fresh Cygwin install (running on XP Home), but I have added support for running without the IPv6 and the getaddrinfo() API.

There are still funnies. In particular, Cygwin and Samba handle symlinks differently, so you can't trivially share a repository via Samba. Linus' "symbolic refs" changes should eventually take care of that.

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

#17 lore raw
Quoted 9 lines · Alex Riesen

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

In this case a better way would be to just add -liberty to all link lines if necessary, but I would expect the core cygwin code to do this.

-hpa

#18 lore raw
Quoted 12 lines · H. Peter Anvin
Quoted 11 lines · Alex Riesen

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the
target and returns an error (probably EPERM, but I have reasons not to trust
strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2)
returns EEXIST).

Quoted 4 lines
Quoted 1 line

PS: Does broken rename(2) qualify a system "not worthy to support"?

In this case a better way would be to just add -liberty to all link lines if necessary, but I would expect the core cygwin code to do this.

AFAIK, cygwin has a working rename(). Many packages rely on it.

If rename() is not working then a bug report with a test case would be appreciated.
--
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#19 lore raw
In reply to Alex Riesen
Quoted 9 lines · Alex Riesen

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

I just tried this with Cygwin 1.5.18-1 and didn't have any such problems. I tried it on NTFS, FAT and Samba, using WinXP.

-hpa

#20 lore raw
Quoted 12 lines · H. Peter Anvin
Quoted 11 lines · Alex Riesen

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the
target and returns an error (probably EPERM, but I have reasons not to trust
strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2)
returns EEXIST).

Quoted 4 lines
Quoted 1 line

PS: Does broken rename(2) qualify a system "not worthy to support"?

I just tried this with Cygwin 1.5.18-1 and didn't have any such problems. I tried it on NTFS, FAT and Samba, using WinXP.

That's a relief. Btw, AFAIK, strerror is working correctly under Cygwin also.
--
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#21 lore raw
Quoted 2 lines · Christopher Faylor

That's a relief. Btw, AFAIK, strerror is working correctly under Cygwin also.

Now if we can only figure out why gitk is messed up...

-hpa

#22 lore raw
In reply to H. Peter Anvin
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

I just tried this with Cygwin 1.5.18-1 and didn't have any such problems. I tried it on NTFS, FAT and Samba, using WinXP.

It's on Win2k, there was multiple cygwin installations in path, the other one supposedly is 1.5.5 (it's from QNX Momentics installation).
I had that old "cygwin1.dll" renamed into "cygwin1.dll-disabled" long ago, though...
I can't reproduce this out of GIT context, and the error is not reproducable after
I removed the other cygwin installation out of PATH.
Anyway, sorry, I should have tried this before posting.

#23 lore raw
Quoted 13 lines · Alex Riesen
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

I just tried this with Cygwin 1.5.18-1 and didn't have any such problems. I tried it on NTFS, FAT and Samba, using WinXP.

Quoted 8 lines

It's on Win2k, there was multiple cygwin installations in path, the other one supposedly is 1.5.5 (it's from QNX Momentics installation).
I had that old "cygwin1.dll" renamed into "cygwin1.dll-disabled" long ago, though...
I can't reproduce this out of GIT context, and the error is not reproducable after
I removed the other cygwin installation out of PATH.
Anyway, sorry, I should have tried this before posting.

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

For everyone interested, I attach cygwin's strace output here.

#24 lore raw
Quoted 23 lines · Alex Riesen
Quoted 13 lines · Alex Riesen
Quoted 11 lines · H. Peter Anvin
Quoted 8 lines

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

PS: Does broken rename(2) qualify a system "not worthy to support"?

I just tried this with Cygwin 1.5.18-1 and didn't have any such problems. I tried it on NTFS, FAT and Samba, using WinXP.

Quoted 8 lines

It's on Win2k, there was multiple cygwin installations in path, the other one supposedly is 1.5.5 (it's from QNX Momentics installation).
I had that old "cygwin1.dll" renamed into "cygwin1.dll-disabled" long ago, though...
I can't reproduce this out of GIT context, and the error is not reproducable after
I removed the other cygwin installation out of PATH.
Anyway, sorry, I should have tried this before posting.

Quoted 5 lines

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

For everyone interested, I attach cygwin's strace output here.

I'm sorry that I missed this thread. I'm usually pretty alert to the word "cygwin" showing up in a subject.

I'll go back and read the archives to catch up but, at the risk of making an observation that has already been made, under windows you can't always rename a file that is open. Is that what's happening here?

--
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#25 lore raw

Christopher Faylor, Wed, Oct 05, 2005 17:54:57 +0200:

Quoted 4 lines
Quoted 4 lines

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

Quoted 9 lines
Quoted 1 line

For everyone interested, I attach cygwin's strace output here.

I'm sorry that I missed this thread. I'm usually pretty alert to the word "cygwin" showing up in a subject.

I'll go back and read the archives to catch up but, at the risk of making an observation that has already been made, under windows you can't always rename a file that is open. Is that what's happening here?

Don't think so, but will check in about 10 hrs. The code in question is in index.c, commit_index_file.

#26 lore raw
Quoted 10 lines · Alex Riesen

Christopher Faylor, Wed, Oct 05, 2005 17:54:57 +0200:

Quoted 9 lines
Quoted 5 lines

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

For everyone interested, I attach cygwin's strace output here.

I'm sorry that I missed this thread. I'm usually pretty alert to the word "cygwin" showing up in a subject.

Quoted 7 lines
Quoted 4 lines

I'll go back and read the archives to catch up but, at the risk of making an observation that has already been made, under windows you can't always rename a file that is open. Is that what's happening here?

Don't think so, but will check in about 10 hrs. The code in question is in index.c, commit_index_file.

Ok. Looks pretty simple. FWIW, I've just built git on windows and I don't see this behavior. For the most part, it "just works".

I do see the gitk behavior but I'm tk illiterate too, unfortunately, so I can't help there.

Cygwin's tcl/tk is a funny beast. It is primarily included (by me) to allow for the operation of the insight debugger and is more windows than POSIX. As was noted, it doesn't interact with Cygwin/X but just draws windows using the Windows API.

So, tcl/tk are just barely supported in Cygwin currently. I'm actually a little surprised that it works as well as it does with gitk. --
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#27 lore raw
Quoted 5 lines · Christopher Faylor
Quoted 2 lines

Don't think so, but will check in about 10 hrs. The code in question is in index.c, commit_index_file.

Ok. Looks pretty simple. FWIW, I've just built git on windows and I don't see this behavior. For the most part, it "just works".

Thanks for the hint. There are open files involved (both index.lock and index). I attach the patch which closes index.lock (this is not really needed, btw:
rename works even without closing index.lock) and unmaps the index (a bit too intrusive). The patch fixes only update-index.c (the one I had problems with), there probably are other places were the situation is alike.

I don't like the patch (and win32 at all; hence the offending comment), so use it only unless there is no other possibility to workaround. I specifically do not request its inclusion into official branch (even though Junio is cc'ed).

Workaround Win32 problems with rename(2) applied to open/mapped files (it fails with EPERM). Thanks to Christopher Faylor for the hint regarding open files.

cache.h +1 -0
@@ -166,6 +166,7 @@ extern int ce_modified(struct cache_entr
 extern int ce_path_match(const struct cache_entry *ce, const char **pathspec);
 extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, const char *type);
 extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
+extern void unmap_active_cache();
 
 struct cache_file {
 	struct cache_file *next;
read-cache.c +13 -0
@@ -454,6 +454,17 @@ static int verify_hdr(struct cache_heade
 	return 0;
 }
 
+static void* active_cache_map = MAP_FAILED;
+static size_t active_cache_map_size = 0;
+
+void unmap_active_cache()
+{
+	if ( active_cache_map != MAP_FAILED ) {
+		active_nr = 0;
+		munmap(active_cache_map, active_cache_map_size);
+	}
+}
+
 int read_cache(void)
 {
 	int fd, i;
@@ -490,6 +501,8 @@ int read_cache(void)
 	if (verify_hdr(hdr, size) < 0)
 		goto unmap;
 
+	active_cache_map = map;
+	active_cache_map_size = size;
 	active_nr = ntohl(hdr->hdr_entries);
 	active_alloc = alloc_nr(active_nr);
 	active_cache = calloc(active_alloc, sizeof(struct cache_entry *));
update-index.c +6 -2
@@ -392,9 +392,13 @@ int main(int argc, const char **argv)
 		}
 	}
 	if (active_cache_changed) {
-		if (write_cache(newfd, active_cache, active_nr) ||
-		    commit_index_file(&cache_file))
+		if (write_cache(newfd, active_cache, active_nr))
 			die("Unable to write new cachefile");
+		/* close and unmap are just to workaround win32 stupidity */
+		close(newfd);
+		unmap_active_cache();
+		if (commit_index_file(&cache_file))
+			die("Unable to commit new cachefile");
 	}
 
 	return has_errors ? 1 : 0;
#28 lore raw
Quoted 2 lines · Alex Riesen

(a bit too intrusive). The patch fixes only update-index.c (the one I had problems with), there probably are other places were the situation is alike.

of course there are "other places". Please, try the attached patch instead.

For the record: the patch is supposed to help people with "Unable to write new cachefile" kind of errors.

Workaround Win32 problems with rename(2) applied to open/mapped files (it fails with EPERM). Thanks to Christopher Faylor for the hint regarding open files. The more complete patch.

apply.c +5 -2
@@ -1453,9 +1453,12 @@ static int apply_patch(int fd)
 		write_out_results(list, skipped_patch);
 
 	if (write_index) {
-		if (write_cache(newfd, active_cache, active_nr) ||
-		    commit_index_file(&cache_file))
+		if (write_cache(newfd, active_cache, active_nr))
 			die("Unable to write new cachefile");
+		close(newfd);
+		unmap_active_cache();
+		if (commit_index_file(&cache_file))
+			die("Unable to commit new cachefile");
 	}
 
 	if (show_files)
cache.h +1 -0
@@ -166,6 +166,7 @@ extern int ce_modified(struct cache_entr
 extern int ce_path_match(const struct cache_entry *ce, const char **pathspec);
 extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, const char *type);
 extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
+extern void unmap_active_cache();
 
 struct cache_file {
 	struct cache_file *next;
checkout-index.c +8 -4
@@ -143,9 +143,13 @@ int main(int argc, char **argv)
 		checkout_file(arg);
 	}
 
-	if (0 <= newfd &&
-	    (write_cache(newfd, active_cache, active_nr) ||
-	     commit_index_file(&cache_file)))
-		die("Unable to write new cachefile");
+	if (0 <= newfd) {
+		if (write_cache(newfd, active_cache, active_nr))
+			die("Unable to write new cachefile");
+		close(newfd);
+		unmap_active_cache();
+		if (commit_index_file(&cache_file))
+			die("Unable to commit new cachefile");
+        }
 	return 0;
 }
read-cache.c +13 -0
@@ -454,6 +454,17 @@ static int verify_hdr(struct cache_heade
 	return 0;
 }
 
+static void* active_cache_map = MAP_FAILED;
+static size_t active_cache_map_size = 0;
+
+void unmap_active_cache()
+{
+	if ( active_cache_map != MAP_FAILED ) {
+		active_nr = 0;
+		munmap(active_cache_map, active_cache_map_size);
+	}
+}
+
 int read_cache(void)
 {
 	int fd, i;
@@ -490,6 +501,8 @@ int read_cache(void)
 	if (verify_hdr(hdr, size) < 0)
 		goto unmap;
 
+	active_cache_map = map;
+	active_cache_map_size = size;
 	active_nr = ntohl(hdr->hdr_entries);
 	active_alloc = alloc_nr(active_nr);
 	active_cache = calloc(active_alloc, sizeof(struct cache_entry *));
read-tree.c +5 -2
@@ -690,8 +690,11 @@ int main(int argc, char **argv)
 	}
 
 	unpack_trees(fn);
-	if (write_cache(newfd, active_cache, active_nr) ||
-	    commit_index_file(&cache_file))
+	if (write_cache(newfd, active_cache, active_nr))
 		die("unable to write new index file");
+	close(newfd);
+	unmap_active_cache();
+	if (commit_index_file(&cache_file))
+		die("unable to commit new index file");
 	return 0;
 }
update-index.c +6 -2
@@ -392,9 +392,13 @@ int main(int argc, const char **argv)
 		}
 	}
 	if (active_cache_changed) {
-		if (write_cache(newfd, active_cache, active_nr) ||
-		    commit_index_file(&cache_file))
+		if (write_cache(newfd, active_cache, active_nr))
 			die("Unable to write new cachefile");
+		/* close and unmap are just to workaround win32 stupidity */
+		close(newfd);
+		unmap_active_cache();
+		if (commit_index_file(&cache_file))
+			die("Unable to commit new cachefile");
 	}
 
 	return has_errors ? 1 : 0;
#29 lore raw
Quoted 7 lines · Alex Riesen
Quoted 2 lines

(a bit too intrusive). The patch fixes only update-index.c (the one I had problems with), there probably are other places were the situation is alike.

of course there are "other places". Please, try the attached patch instead.

For the record: the patch is supposed to help people with "Unable to write new cachefile" kind of errors.

Just as I thought the situation improved (by closing index.lock and unmapping index),
it suddenly get worse: now I'm stuck on git-pull.

git-merge-index (called at some point by git-pull) maps the index in, and starts git-merge-one-file for each (or the given) entry in the index. git-merge-one-file
calls git-update-index, which wants to update the index. Which doesn't work, because it's locked by that piece of s$%^.

The only working walkaround for me atm is unlinking indexfile before rename in commit_index_file :(

#30 lore raw
Quoted 8 lines · Alex Riesen

it suddenly get worse: now I'm stuck on git-pull.

git-merge-index (called at some point by git-pull) maps the index in, and starts git-merge-one-file for each (or the given) entry in the index. git-merge-one-file
calls git-update-index, which wants to update the index. Which doesn't work, because it's locked by that piece of s$%^.

NOTE! git doesn't use mmap() because it _needs_ to use mmap(), but because it was simple to do that way, and it's a total idiosyncracy of mine that I often try to mmap the data. I often also tend to do my own allocators instead of using malloc() (see my "sparse" project in case you're interested in other idiosyncracies of mine - macros to do list traversal etc).

The fact is, "mmap()" isn't really any better than "read()": it has some advantages wrt memory management for the kernel, which is probably one big reason why I do it, but quite frankly, if you were to change every single mmap() to be a "map_file()" instead, and made it optional whether it used mmap() or "malloc + read()", I personally don't think it would be horrible.

And it might make things much simpler for portability. The "use mmap" approach is very much a unixism, particularly the way unix people do it (mmap followed by close, making the file descriptor "go away"). Sure, other OS's have mmap too, but I think on them it tends to be less commonly used.

Linus

#31 lore raw

Linus Torvalds, Fri, Oct 07, 2005 17:34:19 +0200:

Quoted 22 lines
Quoted 7 lines

it suddenly get worse: now I'm stuck on git-pull.

git-merge-index (called at some point by git-pull) maps the index in, and starts git-merge-one-file for each (or the given) entry in the index. git-merge-one-file calls git-update-index, which wants to update the index. Which doesn't work, because it's locked by that piece of s$%^.

NOTE! git doesn't use mmap() because it _needs_ to use mmap(), but because it was simple to do that way, and it's a total idiosyncracy of mine that I often try to mmap the data. I often also tend to do my own allocators instead of using malloc() (see my "sparse" project in case you're interested in other idiosyncracies of mine - macros to do list traversal etc).

The fact is, "mmap()" isn't really any better than "read()": it has some advantages wrt memory management for the kernel, which is probably one big reason why I do it, but quite frankly, if you were to change every single mmap() to be a "map_file()" instead, and made it optional whether it used mmap() or "malloc + read()", I personally don't think it would be horrible.

Quoted 5 lines

And it might make things much simpler for portability. The "use mmap" approach is very much a unixism, particularly the way unix people do it (mmap followed by close, making the file descriptor "go away"). Sure, other OS's have mmap too, but I think on them it tends to be less commonly used.

"Sounds like a thinly veiled threat or a very effective prodding" 8)

---

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

read-cache.c +4 -2
@@ -497,9 +497,11 @@ int read_cache(void)
 	offset = sizeof(*hdr);
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = map + offset;
-		offset = offset + ce_size(ce);
-		active_cache[i] = ce;
+		size_t size = ce_size(ce);
+		offset = offset + size;
+		active_cache[i] = malloc(ce, size);
 	}
+	munmap(map, size);
 	return active_nr;
 
 unmap:
#32 lore raw

Alex Riesen, Fri, Oct 07, 2005 22:54:50 +0200:

Quoted 23 lines

Linus Torvalds, Fri, Oct 07, 2005 17:34:19 +0200:

Quoted 22 lines
Quoted 7 lines

it suddenly get worse: now I'm stuck on git-pull.

git-merge-index (called at some point by git-pull) maps the index in, and starts git-merge-one-file for each (or the given) entry in the index. git-merge-one-file calls git-update-index, which wants to update the index. Which doesn't work, because it's locked by that piece of s$%^.

NOTE! git doesn't use mmap() because it _needs_ to use mmap(), but because it was simple to do that way, and it's a total idiosyncracy of mine that I often try to mmap the data. I often also tend to do my own allocators instead of using malloc() (see my "sparse" project in case you're interested in other idiosyncracies of mine - macros to do list traversal etc).

The fact is, "mmap()" isn't really any better than "read()": it has some advantages wrt memory management for the kernel, which is probably one big reason why I do it, but quite frankly, if you were to change every single mmap() to be a "map_file()" instead, and made it optional whether it used mmap() or "malloc + read()", I personally don't think it would be horrible.

Quoted 8 lines
Quoted 5 lines

And it might make things much simpler for portability. The "use mmap" approach is very much a unixism, particularly the way unix people do it (mmap followed by close, making the file descriptor "go away"). Sure, other OS's have mmap too, but I think on them it tends to be less commonly used.

"Sounds like a thinly veiled threat or a very effective prodding" 8)

Junio C Hamano, Fri, Oct 07, 2005 23:00:02 +0200:

Quoted 1 line

Huh? where is your memcpy?

Unbelievable... I actually tested the change! But not _the_ patch. Thanks. Next time, hit me :)

---

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

read-cache.c +5 -2
@@ -497,9 +497,12 @@ int read_cache(void)
 	offset = sizeof(*hdr);
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = map + offset;
-		offset = offset + ce_size(ce);
-		active_cache[i] = ce;
+		size_t size = ce_size(ce);
+		struct cache_entry *newce = malloc(size);
+		offset = offset + size;
+		active_cache[i] = memcpy(newce, ce, size);
 	}
+	munmap(map, size);
 	return active_nr;
 
 unmap:
#33 lore raw
Quoted 13 lines · Alex Riesen

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

read-cache.c +0 -1
@@ -497,9 +497,12 @@ int read_cache(void)
 	offset = sizeof(*hdr);
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = map + offset;
-		offset = offset + ce_size(ce);

read-cache.c int read_cache(void) · line 508

Quoted 10 lines
read-cache.c +5 -1
-		active_cache[i] = ce;
+		size_t size = ce_size(ce);
+		struct cache_entry *newce = malloc(size);
+		offset = offset + size;
+		active_cache[i] = memcpy(newce, ce, size);
 	}
+	munmap(map, size);
 	return active_nr;
 
 unmap:

s/malloc/xmalloc/

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA email;internet:[email protected]
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

#34 lore raw

Chuck Lever, Fri, Oct 07, 2005 23:29:16 +0200:

Quoted 1 line

s/malloc/xmalloc/

It's not that funny after second repost...

---

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

read-cache.c +5 -2
@@ -497,9 +497,12 @@ int read_cache(void)
 	offset = sizeof(*hdr);
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = map + offset;
-		offset = offset + ce_size(ce);
-		active_cache[i] = ce;
+		size_t size = ce_size(ce);
+		struct cache_entry *newce = xmalloc(size);
+		offset = offset + size;
+		active_cache[i] = memcpy(newce, ce, size);
 	}
+	munmap(map, size);
 	return active_nr;
 
 unmap:
#35 lore raw
Quoted 3 lines · Alex Riesen

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

I really think that you should just get rid of the mmap.

As it is, you're just slowing the code down on sane architectures. That's not good.

So I'd suggest something like this instead.

Totally untested, of course.

Linus

----

read-cache.c +37 -11
@@ -454,13 +454,39 @@ static int verify_hdr(struct cache_heade
 	return 0;
 }
 
+static void *map_index_file(int fd, size_t size)
+{
+	void *map;
+#ifdef NO_MMAP
+	map = malloc(size);
+	if (!map)
+		die("Unable to allocate index file mapping");
+	if (read(fd, map, size) != size)
+		die("Unable to read %z bytes from inde
+#else
+	map = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+	if (map == MAP_FAILED)
+		die("index file mmap failed (%s)", strerror(errno));
+#endif
+	return map;
+}
+
+static void unmap_index_file(void *map, size_t size)
+{
+#ifdef NO_MMAP
+	free(map);
+#else
+	munmap(map, size);
+#endif
+}
+
 int read_cache(void)
 {
 	int fd, i;
 	struct stat st;
 	unsigned long size, offset;
-	void *map;
 	struct cache_header *hdr;
+	void *map;
 
 	errno = EBUSY;
 	if (active_cache)
@@ -475,16 +501,15 @@ int read_cache(void)
 	}
 
 	size = 0; // avoid gcc warning
-	map = MAP_FAILED;
-	if (!fstat(fd, &st)) {
-		size = st.st_size;
-		errno = EINVAL;
-		if (size >= sizeof(struct cache_header) + 20)
-			map = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
-	}
+	if (fstat(fd, &st))
+		die("unable to fstat index file");
+
+	size = st.st_size;
+	errno = EINVAL;
+	if (size < sizeof(struct cache_header) + 20)
+		goto corrupt;
+	map = map_index_file(fd, size);
 	close(fd);
-	if (map == MAP_FAILED)
-		die("index file mmap failed (%s)", strerror(errno));
 
 	hdr = map;
 	if (verify_hdr(hdr, size) < 0)
@@ -503,8 +528,9 @@ int read_cache(void)
 	return active_nr;
 
 unmap:
-	munmap(map, size);
+	unmap_index_file(map, size);
 	errno = EINVAL;
+corrupt:
 	die("index file corrupt");
 }
 
#36 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 3 lines · Alex Riesen

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

I really think that you should just get rid of the mmap.

Quoted 8 lines

As it is, you're just slowing the code down on sane architectures. That's not good.

So I'd suggest something like this instead.

Totally untested, of course.

Linus

Slightly adjusted diff below so it compiles ;) (Note: only the second die() un hunk #1 was changed.)

Best,
Elfyn

----

read-cache.c +37 -11
@@ -454,13 +454,39 @@ static int verify_hdr(struct cache_heade
 	return 0;
 }
 
+static void *map_index_file(int fd, size_t size)
+{
+	void *map;
+#ifdef NO_MMAP
+	map = malloc(size);
+	if (!map)
+		die("Unable to allocate index file mapping");
+	if (read(fd, map, size) != size)
+		die("Unable to read %z bytes from index", size);
+#else
+	map = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
+	if (map == MAP_FAILED)
+		die("index file mmap failed (%s)", strerror(errno));
+#endif
+	return map;
+}
+
+static void unmap_index_file(void *map, size_t size)
+{
+#ifdef NO_MMAP
+	free(map);
+#else
+	munmap(map, size);
+#endif
+}
+
 int read_cache(void)
 {
 	int fd, i;
 	struct stat st;
 	unsigned long size, offset;
-	void *map;
 	struct cache_header *hdr;
+	void *map;
 
 	errno = EBUSY;
 	if (active_cache)
@@ -475,16 +501,15 @@ int read_cache(void)
 	}
 
 	size = 0; // avoid gcc warning
-	map = MAP_FAILED;
-	if (!fstat(fd, &st)) {
-		size = st.st_size;
-		errno = EINVAL;
-		if (size >= sizeof(struct cache_header) + 20)
-			map = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
-	}
+	if (fstat(fd, &st))
+		die("unable to fstat index file");
+
+	size = st.st_size;
+	errno = EINVAL;
+	if (size < sizeof(struct cache_header) + 20)
+		goto corrupt;
+	map = map_index_file(fd, size);
 	close(fd);
-	if (map == MAP_FAILED)
-		die("index file mmap failed (%s)", strerror(errno));
 
 	hdr = map;
 	if (verify_hdr(hdr, size) < 0)
@@ -503,8 +528,9 @@ int read_cache(void)
 	return active_nr;
 
 unmap:
-	munmap(map, size);
+	unmap_index_file(map, size);
 	errno = EINVAL;
+corrupt:
 	die("index file corrupt");
 }
 
Signature · 8 lines
Elfyn McBratney
Gentoo Developer/Perl Team Lead
beu/irc.freenode.net                            http://dev.gentoo.org/~beu/
+------------O.o--------------------- http://dev.gentoo.org/~beu/pubkey.asc

PGP Key ID: 0x69DF17AD
PGP Key Fingerprint:
  DBD3 B756 ED58 B1B4 47B9  B3BD 8D41 E597 69DF 17AD
#37 lore raw
In reply to Linus Torvalds

Er, apologies for the dups - postfix crapped itself :/

*goes and stands in the corner donning the 'D' hat*

Signature · 8 lines
Elfyn McBratney
Gentoo Developer/Perl Team Lead
beu/irc.freenode.net                            http://dev.gentoo.org/~beu/
+------------O.o--------------------- http://dev.gentoo.org/~beu/pubkey.asc

PGP Key ID: 0x69DF17AD
PGP Key Fingerprint:
  DBD3 B756 ED58 B1B4 47B9  B3BD 8D41 E597 69DF 17AD
#38 lore raw
In reply to Linus Torvalds

Hi,

Quoted 8 lines · Linus Torvalds

I really think that you should just get rid of the mmap.

As it is, you're just slowing the code down on sane architectures. That's not good.

So I'd suggest something like this instead.

Totally untested, of course.

Am I missing something? I don't see where the changes are written back to the fd. After all, mmap() is called with PROT_WRITE...

*shameless plug* Of course, this problem does not come up with my NO_MMAP patch.

Ciao,
Dscho

#39 lore raw

Johannes Schindelin, Sat, Oct 08, 2005 20:27:06 +0200:

Quoted 11 lines
Quoted 8 lines

I really think that you should just get rid of the mmap.

As it is, you're just slowing the code down on sane architectures. That's not good.

So I'd suggest something like this instead.

Totally untested, of course.

Am I missing something? I don't see where the changes are written back to the fd. After all, mmap() is called with PROT_WRITE...

It's just becase the file is open for reading only.
Also, it is not an mmap/unmap implementation. Just reading cache in.

#40 lore raw
In reply to Johannes Schindelin
Quoted 2 lines · Johannes Schindelin

Am I missing something? I don't see where the changes are written back to the fd. After all, mmap() is called with PROT_WRITE...

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

- index file is mapped for reading, long ago it was mapped read-only but these days we do PROT_WRITE, but updates are done via opening a new file and writing afresh.

- objects are mapped for reading, but, never updated once created. Creation side is regular open - write - close.

- diff reads original by mapping, but obviously has no business writing.

- local-fetch reads original by mapping for copying.

Quoted 2 lines

*shameless plug* Of course, this problem does not come up with my NO_MMAP patch.

Yes. It might have been overkill that you supported writing changes back, though.
.

#41 lore raw
Quoted 5 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Then PROT_READ probably makes more sense?

Quoted 3 lines

Yes. It might have been overkill that you supported writing changes back, though.

Not just overkill; if we do MAP_PRIVATE it's actively WRONG.

-hpa

#42 lore raw

Hi,

Quoted 8 lines · H. Peter Anvin
Quoted 5 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Then PROT_READ probably makes more sense?

Not necessarily. Sometimes you need to annotate the data from the index, and this does not need to be written back to the index file.

Quoted 4 lines
Quoted 2 lines

Yes. It might have been overkill that you supported writing changes back, though.

Not just overkill; if we do MAP_PRIVATE it's actively WRONG.

See above.

BTW, is there a mechanism to make sure that the index file is locked between reading and writing?

Ciao,
Dscho

#43 lore raw
Quoted 12 lines · Johannes Schindelin
Quoted 9 lines
Quoted 4 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Then PROT_READ probably makes more sense?

Not necessarily. Sometimes you need to annotate the data from the index, and this does not need to be written back to the index file.

In the above sentence, emphasis on "at least we do not intend to." If writes are done legitimately then that's fine, but we shouldn't have "accidental writes" -- those would be program bugs!

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Yes. It might have been overkill that you supported writing changes back, though.

Not just overkill; if we do MAP_PRIVATE it's actively WRONG.

See above.

Eh? If we MAP_PRIVATE, *and* we (intentionally) write to it, we *BETTER* not write anything back.

-hpa

#44 lore raw
Quoted 10 lines · H. Peter Anvin
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

In the above sentence, emphasis on "at least we do not intend to." If writes are done legitimately then that's fine, but we shouldn't have "accidental writes" -- those would be program bugs!

What I meant to say was "we do not intend to write back the changes by expecting the modification on mapped area are written back by mmap() mechanism -- the updates to index file is done by creat - write - close - rename". So your saying "the overkill being actively wrong" was technically correct, but that wrongly written data was renamed out anyway and no real harm was done.

#45 lore raw
Quoted 14 lines · Junio C Hamano
Quoted 9 lines · H. Peter Anvin
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

In the above sentence, emphasis on "at least we do not intend to." If writes are done legitimately then that's fine, but we shouldn't have "accidental writes" -- those would be program bugs!

Quoted 6 lines

What I meant to say was "we do not intend to write back the changes by expecting the modification on mapped area are written back by mmap() mechanism -- the updates to index file is done by creat - write - close - rename". So your saying "the overkill being actively wrong" was technically correct, but that wrongly written data was renamed out anyway and no real harm was done.

Well, it broke the atomicity of an operation, which *is* a real problem.

Anyway, malloc+read is a dead ringer for MAP_PRIVATE with PROT_WRITE, so that makes it even easier to mimic.

-hpa

#46 lore raw
In reply to H. Peter Anvin
Quoted 2 lines · H. Peter Anvin

Eh? If we MAP_PRIVATE, *and* we (intentionally) write to it, we *BETTER* not write anything back.

Correct.

It has already been fixed by Johannes last week and I merged it over the weekend if not earlier if I recall correctly.

#47 lore raw
In reply to H. Peter Anvin

Hi,

Quoted 11 lines · H. Peter Anvin
Quoted 10 lines · Johannes Schindelin
Quoted 9 lines
Quoted 4 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Then PROT_READ probably makes more sense?

Quoted 7 lines
Quoted 3 lines

Not necessarily. Sometimes you need to annotate the data from the index, and this does not need to be written back to the index file.

In the above sentence, emphasis on "at least we do not intend to." If writes are done legitimately then that's fine, but we shouldn't have "accidental writes" -- those would be program bugs!

Yes, those would be bugs. However, if I understood the man page for mmap() correctly, then PROT_WRITE && MAP_PRIVATE makes the data copy-on-write, which means that those bugs would have been found (because the changes would no longer be present when git was called the next time). And I
checked: all mmap() calls in git are MAP_PRIVATE.

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Yes. It might have been overkill that you supported writing changes back, though.

Not just overkill; if we do MAP_PRIVATE it's actively WRONG.

See above.

Eh? If we MAP_PRIVATE, *and* we (intentionally) write to it, we *BETTER* not write anything back.

Yes. That was *my* mistake.

Ciao,
Dscho

#48 lore raw
In reply to Johannes Schindelin
Quoted 11 lines · Johannes Schindelin

Hi,

Quoted 7 lines · H. Peter Anvin
Quoted 5 lines · Junio C Hamano

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Quoted 4 lines
Quoted 1 line

Then PROT_READ probably makes more sense?

Not necessarily. Sometimes you need to annotate the data from the index, and this does not need to be written back to the index file.

In fact, it is intentional that we open the file O_RDONLY, and mmap it PROT_READ | PROT_WRITE, MAP_PRIVATE. We prepare the next index in the memory where we mapped the old index, but we don't want to change what's on the disk using the mapping; we write that later to a different file using write().

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Yes. It might have been overkill that you supported writing changes back, though.

Not just overkill; if we do MAP_PRIVATE it's actively WRONG.

See above.

BTW, is there a mechanism to make sure that the index file is locked between reading and writing?

There's definitely locking; the new file is written to "(filename).lock", which is openned O_CREAT | O_EXCL, and is moved to the destination when it's complete. I believe everything that intends to write a new index gets the lock before reading the old index, although I haven't actually checked.

-Daniel
*This .sig left intentionally blank*

#49 lore raw
In reply to Junio C Hamano

Hi,

Quoted 8 lines · Junio C Hamano
Quoted 2 lines · Johannes Schindelin

Am I missing something? I don't see where the changes are written back to the fd. After all, mmap() is called with PROT_WRITE...

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Ahh! Reading the man page helps!

Quoted 2 lines

Yes. It might have been overkill that you supported writing changes back, though.

Sure. Something like this?

compat/mmap.c +1 -1
index fca6321..fda39fc 100644
@@ -49,7 +49,7 @@ void *gitfakemmap(void *start, size_t le
 		n += count;
 	}
 
-	if(prot & PROT_WRITE) {
+	if((prot & PROT_WRITE) && !(flags & MAP_PRIVATE)) {
 		fakemmapwritable *next = xmalloc(sizeof(fakemmapwritable));
 		next->start = start;
 		next->length = length;
#50 lore raw
Quoted 10 lines · Johannes Schindelin
Quoted 3 lines

PROT_WRITE is true, but we do MAP_PRIVATE, and if I recall correctly we do not write file via mmap -- at least we do not intend to.

Ahh! Reading the man page helps!

Quoted 2 lines

Yes. It might have been overkill that you supported writing changes back, though.

Sure. Something like this?

Not, really. What I meant was to rip out the writing out altogether, and perhaps making sure that the caller never calls us without MAP_PRIVATE.

#51 lore raw

Hi,

Quoted 7 lines · Junio C Hamano
Quoted 1 line · Johannes Schindelin

Sure. Something like this?

Not, really. What I meant was to rip out the writing out altogether, and perhaps making sure that the caller never calls us without MAP_PRIVATE.

How about this, then?

[PATCH] If NO_MMAP is defined, fake mmap() and munmap()

Since some platforms do not support mmap() at all, and others do only just so, this patch introduces the option to fake mmap() and munmap() by malloc()ing the region explicitely.

---

 Makefile      |    6 ++++++
 cache.h       |   16 ++++++++++++++++
 compat/mmap.c |   51 +++++++++++++++++++++++++++++++++++++++++++++++++++
 mailsplit.c   |    1 -
 4 files changed, 73 insertions(+), 1 deletions(-)
 create mode 100644 compat/mmap.c

applies-to: 274542bcbc891cca353c2728ac4075df3d1d2c0d ed334e3e2276fe9d41ed78917544ef6a3fa87eb7

Makefile +6 -0
index 1bdf4de..7ca77cf 100644
@@ -27,6 +27,8 @@
 # Define NEEDS_SOCKET if linking with libc is not enough (SunOS,
 # Patrick Mauritz).
 #
+# Define NO_MMAP if you want to avoid mmap.
+#
 # Define WITH_OWN_SUBPROCESS_PY if you want to use with python 2.3.
 #
 # Define NO_IPV6 if you lack IPv6 support and getaddrinfo().
@@ -258,6 +260,10 @@ ifdef NO_STRCASESTR
 	DEFINES += -Dstrcasestr=gitstrcasestr
 	LIB_OBJS += compat/strcasestr.o
 endif
+ifdef NO_MMAP
+	DEFINES += -Dmmap=gitfakemmap -Dmunmap=gitfakemunmap -DNO_MMAP
+	LIB_OBJS += compat/mmap.o
+endif
 ifdef NO_IPV6
 	DEFINES += -DNO_IPV6 -Dsockaddr_storage=sockaddr_in
 endif
cache.h +16 -0
index 514adb8..5987d4c 100644
@@ -11,7 +11,9 @@
 #include <string.h>
 #include <errno.h>
 #include <limits.h>
+#ifndef NO_MMAP
 #include <sys/mman.h>
+#endif
 #include <sys/param.h>
 #include <netinet/in.h>
 #include <sys/types.h>
@@ -356,4 +358,18 @@ extern void packed_object_info_detail(st
 /* Dumb servers support */
 extern int update_server_info(int);
 
+#ifdef NO_MMAP
+
+#ifndef PROT_READ
+#define PROT_READ 1
+#define PROT_WRITE 2
+#define MAP_PRIVATE 1
+#define MAP_FAILED ((void*)-1)
+#endif
+
+extern void *gitfakemmap(void *start, size_t length, int prot , int flags, int fd, off_t offset);
+extern int gitfakemunmap(void *start, size_t length);
+
+#endif
+
 #endif /* CACHE_H */
compat/mmap.c new file +51 -0

51 lines not on this page · raw message

mailsplit.c +0 -1
index 7981f87..0f8100d 100644
@@ -9,7 +9,6 @@
 #include <fcntl.h>
 #include <sys/types.h>
 #include <sys/stat.h>
-#include <sys/mman.h>
 #include <string.h>
 #include <stdio.h>
 #include <ctype.h>

---
0.99.8.GIT

#52 lore raw

Commit text BEFORE the dashes (Re: First cut at git port to Cygwin)

In reply to Alex Riesen
Quoted 3 lines · Alex Riesen

[ some text ]
---
[ the actual commit text ]

REMINDER: These need to be swapped.

Signature · 1 line
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#54 lore raw

Junio C Hamano, Fri, Oct 07, 2005 23:00:02 +0200:

Quoted 9 lines
Quoted 6 lines

"Sounds like a thinly veiled threat or a very effective prodding" 8)
---

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

Huh? where is your memcpy?

Junio, unless there already are pressing reasons to put the patch in GIT, could you postpone its inclusion (if you ever considered)? Or at least put "#ifdef __cygwin" (I hope this is the define) around it?

It just so ugly... And besides, GIT reportedly works without problems for many people even without it.

Anyway, the patch is out, so anyone with the problems can just patch their copy to workaround this specific win2k problem.

Thanks,
Alex

#55 lore raw
Quoted 8 lines · Alex Riesen

Junio C Hamano, Fri, Oct 07, 2005 23:00:02 +0200:

Quoted 7 lines
Quoted 6 lines

"Sounds like a thinly veiled threat or a very effective prodding" 8)
---

Make read_cache copy the index into memory, to improve portability on other OS's which have mmap too, tend to use it less commonly.

Quoted 6 lines
Quoted 2 lines

Huh? where is your memcpy?

Junio, unless there already are pressing reasons to put the patch in GIT, could you postpone its inclusion (if you ever considered)? Or at least put "#ifdef __cygwin" (I hope this is the define) around it?

Close ;) - the define is "__CYGWIN__".

Best,
Elfyn

Signature · 8 lines
Elfyn McBratney
Gentoo Developer/Perl Team Lead
beu/irc.freenode.net                            http://dev.gentoo.org/~beu/
+------------O.o--------------------- http://dev.gentoo.org/~beu/pubkey.asc

PGP Key ID: 0x69DF17AD
PGP Key Fingerprint:
  DBD3 B756 ED58 B1B4 47B9  B3BD 8D41 E597 69DF 17AD
#56 lore raw
Quoted 7 lines · Elfyn McBratney
Quoted 4 lines

Junio, unless there already are pressing reasons to put the patch in GIT, could you postpone its inclusion (if you ever considered)? Or at least put "#ifdef __cygwin" (I hope this is the define) around it?

Close ;) - the define is "__CYGWIN__".

This should be a feature-control macro in the Makefile.

#57 lore raw
In reply to Christopher Faylor
Quoted 3 lines · Alex Riesen

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

I don't know how Cygwin implemented rename(), but if they used MoveFile() they broke the POSIX rename() since MoveFile() fails if destination already exists. They should have used MoveFileEx(MOVEFILE_REPLACE_EXISTING) instead, to guarantee POSIX semantics. The symptoms you're experiencing make me think this might be the case (even if it is strange, since other Unix software would fail the same way).

- Davide

#58 lore raw
Quoted 5 lines · Davide Libenzi
Quoted 3 lines · Alex Riesen

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

Quoted 7 lines

I don't know how Cygwin implemented rename(), but if they used MoveFile() they broke the POSIX rename() since MoveFile() fails if destination already exists. They should have used MoveFileEx(MOVEFILE_REPLACE_EXISTING) instead, to guarantee POSIX semantics. The symptoms you're experiencing make me think this might be the case (even if it is strange, since other Unix software would fail the same way).

Cygwin's rename is much more than just a simple wrapper around MoveFile or MoveFileEx. It tries hard to guarantee POSIX semantics within the strictures imposed by Windows.
--
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#59 lore raw
Quoted 3 lines · Christopher Faylor

Cygwin's rename is much more than just a simple wrapper around MoveFile or MoveFileEx. It tries hard to guarantee POSIX semantics within the strictures imposed by Windows.

Certainly, but presumably different versions of Win32 have different limitations, no?

-hpa

#60 lore raw
Quoted 7 lines · H. Peter Anvin
Quoted 3 lines · Christopher Faylor

Cygwin's rename is much more than just a simple wrapper around MoveFile or MoveFileEx. It tries hard to guarantee POSIX semantics within the strictures imposed by Windows.

Certainly, but presumably different versions of Win32 have different limitations, no?

Yes, definitely. That's why the Cygwin rename code (and unlink code for that matter) is so headache-inducing.

FWIW, I just looked at the source and, AFAICT, if rename() is setting EPERM on Windows NT/2000/XP, it should be the result of a failing MoveFileEx (old, new, MOVEFILE_REPLACE_EXISTING).
--
Christopher Faylor spammer? -> [email protected]
Cygwin Co-Project Leader [email protected]
TimeSys, Inc.

#61 lore raw
In reply to Christopher Faylor
Quoted 14 lines · Christopher Faylor
Quoted 5 lines · Davide Libenzi
Quoted 3 lines · Alex Riesen

Still does not work for me. I cannot isolate the problem out of git, but at the moment the only way for me to make commit_index_file to work is to put unlink(indexfile) before rename(cf->lockfile, indexfile).

Quoted 7 lines

I don't know how Cygwin implemented rename(), but if they used MoveFile() they broke the POSIX rename() since MoveFile() fails if destination already exists. They should have used MoveFileEx(MOVEFILE_REPLACE_EXISTING) instead, to guarantee POSIX semantics. The symptoms you're experiencing make me think this might be the case (even if it is strange, since other Unix software would fail the same way).

Quoted 3 lines

Cygwin's rename is much more than just a simple wrapper around MoveFile or MoveFileEx. It tries hard to guarantee POSIX semantics within the strictures imposed by Windows.

Ouch, IC:

http://cygwin.com/cgi-bin/cvsweb.cgi/~checkout~/src/winsup/cygwin/syscalls.cc?rev=1.390&content-type=text/plain&cvsroot=src

That's quite some code.

- Davide

#62 lore raw
In reply to Alex Riesen
Quoted 10 lines · Alex Riesen
Quoted 8 lines · H. Peter Anvin

I have made a first cut at a git port to Cygwin. It looks like the "git-diff-tree -p" problem has been resolved independently, or at least I can't reproduce it on a fresh Cygwin install (running on XP Home), but I have added support for running without the IPv6 and the getaddrinfo() API.

There are still funnies. In particular, Cygwin and Samba handle symlinks differently, so you can't trivially share a repository via Samba. Linus' "symbolic refs" changes should eventually take care of that.

Quoted 6 lines

I noticed that rename(2) in my copy of cygwin (1.5.18-1) does not remove the target and returns an error (probably EPERM, but I have reasons not to trust strerror on that thing).
The repository was on FAT.
Taking "rename(2)" from cygwin's libiberty solved this (they unlink if link(2) returns EEXIST).

I think I have to clarify: I copied the function (like in strcasestr case) into compat/