Windows: Skip fstat/lstat optimization in write_entry()
PATCH 2/2 Windows: Skip fstat/lstat optimization in write_entry()
From: Johannes Sixt <[email protected]>
Commit e4c72923 (write_entry(): use fstat() instead of lstat() when file is open, 2009-02-09) introduced an optimization of write_entry(). Unfortunately, we cannot take advantage of this optimization on Windows because there is no guarantee that the time stamps are updated before the file is closed:
"The only guarantee about a file timestamp is that the file time is
correctly reflected when the handle that makes the change is closed."
(http://msdn.microsoft.com/en-us/library/ms724290(VS.85).aspx)
The failure of this optimization on Windows can be observed most easily by running a 'git checkout' that has to update several large files. In this case, 'git checkout' will report modified files, but infact only the timestamps were incorrectly recorded in the index, as can be verified by a subsequent 'git diff', which shows no change.
---
My gut feeling was right: We cannot have this optimization on Windows.
http://thread.gmane.org/gmane.comp.version-control.git/108351/focus=108357
I've a repository where I can reproduce the error quite easily and this fixes it.
-- Hannes (who forgot to add Dscho and git@vger on the first send attempt)
Makefile | 8 ++++++++ entry.c | 3 ++- git-compat-util.h | 6 ++++++ 3 files changed, 16 insertions(+), 1 deletions(-)
Makefile +8 -0
@@ -167,6 +167,10 @@ all:: # Define NO_EXTERNAL_GREP if you don't want "git grep" to ever call # your external grep (e.g., if your system lacks grep, if its grep is # broken, or spawning external process is slower than built-in grep git has). +# +# Define UNRELIABLE_FSTAT if your system's fstat does not return the same +# information on a not yet closed file that lstat would return for the same +# file after it was closed. GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE @$(SHELL_PATH) ./GIT-VERSION-GEN
@@ -833,6 +837,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease NO_NSEC = YesPlease USE_WIN32_MMAP = YesPlease + UNRELIABLE_FSTAT = UnfortunatelyYes COMPAT_CFLAGS += -D__USE_MINGW_ACCESS -DNOGDI -Icompat -Icompat/regex -Icompat/fnmatch COMPAT_CFLAGS += -DSNPRINTF_SIZE_CORR=1 COMPAT_CFLAGS += -DSTRIP_EXTENSION=\".exe\"
@@ -1111,6 +1116,9 @@ endif ifdef NO_EXTERNAL_GREP BASIC_CFLAGS += -DNO_EXTERNAL_GREP endif +ifdef UNRELIABLE_FSTAT + BASIC_CFLAGS += -DUNRELIABLE_FSTAT +endif ifeq ($(TCLTK_PATH),) NO_TCLTK=NoThanks
entry.c +2 -1
@@ -147,7 +147,8 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout wrote =write_in_full (fd, new, size);/* use fstat() only when path == ce->name */ -if (state->refresh_cache && !to_tempfile && !state->base_dir_len) { +if (fstat_is_reliable () && + state->refresh_cache && !to_tempfile && !state->base_dir_len) {fstat (fd, &st); fstat_done =1 ; }
git-compat-util.h +6 -0
@@ -409,4 +409,10 @@ void git_qsort(void *base, size_t nmemb, size_t size,#endif #endif +#ifdef UNRELIABLE_FSTAT +#define fstat_is_reliable () 0 +#else +#define fstat_is_reliable () 1 +#endif +#endif
Signature · 1 line
1.6.3.rc1.989.ga175e.dirty
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.
-- >8 --
From 1f957680d9b0e0bfeda9bf0e20397b0323b45334 Mon Sep 17 00:00:00 2001
From: Dmitry Potapov <[email protected]>
Date: Mon, 20 Apr 2009 14:54:16 +0400
Subject: [PATCH] cygwin: Skip fstat/lstat optimization in write_entry()
--- Makefile | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
Makefile +1 -0
@@ -809,6 +809,7 @@ ifeq ($(uname_S),HP-UX) endif ifneq (,$(findstring CYGWIN,$(uname_S))) COMPAT_OBJS += compat/cygwin.o + UNRELIABLE_FSTAT = UnfortunatelyYes endif ifneq (,$(findstring MINGW,$(uname_S))) NO_PREAD = YesPlease
Signature · 2 lines
1.6.1.20.gee856 -- >8 --
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 7 lines
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like
i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.
I just sent a patch with an "optimized" fstat. I see no problems (at least none like these) with that patch. Timestamps match. Windows XP, yes. But since that MSDN article mentions that it is not guaranteed, I guess I just been lucky.
Quoted 9 lines · Alex Riesen
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 7 lines
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like
i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.
Quoted 3 lines
I just sent a patch with an "optimized" fstat. I see no problems (at least none like these) with that patch. Timestamps match. Windows XP, yes. But since that MSDN article mentions that it is not guaranteed, I guess I just been lucky.
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
-- >8 --
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define FILENAME "stat-test.tmp"
int main()
{
struct stat st1, st2;
memset(&st1, 0, sizeof(st1));
memset(&st2, 0, sizeof(st2));
unlink(FILENAME);
int fd = open(FILENAME, O_CREAT|O_RDWR|O_TRUNC, S_IRWXU);
if (fd == -1)
{
perror("Cannot open " FILENAME);
return -1;
}
sleep(1); /* It is IMPORTANT! */
write(fd, "test\n", 5);
fstat(fd, &st1);
close(fd);
lstat(FILENAME, &st2);
if (memcmp(&st1, &st2, sizeof(st1))==0)
printf("fstat is OK\n");
else
printf("fstat is broken\n");
return 0;
}-- >8 --
Dmitry
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 14 lines · Alex Riesen
Quoted 9 lines
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 7 lines
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like
i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.Quoted 3 lines
I just sent a patch with an "optimized" fstat. I see no problems (at least none like these) with that patch. Timestamps match. Windows XP, yes. But since that MSDN article mentions that it is not guaranteed, I guess I just been lucky.
Quoted 5 lines
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
And the Windows being as slow as it is, the problem can stay undetected for a long time in a real working code.
Quoted 9 lines · Alex Riesen
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 5 lines
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
And the Windows being as slow as it is, the problem can stay undetected for a long time in a real working code.
You got that wrong: If Windows were slow, the error would have been triggered more often and it would have been detected earlier. There you have the proof: Windows is fast ... enough :-P
-- Hannes
2009/4/20 Johannes Sixt <[email protected]>:
Quoted 8 lines
Quoted 7 lines · Alex Riesen
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 5 lines
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
Quoted 6 lines
Quoted 2 lines
And the Windows being as slow as it is, the problem can stay undetected for a long time in a real working code.
You got that wrong: If Windows were slow, the error would have been triggered more often and it would have been detected earlier. There you have the proof: Windows is fast ... enough :-P
Err, yes. Still a piece of cpp junk
2009/4/20 Alex Riesen <[email protected]>:
Quoted 7 lines
2009/4/20 Johannes Sixt <[email protected]>:
Quoted 3 lines
You got that wrong: If Windows were slow, the error would have been triggered more often and it would have been detected earlier. There you have the proof: Windows is fast ... enough :-P
Err, yes. Still a piece of cpp junk
Err, a _pile_.
Quoted 14 lines · Dmitry Potapov
Quoted 9 lines · Alex Riesen
2009/4/20 Dmitry Potapov <[email protected]>:
Quoted 7 lines
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like
i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.Quoted 3 lines
I just sent a patch with an "optimized" fstat. I see no problems (at least none like these) with that patch. Timestamps match. Windows XP, yes. But since that MSDN article mentions that it is not guaranteed, I guess I just been lucky.
Quoted 5 lines
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
I take that you mean that Alex's patch does not work as intended. In the meantime, I've squashed your one-liner "Cygwin-too" into Hannes's patch.
Thanks.
2009/4/20 Junio C Hamano <[email protected]>:
Quoted 9 lines
Quoted 6 lines · Dmitry Potapov
If the time passed between the creating file and end of writing to it is small (less than timestamp resolution), you may not notice the problem. The following program demonstrates the problem with fstat on Windows. (I compiled it using Cygwin). If you remove 'sleep' then you may not notice the problem for a long time.
I take that you mean that Alex's patch does not work as intended. ...
Yes, it just makes the problem harder to notice by providing a faster fstat.
Quoted 7 lines · Dmitry Potapov
The cygwin version has the same problem. (In fact, it is even worse, because we have an optimized version for lstat/stat but not for fstat, and they return different values for some fields like
i_no). But even if we used the only Cygwin functions, we would still face the problem, because Windows returns the wrong values for timestamps (and maybe even size on FAT?). So I think the following patch should be squashed on top.
I can verify that this fixes the rebase problem I've been having in Cygwin and that I just mailed about separately.
Signature · 1 line
Hannu
Hi,
Quoted 23 lines · Johannes Sixt
From: Johannes Sixt <[email protected]>
Commit
e4c72923(write_entry(): usefstat()instead oflstat()when file is open, 2009-02-09) introduced an optimization ofwrite_entry(). Unfortunately, we cannot take advantage of this optimization on Windows because there is no guarantee that the time stamps are updated before the file is closed:"The only guarantee about a file timestamp is that the file time is
correctly reflected when the handle that makes the change is closed."(http://msdn.microsoft.com/en-us/library/ms724290(VS.85).aspx)
The failure of this optimization on Windows can be observed most easily by running a 'git checkout' that has to update several large files. In this case, 'git checkout' will report modified files, but infact only the timestamps were incorrectly recorded in the index, as can be verified by a subsequent 'git diff', which shows no change.
Signed-off-by Johannes Sixt <[email protected]>---
My gut feeling was right: We cannot have this optimization on Windows.
Quoted 6 lines
http://thread.gmane.org/gmane.comp.version-control.git/108351/focus=108357
I've a repository where I can reproduce the error quite easily and this fixes it.
-- Hannes (who forgot to add Dscho and git@vger on the first send attempt)
You want this in 4msysgit's 'devel' branch, correct?
Ciao,
Dscho "who is still interim maintainer ;-)"