Build Git with MSVC

=== V4 2009.09.16 ===

1) Removed the _fmode extern, and if(std*) statements, as pointed out by Alexey Borzenkov.
2) Renumbered the points in the compat/vcbuild/README (affects 2 patches), as pointed out by Thiago Farina.
3) Added flag to lib command to invoke the Link Time Code Generation to avoid warnings about restarting the libifier with this option.
4) Added a patch to the end of the series, which tags the GIT_VERSION with ".MSVC", to ease debugging/problem solving on Windows, when we start getting reports from both MSysGit and MSVCGit people.
NOTE: If people are OK with this patch, it should probably be squashed into patch 12. If not, just skip it.

The overall difference between v4 and v3 of this series is rather small but since it touches several patches due to context (thus conflicts appear), I push out a new version of the whole serie.

v3 to v4 diff:

index 761cef1..2c20922 100644
@@ -878,6 +878,7 @@ ifneq (,$(findstring CYGWIN,$(uname_S)))
 	UNRELIABLE_FSTAT = UnfortunatelyYes
 endif
 ifdef MSVC
+	GIT_VERSION := $(GIT_VERSION).MSVC
 	pathsep = ;
 	NO_PREAD = YesPlease
 	NO_OPENSSL = YesPlease

@@ -923,6 +924,7 @@ ifdef MSVC
 ifndef DEBUG
 	BASIC_CFLAGS += -GL -Os -MT
 	BASIC_LDFLAGS += -LTCG
+	AR += -LTCG
 else
 	BASIC_CFLAGS += -Zi -MTd
 endif

index 544b576..5b5258b 100644
@@ -229,20 +229,17 @@ void free_environ(char **env);
 
 /*
  * A replacement of main() that ensures that argv[0] has a path
+ * and that default fmode and std(in|out|err) are in binary mode
  */
 
-extern int _fmode;
 #define main(c,v) dummy_decl_mingw_main(); \
 static int mingw_main(); \
 int main(int argc, const char **argv) \
 { \
 	_fmode = _O_BINARY; \
-	if (stdin) \
-		_setmode(_fileno(stdin), _O_BINARY); \
-	if (stdout) \
-		_setmode(_fileno(stdout), _O_BINARY); \
-	if (stderr) \
-		_setmode(_fileno(stderr), _O_BINARY); \
+	_setmode(_fileno(stdin), _O_BINARY); \
+	_setmode(_fileno(stdout), _O_BINARY); \
+	_setmode(_fileno(stderr), _O_BINARY); \
 	argv[0] = xstrdup(_pgmptr); \
 	return mingw_main(argc, argv); \
 } \

index e64f7e5..df8a657 100644
@@ -29,11 +29,11 @@ The Steps of Build Git with VS2008
    executables, since Git might need to run scripts which are part of
    the git operations.
 
-4. Inside Git's directory run the command:
+3. Inside Git's directory run the command:
        make common-cmds.h
    to generate the common-cmds.h file needed to compile git.
 
-5. Then either build Git with the GNU Make Makefile in the Git projects
+4. Then either build Git with the GNU Make Makefile in the Git projects
    root
        make MSVC=1
    or generate Visual Studio solution/projects (.sln/.vcproj) with the

EOD

=== V3 2009.09.15 ===

1) Reworded patch 01: s/Microsoft Visual C++/MSVC/
2) Nuked patch 02
3) Merged patch 03 into path 08
4) Rewrote patch 06 to set _fmode instead for both MinGW and MSVC
5) Fix patch 07 to use __stdcall for thread functions instead of WINAPI
6) Rewrote patch 14 to rather use a define to _chsize
7) Fix patch 15 to use "-o $@" instead of "-o git.o"

Think that's it..

=== V2 2009.09.14 ===

Here's a second roll of the series for supporting compilation of Git
using Visual C++ (MSVC). (Sorry for the long delay on a followup, time
is in short supply these days)

There's no guarantee that the compiled result will work as well as the
current MinGW compile, or at all. However, I think it's important to
get the Git repo to a compilable state with MSVC as quickly as
possible, to further enable contributions from the Windows developers
which we are sorely lacking at the moment.
I hope that a repo which compiles successfully with the tools they are
accustomed to (also a very good debugger) will entice them to send more
patches.

In addition to this series, I have also setup a repo with binaries of
the required libs to compile Git with MSVC. Only 32bit versions for now.
So, the developer can choose to either use that, or Frank's source code
repo to build Git with MSVC. You'll find the binary repo here:
    http://repo.or.cz/w/msvcgit.git

Note that the binaries will still require the msysgit environment for
execution, due to the non-binary components of Git. (Scripts, gitk,
git-gui, etc). You'll find that repo here:
    http://repo.or.cz/w/msysgit.git

1) Rebased ontop on current git.git 'next'.
2) Fixed code and commit msgs based on the previous feedback.
   Let me know if I forgot anything!
3) Added a proper Makefile patch for MSVC support.
   You now compile with 'make MSVC=1'.
4) Added perl scripts (contrib) which uses GNU Make output to generate
   MSVC IDE (.sln & .vcproj) projects and QMake projects, and picking
   up the project settings from the Makefile, so it's all in one place.
5) Added necessary patches to make things compile ontop of current
   'next'.

=== V1 2009.08.21 ===
So, Frank Li started this series, and I took it upon my self to help
out a bit; cleaning, reorganizing, rebasing the series. Hopefully
we're now a bit closer to including the series into mainline..

Here's a summary of what has happend:
1) This series is rebased ontop of git.git 'next', which needed an
   extra patch to avoid a non-constant array creation, which
   mscv doesn't like.
2) I've polished (tied to anyways) the commit messages a bit.
3) I've applied much of the feedback provided to the first round of
   the patches.
4) I've split, merged and reordered some of the patches, so things
   that belong together are in the same commits, and in a order of
   'importance'
5) I've removed the
       #define func _func
   stuff, as it's not needed and Microsoft cannot really kill the
   compatibility functions anyways. So, adding the define
       _CRT_NONSTDC_NO_DEPRECATE
   will kill the warnings seen without the defines above.
6) ..probably much more as well, but I forget..

Note: I did not sign off on the last two commits, which involve the
adding of the vcproj files, since I don't agree on adding them as is.
We need a Makefile way of compiling primarily, and second, a script
to generate the vcproj, as already discussed. But the commits are
included for completeness, at to let others compile and play with it.

I've kept the original author as is, and just signed the patches..
Thanks for watching, now bring on the comments!

Frank Li (6):
  Avoid declaration after statement
  Change regerror() declaration from K&R style to ANSI C (C89)
  Fix __stdcall placement and function prototype
  Test for WIN32 instead of __MINGW32_
  Add MinGW header files to build git with MSVC
  Add platform files for MSVC porting

Marius Storm-Olsen (9):
  Add define guards to compat/win32.h
  Set _O_BINARY as default fmode for both MinGW and MSVC
  Add empty header files for MSVC port
  Make usage of windows.h lean and mean
  Define strncasecmp and ftruncate for MSVC
  Add MSVC to Makefile
  Add README for MSVC build
  Add scripts to generate projects for other buildsystems (MSVC vcproj,
    QMake)
  Tag GIT_VERSION when Git is built with MSVC

 .gitignore                                |   11 +
 Makefile                                  |   56 +++-
 compat/mingw.c                            |   20 +-
 compat/mingw.h                            |    7 +
 compat/msvc.c                             |   35 ++
 compat/msvc.h                             |   50 +++
 compat/regex/regex.c                      |    7 +-
 compat/snprintf.c                         |   10 +-
 compat/vcbuild/README                     |   50 +++
 compat/vcbuild/include/alloca.h           |    1 +
 compat/vcbuild/include/arpa/inet.h        |    1 +
 compat/vcbuild/include/dirent.h           |  128 ++++++
 compat/vcbuild/include/grp.h              |    1 +
 compat/vcbuild/include/inttypes.h         |    1 +
 compat/vcbuild/include/netdb.h            |    1 +
 compat/vcbuild/include/netinet/in.h       |    1 +
 compat/vcbuild/include/netinet/tcp.h      |    1 +
 compat/vcbuild/include/pwd.h              |    1 +
 compat/vcbuild/include/sys/ioctl.h        |    1 +
 compat/vcbuild/include/sys/param.h        |    1 +
 compat/vcbuild/include/sys/poll.h         |    1 +
 compat/vcbuild/include/sys/select.h       |    1 +
 compat/vcbuild/include/sys/socket.h       |    1 +
 compat/vcbuild/include/sys/time.h         |    1 +
 compat/vcbuild/include/sys/utime.h        |   34 ++
 compat/vcbuild/include/sys/wait.h         |    1 +
 compat/vcbuild/include/unistd.h           |   92 ++++
 compat/vcbuild/include/utime.h            |    1 +
 compat/vcbuild/scripts/clink.pl           |   48 +++
 compat/vcbuild/scripts/lib.pl             |   26 ++
 compat/win32.h                            |    7 +
 compat/winansi.c                          |    1 -
 contrib/buildsystems/Generators.pm        |   42 ++
 contrib/buildsystems/Generators/QMake.pm  |  189 +++++++++
 contrib/buildsystems/Generators/Vcproj.pm |  639 +++++++++++++++++++++++++++++
 contrib/buildsystems/engine.pl            |  353 ++++++++++++++++
 contrib/buildsystems/generate             |   29 ++
 contrib/buildsystems/parse.pl             |  228 ++++++++++
 git-compat-util.h                         |    9 +
 help.c                                    |    5 +-
 pager.c                                   |    4 +-
 run-command.c                             |   12 +-
 run-command.h                             |    2 +-
 setup.c                                   |    2 +-
 thread-utils.c                            |    5 +-
 45 files changed, 2084 insertions(+), 33 deletions(-)
 create mode 100644 compat/msvc.c
 create mode 100644 compat/msvc.h
 create mode 100644 compat/vcbuild/README
 create mode 100644 compat/vcbuild/include/alloca.h
 create mode 100644 compat/vcbuild/include/arpa/inet.h
 create mode 100644 compat/vcbuild/include/dirent.h
 create mode 100644 compat/vcbuild/include/grp.h
 create mode 100644 compat/vcbuild/include/inttypes.h
 create mode 100644 compat/vcbuild/include/netdb.h
 create mode 100644 compat/vcbuild/include/netinet/in.h
 create mode 100644 compat/vcbuild/include/netinet/tcp.h
 create mode 100644 compat/vcbuild/include/pwd.h
 create mode 100644 compat/vcbuild/include/sys/ioctl.h
 create mode 100644 compat/vcbuild/include/sys/param.h
 create mode 100644 compat/vcbuild/include/sys/poll.h
 create mode 100644 compat/vcbuild/include/sys/select.h
 create mode 100644 compat/vcbuild/include/sys/socket.h
 create mode 100644 compat/vcbuild/include/sys/time.h
 create mode 100644 compat/vcbuild/include/sys/utime.h
 create mode 100644 compat/vcbuild/include/sys/wait.h
 create mode 100644 compat/vcbuild/include/unistd.h
 create mode 100644 compat/vcbuild/include/utime.h
 create mode 100644 compat/vcbuild/scripts/clink.pl
 create mode 100644 compat/vcbuild/scripts/lib.pl
 create mode 100644 contrib/buildsystems/Generators.pm
 create mode 100644 contrib/buildsystems/Generators/QMake.pm
 create mode 100644 contrib/buildsystems/Generators/Vcproj.pm
 create mode 100644 contrib/buildsystems/engine.pl
 create mode 100644 contrib/buildsystems/generate
 create mode 100644 contrib/buildsystems/parse.pl

From: Frank Li <[email protected]>

MSVC does not understand this C99 style

---
 compat/mingw.c |   14 ++++++++++----
 help.c         |    3 ++-
 run-command.c  |    2 ++
 3 files changed, 14 insertions(+), 5 deletions(-)

index 36ef8d3..5478b74 100644
@@ -123,13 +123,17 @@ int mingw_open (const char *filename, int oflags, ...)
 {
 	va_list args;
 	unsigned mode;
+	int fd;
+
 	va_start(args, oflags);
 	mode = va_arg(args, int);
 	va_end(args);
 
 	if (!strcmp(filename, "/dev/null"))
 		filename = "nul";
-	int fd = open(filename, oflags, mode);
+
+	fd = open(filename, oflags, mode);
+
 	if (fd < 0 && (oflags & O_CREAT) && errno == EACCES) {
 		DWORD attrs = GetFileAttributes(filename);
 		if (attrs != INVALID_FILE_ATTRIBUTES && (attrs & FILE_ATTRIBUTE_DIRECTORY))

@@ -580,10 +584,11 @@ static char **get_path_split(void)
 
 static void free_path_split(char **path)
 {
+	char **p = path;
+
 	if (!path)
 		return;
 
-	char **p = path;
 	while (*p)
 		free(*p++);
 	free(path);

@@ -1120,9 +1125,9 @@ int sigaction(int sig, struct sigaction *in, struct sigaction *out)
 #undef signal
 sig_handler_t mingw_signal(int sig, sig_handler_t handler)
 {
+	sig_handler_t old = timer_fn;
 	if (sig != SIGALRM)
 		return signal(sig, handler);
-	sig_handler_t old = timer_fn;
 	timer_fn = handler;
 	return old;
 }

@@ -1209,8 +1214,9 @@ struct dirent *mingw_readdir(DIR *dir)
 
 	if (dir->dd_handle == (long)INVALID_HANDLE_VALUE && dir->dd_stat == 0)
 	{
+		DWORD lasterr;
 		handle = FindFirstFileA(dir->dd_name, &buf);
-		DWORD lasterr = GetLastError();
+		lasterr = GetLastError();
 		dir->dd_handle = (long)handle;
 		if (handle == INVALID_HANDLE_VALUE && (lasterr != ERROR_NO_MORE_FILES)) {
 			errno = err_win_to_posix(lasterr);

index 294337e..fd51b8e 100644
@@ -127,7 +127,7 @@ static int is_executable(const char *name)
 		return 0;
 
 #ifdef __MINGW32__
-	/* cannot trust the executable bit, peek into the file instead */
+{	/* cannot trust the executable bit, peek into the file instead */
 	char buf[3] = { 0 };
 	int n;
 	int fd = open(name, O_RDONLY);

@@ -140,6 +140,7 @@ static int is_executable(const char *name)
 				st.st_mode |= S_IXUSR;
 		close(fd);
 	}
+}
 #endif
 	return st.st_mode & S_IXUSR;
 }

index ac314a5..02aaedf 100644
@@ -134,6 +134,7 @@ fail_pipe:
 		error("cannot fork() for %s: %s", cmd->argv[0],
 			strerror(failed_errno = errno));
 #else
+{
 	int s0 = -1, s1 = -1, s2 = -1;	/* backups of stdin, stdout, stderr */
 	const char **sargv = cmd->argv;
 	char **env = environ;

@@ -197,6 +198,7 @@ fail_pipe:
 		dup2(s1, 1), close(s1);
 	if (s2 >= 0)
 		dup2(s2, 2), close(s2);
+}
 #endif
 
 	if (cmd->pid < 0) {

1.6.2.1.418.g33d56.dirty

---
 compat/win32.h |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

index c26384e..e8c178d 100644
@@ -1,3 +1,6 @@
+#ifndef WIN32_H
+#define WIN32_H
+
 /* common Win32 functions for MinGW and Cygwin */
 #include <windows.h>
 

@@ -32,3 +35,5 @@ static inline int get_file_attr(const char *fname, WIN32_FILE_ATTRIBUTE_DATA *fd
 		return ENOENT;
 	}
 }
+
+#endif

1.6.2.1.418.g33d56.dirty

Erik Faye-Lund said the following on 16.09.2009 11:42:

Yup, of course they are. *thump*

--
.marius

From: Frank Li <[email protected]>

The MSVC headers typedef errcode as int, and thus confused the compiler in the K&R style definition. ANSI style deconfuses it.

---
 compat/regex/regex.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)

index 5ea0075..67d5c37 100644
@@ -4852,11 +4852,8 @@ regexec (preg, string, nmatch, pmatch, eflags)
    from either regcomp or regexec.   We don't use PREG here.  */
 
 size_t
-regerror (errcode, preg, errbuf, errbuf_size)
-    int errcode;
-    const regex_t *preg;
-    char *errbuf;
-    size_t errbuf_size;
+regerror(int errcode, const regex_t *preg,
+	 char *errbuf, size_t errbuf_size)
 {
   const char *msg;
   size_t msg_size;

1.6.2.1.418.g33d56.dirty

From: Marius Storm-Olsen <[email protected]>

MinGW set the _CRT_fmode to set both the default fmode and _O_BINARY on stdin/stdout/stderr. Rather use the main() define in mingw.h to set this for both MinGW and MSVC.

This will ensure that a MinGW and MSVC build will handle input and output identically.

---
 compat/mingw.c |    2 --
 compat/mingw.h |    5 +++++
 2 files changed, 5 insertions(+), 2 deletions(-)

index 5478b74..5a8fae8 100644
@@ -3,8 +3,6 @@
 #include <conio.h>
 #include "../strbuf.h"
 
-unsigned int _CRT_fmode = _O_BINARY;
-
 static int err_win_to_posix(DWORD winerr)
 {
 	int error = ENOSYS;

index c43917c..bcd23b0 100644
@@ -227,12 +227,17 @@ void free_environ(char **env);
 
 /*
  * A replacement of main() that ensures that argv[0] has a path
+ * and that default fmode and std(in|out|err) are in binary mode
  */
 
 #define main(c,v) dummy_decl_mingw_main(); \
 static int mingw_main(); \
 int main(int argc, const char **argv) \
 { \
+	_fmode = _O_BINARY; \
+	_setmode(_fileno(stdin), _O_BINARY); \
+	_setmode(_fileno(stdout), _O_BINARY); \
+	_setmode(_fileno(stderr), _O_BINARY); \
 	argv[0] = xstrdup(_pgmptr); \
 	return mingw_main(argc, argv); \
 } \

1.6.2.1.418.g33d56.dirty

MinGW set the _CRT_fmode to set both the default fmode and _O_BINARY on stdin/stdout/stderr. Rather use the main() define in mingw.h to set this for both MinGW and MSVC.

This will ensure that a MinGW and MSVC build will handle input and output identically.

Alexey Borzenkov:
Include git-compat-util.h from the test-genrandom.c so that the application also will use the main() define, and thus the stdout mode is set correctly and generated data stays the same.

Johannes Sixt:
Before, since the test-genrandom application would link against libgit.a, the MinGW process initialization code would pick up the definition of _CRT_fmode from mingw.c that is initialized to _O_BINARY. After this patch, however, the _CRT_fmode symbol is no longer present in libgit.a, so MinGWs process initialization code will not set the correct std(in|out|err) mode. So, the include is needed to override main() and explicitly set the mode.

---
 compat/mingw.c   |    2 --
 compat/mingw.h   |    5 +++++
 test-genrandom.c |    1 +
 3 files changed, 6 insertions(+), 2 deletions(-)

index 5478b74..5a8fae8 100644
@@ -3,8 +3,6 @@
 #include <conio.h>
 #include "../strbuf.h"
 
-unsigned int _CRT_fmode = _O_BINARY;
-
 static int err_win_to_posix(DWORD winerr)
 {
 	int error = ENOSYS;

index c43917c..bcd23b0 100644
@@ -227,12 +227,17 @@ void free_environ(char **env);
 
 /*
  * A replacement of main() that ensures that argv[0] has a path
+ * and that default fmode and std(in|out|err) are in binary mode
  */
 
 #define main(c,v) dummy_decl_mingw_main(); \
 static int mingw_main(); \
 int main(int argc, const char **argv) \
 { \
+	_fmode = _O_BINARY; \
+	_setmode(_fileno(stdin), _O_BINARY); \
+	_setmode(_fileno(stdout), _O_BINARY); \
+	_setmode(_fileno(stderr), _O_BINARY); \
 	argv[0] = xstrdup(_pgmptr); \
 	return mingw_main(argc, argv); \
 } \

index 8ad276d..5849599 100644
@@ -6,6 +6,7 @@
 
 #include <stdio.h>
 #include <stdlib.h>
+#include "git-compat-util.h"
 
 int main(int argc, char *argv[])
 {

1.6.2.1.418.g33d56.dirty

MinGW set the _CRT_fmode to set both the default fmode and _O_BINARY on stdin/stdout/stderr. Rather use the main() define in mingw.h to set this for both MinGW and MSVC.

This will ensure that a MinGW and MSVC build will handle input and output identically.

Alexey Borzenkov:
Include git-compat-util.h from the test-genrandom.c so that the application also will use the main() define, and thus the stdout mode is set correctly and generated data stays the same.

Johannes Sixt:
Before, since the test-genrandom application would link against libgit.a, the MinGW process initialization code would pick up the definition of _CRT_fmode from mingw.c that is initialized to _O_BINARY. After this patch, however, the _CRT_fmode symbol is no longer present in libgit.a, so MinGWs process initialization code will not set the correct std(in|out|err) mode. So, the include is needed to override main() and explicitly set the mode.

---
Gaaaaah! Sorry for not reading Alexey's patch properly. Multitasking is not working to my advantage these days..

 compat/mingw.c   |    2 --
 compat/mingw.h   |    5 +++++
 test-genrandom.c |    3 +--
 3 files changed, 6 insertions(+), 4 deletions(-)

index 5478b74..5a8fae8 100644
@@ -3,8 +3,6 @@
 #include <conio.h>
 #include "../strbuf.h"
 
-unsigned int _CRT_fmode = _O_BINARY;
-
 static int err_win_to_posix(DWORD winerr)
 {
 	int error = ENOSYS;

index c43917c..bcd23b0 100644
@@ -227,12 +227,17 @@ void free_environ(char **env);
 
 /*
  * A replacement of main() that ensures that argv[0] has a path
+ * and that default fmode and std(in|out|err) are in binary mode
  */
 
 #define main(c,v) dummy_decl_mingw_main(); \
 static int mingw_main(); \
 int main(int argc, const char **argv) \
 { \
+	_fmode = _O_BINARY; \
+	_setmode(_fileno(stdin), _O_BINARY); \
+	_setmode(_fileno(stdout), _O_BINARY); \
+	_setmode(_fileno(stderr), _O_BINARY); \
 	argv[0] = xstrdup(_pgmptr); \
 	return mingw_main(argc, argv); \
 } \

index 8ad276d..b3c28d9 100644
@@ -4,8 +4,7 @@
  * Copyright (C) 2007 by Nicolas Pitre, licensed under the GPL version 2.
  */
 
-#include <stdio.h>
-#include <stdlib.h>
+#include "git-compat-util.h"
 
 int main(int argc, char *argv[])
 {

1.6.2.1.418.g33d56.dirty

This one breaks t5302-pack-index.sh (test 15, "[index v1] 2) create a stealth corruption in a delta base reference") in my MinGW build. I have yet to find out what exactly goes wrong and how it could be fixed.

-- Hannes

Looks like calling _setmode in main is too late for something. :-/

I tried defining int _fmode in mingw.c (which sets msvcrt's _fmode early, without calling _setmode), and it didn't work either (i.e. the bug was still happening). So the culprit must be in _setmode. I also tried logging what index_obj_offset and index_obj_nr are showing, and here are results:

/* with _CRT_fmode = _O_BINARY */
index_obj_offset: 844032
index_obj_nr: 205

/* without _CRT_fmode = _O_BINARY */
index_obj_offset: 844235
index_obj_nr: 298

I then tried to see what git show-index actually returns there, and here are several top lines of the results:

/* with _CRT_fmode = _O_BINARY */
850186 0106e17481932f5c223fafadc1d26abc6adf40d6
203652 01179d82b18c252824e20e190543e0e84950d820
15576 0246ff570c396c37ff08c0f5f9d88c84fc21a297
851536 035e7e54cee9eb197af435b6b6dcced489c233f6
376062 03a6f56dbbb556425ef8c43c31519b32eb5629a4
39452 0471b3b8ab8da19f3968dc238e9555f4e48ce858
8230 0483297992ed4cd5ab3b5790912b1f03598ea660
72292 0508b858e01f0367552fb14722388ba48dbd36f7

/* without _CRT_fmode = _O_BINARY */
844235 01a8d3f6d063d48baf40ee656804aa58486a34b2
851754 035e7e54cee9eb197af435b6b6dcced489c233f6
851844 04ff7bd225fbb4842697af96c147a4ddecb6a657
848233 088ddcb945af72c47348b65a63992c71b6b9dbe8
849676 088e69e7ab273e35b4f63ecb1b915a3997f13dff
14825 09bd72201f82a0da14a52230e2ccd0e43529e779
9848 0a5916ad398380ba103d98bad6fa4f2b20b74e75
695905 0ae5469e687083bbf954262873ee8c0c7d017e75

As you can see the hashes are completely different! Unfortunately I also can't pinpoint where this actually happens... :-/

*several minutes later*

Searching which executables set _fmode and which don't I found the culprit. test-genrandom.c didn't include git-compat-util.h, so mingw.h was never included. This caused different random data to be generated, and as it seems more importantly, of different sizes. Can be fixed with this patch:

index 8ad276d..b3c28d9 100644
@@ -4,8 +4,7 @@
  * Copyright (C) 2007 by Nicolas Pitre, licensed under the GPL version 2.
  */
 
-#include <stdio.h>
-#include <stdlib.h>
+#include "git-compat-util.h"
 
 int main(int argc, char *argv[])
 {

Or maybe there's a hidden bug in git that gets uncovered with slightly different random data, I don't know. I'm glad I could finally solve this mindbending issue. :)

Johannes Sixt said the following on 17.09.2009 09:11:

Yup, are you ok with squashing this hunk into the patch then?

--
.marius

Of course; with some extra words in the commit message.

-- Hannes

Johannes Sixt said the following on 17.09.2009 09:36:

Great. Hold tight, I'll resend an updated version.

BTW, I ran all the tests (make /k test) before and after the whole series, with msysgit 'devel' branch + plain git.git 'next', and it turns out that 2 more tests pass after this series :)

Before:
fixed 4
success 4017

failed  380
broken  9
total   4524

After:
fixed 4
success 4019

failed  378
broken  9
total   4524

Both complete logs attached in 7zip format.

--
.marius

Sorry to disappoint you: these 2 tests only passed by chance. For a work-around see e95a73ef in mingw.git (it papers over an undetected racily-clean index).

-- Hannes

Johannes Sixt said the following on 17.09.2009 10:10:

Ah, gotcha. Oh well, at least no regressions then, which is also a good thing :)

--
.marius

Maybe we should just implement nsec for mingw? The code is already almost there:

Converter filetime_to_timespec is available in cygwin.c Definition of struct timespec and struct stat that uses it can be borrowed from cygwin's cygwin/stat.h and cygwin/types.h, we could call it struct mingw_timespec and struct mingw_stat. Then #define timespec mingw_timespec and #define stat mingw_stat, as well as #define st_atime (st_atim.tv_sec) and similarly others, just like cygwin/stat.h does.
Since we already reimplement stat/lstat/fstat it should be relatively simple and we don't care about usage of mingw's stat.
Do you want me to cook up a patch?

Alexey Borzenkov said the following on 17.09.2009 10:39:

we already override the stat() implementation with our own :)

compat/mingw.h (~174):

/* Use mingw_lstat() instead of lstat()/stat() and
* mingw_fstat() instead of fstat() on Windows. */
#define off_t off64_t
#define stat _stati64
#define lseek _lseeki64
int mingw_lstat(const char *file_name, struct stat *buf); int mingw_fstat(int fd, struct stat *buf); #define fstat mingw_fstat
#define lstat mingw_lstat
#define _stati64(x,y) mingw_lstat(x,y)

int mingw_utime(const char *file_name, const struct utimbuf *times); #define utime mingw_utime

See compat/mingw.c for implementation..

--
.marius

Yes, that's why implementing nsec whould be easy. See compat/cygwin.c (filetime_to_timespec) and git-compat-util.h (#ifdef NO_NSEC). Currently mingw doesn't support tv_nsec.

First thing to do is to understand what is going on: There are other architectures that do not have nsec and that do *not* have the problem; why do we have a problem on Windows? If you cannot answer this question, an nsec solution would still just be "it happens to work", and not "it works by design".

I think I had analyzed this particular case, and I understood it, but did not make notes why I found it satisfactory to just paper over the issue, and today I don't remember anymore. :-/

Don't forget utime().

I consider an answer to the question "why do we need nsec?" more important.

BTW: t4130-apply-criss-cross-rename.sh fails quite often as well. This time it is because we fill in st_ino = 0 instead of some useful value. The test moves files around that happen to have identical sizes. After the operation, the dirty working tree could be noticed because st_ino differs from the index. But since we always set it to 0, it is not noticed. (That's my theory.)

If you make a patch, can you think about this issue as well?

-- Hannes

Johannes Sixt said the following on 17.09.2009 11:03:

Cygwin version also sets st_ino = 0, so I'm not sure this is the cause of failure; unless this test also fails there? Does anyone know?

--
.marius

This is slightly more than a repost, forgot to reply to all, then found more info.

Ah, yes, I didn't realize that nsec checking isn't even enabled by default. So implementing it wouldn't help. As for st_ino it doesn't matter in this case, as it wouldn't change anyway.

It's interesting that putting git diff or git status before calling git stash shows that the changes are actually there (and then they can be picked up). It seems that something manages to spend enough time so that cache_entry's mtime and index.timestamp start to differ. For example, here I modified check-racy to show more info for this particular case:

index e0f9a63..00f425c 100644
@@ -364,6 +364,7 @@ PROGRAMS += git-unpack-file$X
 PROGRAMS += git-update-server-info$X
 PROGRAMS += git-upload-pack$X
 PROGRAMS += git-var$X
+PROGRAMS += git-check-racy$X
 
 # List built-in command $C whose implementation cmd_$C() is not in
 # builtin-$C.o but is linked in as part of some other command.

index 00d92a1..72b2878 100644
@@ -16,6 +16,12 @@ int main(int ac, char **av)
                       continue;
               }
 
+               if (!strcmp(ce->name, "file")) {
+                       printf("for file: index.timestamp.sec: %d\n",

the_index.timestamp.sec);
+ printf("for file: ce->ce_mtime.sec: %d\n",
ce->ce_mtime.sec);
+ printf("for file: st.st_mtime: %d\n",
(int)st.st_mtime);
+ }
+

               if (ce_match_stat(ce, &st, 0))
                       dirty++;
               else if (ce_match_stat(ce, &st, CE_MATCH_RACY_IS_DIRTY))

index 03e589f..6eb73fe 100755
@@ -74,8 +74,14 @@ create_stash () {
               cp -p ${GIT_INDEX_FILE-"$GIT_DIR/index"} "$TMP-index" &&
               GIT_INDEX_FILE="$TMP-index" &&
               export GIT_INDEX_FILE &&
+               echo ">>> before git read-tree:" >&2 &&
+               git check-racy >&2 &&
               git read-tree -m $i_tree &&
+               echo ">>> before git add:" >&2 &&
+               git check-racy >&2 &&
               git add -u &&
+               echo ">>> before git write-tree" >&2 &&
+               git check-racy >&2 &&
               git write-tree &&
               rm -f "$TMP-index"
       ) ) ||

Now where I run it and it doesn't fail:

HEAD is now at dfc6a3a other-file

for file: index.timestamp.sec: 1253191568
for file:    ce->ce_mtime.sec: 1253191568
for file:         st.st_mtime: 1253191568

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253191570
for file:    ce->ce_mtime.sec: 1253191568
for file:         st.st_mtime: 1253191568

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253191571
for file:    ce->ce_mtime.sec: 1253191568
for file:         st.st_mtime: 1253191568

dirty 0, clean 2, racy 0
Saved working directory and index state WIP on master: dfc6a3a other-file HEAD is now at dfc6a3a other-file
(To restore them type "git stash apply")

for file: index.timestamp.sec: 1253191572
for file:    ce->ce_mtime.sec: 1253191571
for file:         st.st_mtime: 1253191572

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253191574
for file:    ce->ce_mtime.sec: 1253191571
for file:         st.st_mtime: 1253191572

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253191574
for file:    ce->ce_mtime.sec: 1253191572
for file:         st.st_mtime: 1253191572

dirty 0, clean 2, racy 0
Saved working directory and index state WIP on master: dfc6a3a other-file

But, when this test fails, the results are:

HEAD is now at dfc6a3a other-file

for file: index.timestamp.sec: 1253191677
for file:    ce->ce_mtime.sec: 1253191676
for file:         st.st_mtime: 1253191676

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253191679
for file:    ce->ce_mtime.sec: 1253191676
for file:         st.st_mtime: 1253191676

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253191679
for file:    ce->ce_mtime.sec: 1253191676
for file:         st.st_mtime: 1253191676

dirty 0, clean 2, racy 0
Saved working directory and index state WIP on master: dfc6a3a other-file HEAD is now at dfc6a3a other-file
(To restore them type "git stash apply")

for file: index.timestamp.sec: 1253191681
for file:    ce->ce_mtime.sec: 1253191680
for file:         st.st_mtime: 1253191680

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253191683
for file:    ce->ce_mtime.sec: 1253191680
for file:         st.st_mtime: 1253191680

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253191683
for file:    ce->ce_mtime.sec: 1253191680
for file:         st.st_mtime: 1253191680

dirty 0, clean 2, racy 0
Saved working directory and index state WIP on master: dfc6a3a other-file

As you can see, even for "8" (first git stash in that test) it failed:
it seemed as if working tree is clean, index was not updated, changes were not stashed. But it just so happens that it didn't matter.

Also, as you can see, in both cases index timestamp is slightly bigger than cache_entry's timestamp. That's why git doesn't see file as racy. Now I'm wondering WHAT manages to modify index timestamp. :-/

Update:

Ah, I just added more git check-racy calls and found one extremely interesting case:

HEAD is now at dfc6a3a other-file

for file: index.timestamp.sec: 1253192315
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253192315
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253192315
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253192315
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 1, clean 1, racy 0

for file: index.timestamp.sec: 1253192316
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253192318
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 0, clean 2, racy 0

for file: index.timestamp.sec: 1253192318
for file:    ce->ce_mtime.sec: 1253192315
for file:         st.st_mtime: 1253192315

dirty 0, clean 2, racy 0
Saved working directory and index state WIP on master: dfc6a3a other-file

"after finding i_tree and i_commit" is immediately before calculating w_tree. As you can see, "before git read-tree" is off by a second. I think it's just a bug in msys, cp -p doesn't preserve mtime exactly. :-/

[for the archives: this is about intermittent failures of t3903-stash.sh with MinGW git]

Ah, now I recall everything: git-stash.sh does "cp -p", and from the times where git-commit was still a shell script I know that this does not always preserve the timestamp correctly. Since this is a bug attributable to MSYS (and not MinGW git), I was satisfied by simply inserting 'sleep 1' in the test case to avoid the racily-clean situation. (Although today, I would modify the content to have a different size.)

-- Hannes

Beautiful. I think it is worth mentioning some of the above in the commit log message.

Thanks.

From: Frank Li <[email protected]>

MSVC requires __stdcall to be between the functions return value and the function name, and that the function pointer type is in the form of
return_type (WINAPI *function_name)(arguments...)

---
 compat/mingw.c |    4 ++--
 run-command.c  |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

index 5a8fae8..34ee427 100644
@@ -1027,7 +1027,7 @@ static sig_handler_t timer_fn = SIG_DFL;
  * length to call the signal handler.
  */
 
-static __stdcall unsigned ticktack(void *dummy)
+static unsigned __stdcall ticktack(void *dummy)
 {
 	while (WaitForSingleObject(timer_event, timer_interval) == WAIT_TIMEOUT) {
 		if (timer_fn == SIG_DFL)

@@ -1154,7 +1154,7 @@ void mingw_open_html(const char *unixpath)
 
 int link(const char *oldpath, const char *newpath)
 {
-	typedef BOOL WINAPI (*T)(const char*, const char*, LPSECURITY_ATTRIBUTES);
+	typedef BOOL (WINAPI *T)(const char*, const char*, LPSECURITY_ATTRIBUTES);
 	static T create_hard_link = NULL;
 	if (!create_hard_link) {
 		create_hard_link = (T) GetProcAddress(

index 02aaedf..bb76750 100644
@@ -316,7 +316,7 @@ int run_command_v_opt_cd_env(const char **argv, int opt, const char *dir, const
 }
 
 #ifdef __MINGW32__
-static __stdcall unsigned run_thread(void *data)
+static unsigned __stdcall run_thread(void *data)
 {
 	struct async *async = data;
 	return async->proc(async->fd_for_proc, async->data);

1.6.2.1.418.g33d56.dirty

From: Frank Li <[email protected]>

The code which is conditional on MinGW32 is actually conditional on Windows. Use the WIN32 symbol, which is defined by the MINGW32 and MSVC environments, but not by Cygwin.

Define SNPRINTF_SIZE_CORR=1 for MSVC too, as its vsnprintf function does not add NUL at the end of the buffer if the result fits the buffer size exactly.

---
 compat/snprintf.c |   10 ++++++----
 help.c            |    2 +-
 pager.c           |    4 ++--
 run-command.c     |    8 ++++----
 run-command.h     |    2 +-
 setup.c           |    2 +-
 6 files changed, 15 insertions(+), 13 deletions(-)

index 4d07087..e1e0e75 100644
@@ -2,12 +2,14 @@
 
 /*
  * The size parameter specifies the available space, i.e. includes
- * the trailing NUL byte; but Windows's vsnprintf expects the
- * number of characters to write, and does not necessarily write the
- * trailing NUL.
+ * the trailing NUL byte; but Windows's vsnprintf uses the entire
+ * buffer and avoids the trailing NUL, should the buffer be exactly
+ * big enough for the result. Defining SNPRINTF_SIZE_CORR to 1 will
+ * therefore remove 1 byte from the reported buffer size, so we
+ * always have room for a trailing NUL byte.
  */
 #ifndef SNPRINTF_SIZE_CORR
-#if defined(__MINGW32__) && defined(__GNUC__) && __GNUC__ < 4
+#if defined(WIN32) && (!defined(__GNUC__) || __GNUC__ < 4)
 #define SNPRINTF_SIZE_CORR 1
 #else
 #define SNPRINTF_SIZE_CORR 0

index fd51b8e..e8db31f 100644
@@ -126,7 +126,7 @@ static int is_executable(const char *name)
 	    !S_ISREG(st.st_mode))
 		return 0;
 
-#ifdef __MINGW32__
+#ifdef WIN32
 {	/* cannot trust the executable bit, peek into the file instead */
 	char buf[3] = { 0 };
 	int n;

index f416d38..86facec 100644
@@ -9,7 +9,7 @@
 
 static int spawned_pager;
 
-#ifndef __MINGW32__
+#ifndef WIN32
 static void pager_preexec(void)
 {
 	/*

@@ -72,7 +72,7 @@ void setup_pager(void)
 		static const char *env[] = { "LESS=FRSX", NULL };
 		pager_process.env = env;
 	}
-#ifndef __MINGW32__
+#ifndef WIN32
 	pager_process.preexec_cb = pager_preexec;
 #endif
 	if (start_command(&pager_process))

index bb76750..cf2d8f7 100644
@@ -75,7 +75,7 @@ fail_pipe:
 
 	trace_argv_printf(cmd->argv, "trace: run_command:");
 
-#ifndef __MINGW32__
+#ifndef WIN32
 	fflush(NULL);
 	cmd->pid = fork();
 	if (!cmd->pid) {

@@ -315,7 +315,7 @@ int run_command_v_opt_cd_env(const char **argv, int opt, const char *dir, const
 	return run_command(&cmd);
 }
 
-#ifdef __MINGW32__
+#ifdef WIN32
 static unsigned __stdcall run_thread(void *data)
 {
 	struct async *async = data;

@@ -331,7 +331,7 @@ int start_async(struct async *async)
 		return error("cannot create pipe: %s", strerror(errno));
 	async->out = pipe_out[0];
 
-#ifndef __MINGW32__
+#ifndef WIN32
 	/* Flush stdio before fork() to avoid cloning buffers */
 	fflush(NULL);
 

@@ -360,7 +360,7 @@ int start_async(struct async *async)
 
 int finish_async(struct async *async)
 {
-#ifndef __MINGW32__
+#ifndef WIN32
 	int ret = wait_or_whine(async->pid, "child process", 0);
 #else
 	DWORD ret = 0;

index 0c00b25..fb34209 100644
@@ -70,7 +70,7 @@ struct async {
 	int (*proc)(int fd, void *data);
 	void *data;
 	int out;	/* caller reads from here and closes it */
-#ifndef __MINGW32__
+#ifndef WIN32
 	pid_t pid;
 #else
 	HANDLE tid;

index e3781b6..029371e 100644
@@ -41,7 +41,7 @@ const char *prefix_path(const char *prefix, int len, const char *path)
 const char *prefix_filename(const char *pfx, int pfx_len, const char *arg)
 {
 	static char path[PATH_MAX];
-#ifndef __MINGW32__
+#ifndef WIN32
 	if (!pfx || !*pfx || is_absolute_path(arg))
 		return arg;
 	memcpy(path, pfx, pfx_len);

1.6.2.1.418.g33d56.dirty

MSVC lacks many of the header files included by git-compat-util.h, so add blank header files for these instead of going ifdef crazy.

---
 compat/vcbuild/include/arpa/inet.h   |    1 +
 compat/vcbuild/include/grp.h         |    1 +
 compat/vcbuild/include/inttypes.h    |    1 +
 compat/vcbuild/include/netdb.h       |    1 +
 compat/vcbuild/include/netinet/in.h  |    1 +
 compat/vcbuild/include/netinet/tcp.h |    1 +
 compat/vcbuild/include/pwd.h         |    1 +
 compat/vcbuild/include/sys/ioctl.h   |    1 +
 compat/vcbuild/include/sys/param.h   |    1 +
 compat/vcbuild/include/sys/poll.h    |    1 +
 compat/vcbuild/include/sys/select.h  |    1 +
 compat/vcbuild/include/sys/socket.h  |    1 +
 compat/vcbuild/include/sys/time.h    |    1 +
 compat/vcbuild/include/sys/wait.h    |    1 +
 14 files changed, 14 insertions(+), 0 deletions(-)
 create mode 100644 compat/vcbuild/include/arpa/inet.h
 create mode 100644 compat/vcbuild/include/grp.h
 create mode 100644 compat/vcbuild/include/inttypes.h
 create mode 100644 compat/vcbuild/include/netdb.h
 create mode 100644 compat/vcbuild/include/netinet/in.h
 create mode 100644 compat/vcbuild/include/netinet/tcp.h
 create mode 100644 compat/vcbuild/include/pwd.h
 create mode 100644 compat/vcbuild/include/sys/ioctl.h
 create mode 100644 compat/vcbuild/include/sys/param.h
 create mode 100644 compat/vcbuild/include/sys/poll.h
 create mode 100644 compat/vcbuild/include/sys/select.h
 create mode 100644 compat/vcbuild/include/sys/socket.h
 create mode 100644 compat/vcbuild/include/sys/time.h
 create mode 100644 compat/vcbuild/include/sys/wait.h

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

new file mode 100644
index 0000000..0d8552a
@@ -0,0 +1 @@
+/* Intentionally empty file to support building git with MSVC */

1.6.2.1.418.g33d56.dirty

From: Frank Li <[email protected]>

Added the header files dirent.h, unistd.h and utime.h Add alloca.h, which simply includes malloc.h, which defines alloca

---
 compat/vcbuild/include/alloca.h    |    1 +
 compat/vcbuild/include/dirent.h    |  128 ++++++++++++++++++++++++++++++++++++
 compat/vcbuild/include/sys/utime.h |   34 ++++++++++
 compat/vcbuild/include/unistd.h    |   92 ++++++++++++++++++++++++++
 compat/vcbuild/include/utime.h     |    1 +
 5 files changed, 256 insertions(+), 0 deletions(-)
 create mode 100644 compat/vcbuild/include/alloca.h
 create mode 100644 compat/vcbuild/include/dirent.h
 create mode 100644 compat/vcbuild/include/sys/utime.h
 create mode 100644 compat/vcbuild/include/unistd.h
 create mode 100644 compat/vcbuild/include/utime.h

new file mode 100644
index 0000000..c0d7985
@@ -0,0 +1 @@
+#include <malloc.h>

new file mode 100644
index 0000000..582589c
@@ -0,0 +1,34 @@
+#ifndef	_UTIME_H_
+#define	_UTIME_H_
+/*
+ * UTIME.H
+ * This file has no copyright assigned and is placed in the Public Domain.
+ * This file is a part of the mingw-runtime package.
+ *
+ * The mingw-runtime package and its code is distributed in the hope that it
+ * will be useful but WITHOUT ANY WARRANTY.  ALL WARRANTIES, EXPRESSED OR
+ * IMPLIED ARE HEREBY DISCLAIMED.  This includes but is not limited to
+ * warranties of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * You are free to use this package and its code without limitation.
+ */
+
+/*
+ * Structure used by _utime function.
+ */
+struct _utimbuf
+{
+	time_t	actime;		/* Access time */
+	time_t	modtime;	/* Modification time */
+};
+
+#ifndef	_NO_OLDNAMES
+/* NOTE: Must be the same as _utimbuf above. */
+struct utimbuf
+{
+	time_t	actime;
+	time_t	modtime;
+};
+#endif	/* Not _NO_OLDNAMES */
+
+#endif

new file mode 100644
index 0000000..8285f38
@@ -0,0 +1 @@
+#include <sys/utime.h>

1.6.2.1.418.g33d56.dirty

From: Frank Li <[email protected]>

Add msvc.c and msvc.h to build git under MSVC

---
 compat/msvc.c     |   35 +++++++++++++++++++++++++++++++++
 compat/msvc.h     |   55 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 git-compat-util.h |    3 ++
 3 files changed, 93 insertions(+), 0 deletions(-)
 create mode 100644 compat/msvc.c
 create mode 100644 compat/msvc.h

new file mode 100644
index 0000000..ac04a4c
@@ -0,0 +1,35 @@
+#include "../git-compat-util.h"
+#include "win32.h"
+#include <conio.h>
+#include "../strbuf.h"
+
+DIR *opendir(const char *name)
+{
+	int len;
+	DIR *p;
+	p = (DIR*)malloc(sizeof(DIR));
+	memset(p, 0, sizeof(DIR));
+	strncpy(p->dd_name, name, PATH_MAX);
+	len = strlen(p->dd_name);
+	p->dd_name[len] = '/';
+	p->dd_name[len+1] = '*';
+
+	if (p == NULL)
+		return NULL;
+
+	p->dd_handle = _findfirst(p->dd_name, &p->dd_dta);
+
+	if (p->dd_handle == -1) {
+		free(p);
+		return NULL;
+	}
+	return p;
+}
+int closedir(DIR *dir)
+{
+	_findclose(dir->dd_handle);
+	free(dir);
+	return 0;
+}
+
+#include "mingw.c"

index e5e9f39..8ea444f 100644
@@ -113,6 +113,9 @@
 /* pull in Windows compatibility stuff */
 #include "compat/mingw.h"
 #endif	/* __MINGW32__ */
+#ifdef _MSC_VER
+#include "compat/msvc.h"
+#endif
 
 #ifndef NO_LIBGEN_H
 #include <libgen.h>

1.6.2.1.418.g33d56.dirty

Centralize the include of windows.h in git-compat-util.h, turn on WIN32_LEAN_AND_MEAN to avoid including plenty of other header files which is not needed in Git. Also ensure we load winsock2.h first, so we don't load the older winsock definitions at a later stage, since they contain duplicate definitions.

When moving windows.h into git-compat-util.h, we need to protect the definition of struct pollfd in mingw.h, since this file is used by both MinGW and MSVC, and the latter defines this struct in winsock2.h.

We need to keep the windows.h include in compat/win32.h, since its shared by both MinGW and Cygwin, and we're not touching Cygwin in this commit. The include in git-compat-util.h is protected with an ifdef WIN32, which is not the case when compiling for Cygwin.

---
 compat/mingw.c    |    2 ++
 compat/mingw.h    |    2 ++
 compat/msvc.h     |    7 -------
 compat/win32.h    |    2 ++
 compat/winansi.c  |    1 -
 git-compat-util.h |    6 ++++++
 thread-utils.c    |    5 +----
 7 files changed, 13 insertions(+), 12 deletions(-)

index 34ee427..6b5b5b2 100644
@@ -3,6 +3,8 @@
 #include <conio.h>
 #include "../strbuf.h"
 
+#include <shellapi.h>
+
 static int err_win_to_posix(DWORD winerr)
 {
 	int error = ENOSYS;

index bcd23b0..5b5258b 100644
@@ -41,6 +41,7 @@ struct passwd {
 
 extern char *getpass(const char *prompt);
 
+#ifndef POLLIN
 struct pollfd {
 	int fd;           /* file descriptor */
 	short events;     /* requested events */

@@ -48,6 +49,7 @@ struct pollfd {
 };
 #define POLLIN 1
 #define POLLHUP 2
+#endif
 
 typedef void (__cdecl *sig_handler_t)(int);
 struct sigaction {

index 6daf313..53a6d30 100644
@@ -1,13 +1,6 @@
 #ifndef __MSVC__HEAD
 #define __MSVC__HEAD
 
-/* Define minimize windows version */
-#define WINVER 0x0500
-#define _WIN32_WINNT 0x0500
-#define _WIN32_WINDOWS 0x0410
-#define _WIN32_IE 0x0700
-#define NTDDI_VERSION NTDDI_WIN2KSP1
-#include <winsock2.h>
 #include <direct.h>
 #include <process.h>
 #include <malloc.h>

index e8c178d..8ce9104 100644
@@ -2,7 +2,9 @@
 #define WIN32_H
 
 /* common Win32 functions for MinGW and Cygwin */
+#ifndef WIN32         /* Not defined by Cygwin */
 #include <windows.h>
+#endif
 
 static inline int file_attr_to_st_mode (DWORD attr)
 {

index 9217c24..dedce21 100644
@@ -2,7 +2,6 @@
  * Copyright 2008 Peter Harris <[email protected]>
  */
 
-#include <windows.h>
 #include "../git-compat-util.h"
 
 /*

index 8ea444f..8d6e29c 100644
@@ -65,6 +65,12 @@
 #define _NETBSD_SOURCE 1
 #define _SGI_SOURCE 1
 
+#ifdef WIN32 /* Both MinGW and MSVC */
+#define WIN32_LEAN_AND_MEAN  /* stops windows.h including winsock.h */
+#include <winsock2.h>
+#include <windows.h>
+#endif
+
 #include <unistd.h>
 #include <stdio.h>
 #include <sys/stat.h>

index 55e7e29..4f9c829 100644
@@ -1,9 +1,6 @@
 #include "cache.h"
 
-#ifdef _WIN32
-#  define WIN32_LEAN_AND_MEAN
-#  include <windows.h>
-#elif defined(hpux) || defined(__hpux) || defined(_hpux)
+#if defined(hpux) || defined(__hpux) || defined(_hpux)
 #  include <sys/pstat.h>
 #endif
 

1.6.2.1.418.g33d56.dirty

---
 compat/msvc.h |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

index 53a6d30..9c753a5 100644
@@ -10,6 +10,8 @@
 #define __inline__ __inline
 #define __attribute__(x)
 #define va_copy(dst, src)     ((dst) = (src))
+#define strncasecmp  _strnicmp
+#define ftruncate    _chsize
 
 static __inline int strcasecmp (const char *s1, const char *s2)
 {

1.6.2.1.418.g33d56.dirty

Enable MSVC builds with GNU Make by simply calling
make MSVC=1
(Debug build possible by adding DEBUG=1 as well)

Two scripts, clink.pl and lib.pl, are used to convert certain GCC specific command line options into something MSVC understands. By building for MSVC with GNU Make, we can ensure that the MSVC port always follows the latest code, and does not lag behind due to unmaintained NMake Makefile or IDE projects.

---
 Makefile                        |   55 ++++++++++++++++++++++++++++++++++++++-
 compat/vcbuild/scripts/clink.pl |   48 ++++++++++++++++++++++++++++++++++
 compat/vcbuild/scripts/lib.pl   |   26 ++++++++++++++++++
 3 files changed, 128 insertions(+), 1 deletions(-)
 create mode 100644 compat/vcbuild/scripts/clink.pl
 create mode 100644 compat/vcbuild/scripts/lib.pl

index bde2acd..aa918eb 100644
@@ -877,6 +877,58 @@ ifneq (,$(findstring CYGWIN,$(uname_S)))
 	COMPAT_OBJS += compat/cygwin.o
 	UNRELIABLE_FSTAT = UnfortunatelyYes
 endif
+ifdef MSVC
+	pathsep = ;
+	NO_PREAD = YesPlease
+	NO_OPENSSL = YesPlease
+	NO_LIBGEN_H = YesPlease
+	NO_SYMLINK_HEAD = YesPlease
+	NO_IPV6 = YesPlease
+	NO_SETENV = YesPlease
+	NO_UNSETENV = YesPlease
+	NO_STRCASESTR = YesPlease
+	NO_STRLCPY = YesPlease
+	NO_MEMMEM = YesPlease
+	# NEEDS_LIBICONV = YesPlease
+	NO_ICONV = YesPlease
+	NO_C99_FORMAT = YesPlease
+	NO_STRTOUMAX = YesPlease
+	NO_STRTOULL = YesPlease
+	NO_MKDTEMP = YesPlease
+	NO_MKSTEMPS = YesPlease
+	SNPRINTF_RETURNS_BOGUS = YesPlease
+	NO_SVN_TESTS = YesPlease
+	NO_PERL_MAKEMAKER = YesPlease
+	RUNTIME_PREFIX = YesPlease
+	NO_POSIX_ONLY_PROGRAMS = YesPlease
+	NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease
+	NO_NSEC = YesPlease
+	USE_WIN32_MMAP = YesPlease
+	# USE_NED_ALLOCATOR = YesPlease
+	UNRELIABLE_FSTAT = UnfortunatelyYes
+	OBJECT_CREATION_USES_RENAMES = UnfortunatelyNeedsTo
+	NO_REGEX = YesPlease
+	NO_CURL = YesPlease
+	NO_PTHREADS = YesPlease
+
+	CC = compat/vcbuild/scripts/clink.pl
+	AR = compat/vcbuild/scripts/lib.pl
+	CFLAGS =
+	BASIC_CFLAGS = -nologo -I. -I../zlib -Icompat/vcbuild -Icompat/vcbuild/include -DWIN32-D_CONSOLE -DHAVE_STRING_H -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE
+	COMPAT_OBJS = compat/msvc.o compat/fnmatch/fnmatch.o compat/winansi.o
+	COMPAT_CFLAGS = -D__USE_MINGW_ACCESS -DNOGDI -DHAVE_STRING_H -DHAVE_ALLOCA_H -Icompat -Icompat/fnmatch -Icompat/regex -Icompat/fnmatch -DSTRIP_EXTENSION=\".exe\"
+	BASIC_LDFLAGS = -IGNORE:4217 -IGNORE:4049 -NOLOGO -SUBSYSTEM:CONSOLE -NODEFAULTLIB:MSVCRT.lib
+	EXTLIBS = advapi32.lib shell32.lib wininet.lib ws2_32.lib
+	lib =
+ifndef DEBUG
+	BASIC_CFLAGS += -GL -Os -MT
+	BASIC_LDFLAGS += -LTCG
+	AR += -LTCG
+else
+	BASIC_CFLAGS += -Zi -MTd
+endif
+	X = .exe
+else
 ifneq (,$(findstring MINGW,$(uname_S)))
 	pathsep = ;
 	NO_PREAD = YesPlease

@@ -925,6 +977,7 @@ else
 	NO_PTHREADS = YesPlease
 endif
 endif
+endif
 
 -include config.mak.autogen
 -include config.mak

@@ -1327,7 +1380,7 @@ strip: $(PROGRAMS) git$X
 git.o: git.c common-cmds.h GIT-CFLAGS
 	$(QUIET_CC)$(CC) -DGIT_VERSION='"$(GIT_VERSION)"' \
 		'-DGIT_HTML_PATH="$(htmldir_SQ)"' \
-		$(ALL_CFLAGS) -c $(filter %.c,$^)
+		$(ALL_CFLAGS) -o $@ -c $(filter %.c,$^)
 
 git$X: git.o $(BUILTIN_OBJS) $(GITLIBS)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ git.o \

new file mode 100644
index 0000000..68f6644
@@ -0,0 +1,26 @@
+#!/usr/bin/perl -w
+######################################################################
+# Libifies files on Windows
+#
+# This is a wrapper to facilitate the compilation of Git with MSVC
+# using GNU Make as the build system. So, instead of manipulating the
+# Makefile into something nasty, just to support non-space arguments
+# etc, we use this wrapper to fix the command line options
+#
+# Copyright (C) 2009 Marius Storm-Olsen <[email protected]>
+######################################################################
+use strict;
+my @args = ();
+while (@ARGV) {
+	my $arg = shift @ARGV;
+	if ("$arg" eq "rcs") {
+		# Consume the rcs option
+	} elsif ("$arg" =~ /\.a$/) {
+		push(@args, "-OUT:$arg");
+	} else {
+		push(@args, $arg);
+	}
+}
+unshift(@args, "lib.exe");
+# printf("**** @args\n");
+exit system(@args);

1.6.2.1.418.g33d56.dirty

Based on original README patch from Frank Li

---
 compat/vcbuild/README |   39 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 39 insertions(+), 0 deletions(-)
 create mode 100644 compat/vcbuild/README

new file mode 100644
index 0000000..354526a
@@ -0,0 +1,39 @@
+The Steps of Build Git with VS2008
+
+1. You need the build environment, which contains the Git dependencies
+   to be able to compile, link and run Git with MSVC.
+
+   You can either use the binary repository:
+
+       WWW: http://repo.or.cz/w/msvcgit.git
+       Git: git clone git://repo.or.cz/msvcgit.git
+       Zip: http://repo.or.cz/w/msvcgit.git?a=snapshot;h=master;sf=zip
+
+   and call the setup_32bit_env.cmd batch script before compiling Git,
+   (see repo/package README for details), or the source repository:
+
+       WWW: http://repo.or.cz/w/gitbuild.git
+       Git: git clone git://repo.or.cz/gitbuild.git
+       Zip: (None, as it's a project with submodules)
+
+   and build the support libs as instructed in that repo/package.
+
+2. Ensure you have the msysgit environment in your path, so you have
+   GNU Make, bash and perl available.
+
+       WWW: http://repo.or.cz/w/msysgit.git
+       Git: git clone git://repo.or.cz/msysgit.git
+       Zip: http://repo.or.cz/w/msysgit.git?a=snapshot;h=master;sf=zip
+
+   This environment is also needed when you use the resulting
+   executables, since Git might need to run scripts which are part of
+   the git operations.
+
+3. Inside Git's directory run the command:
+       make common-cmds.h
+   to generate the common-cmds.h file needed to compile git.
+
+4. Then build Git with the GNU Make Makefile in the Git projects root
+       make MSVC=1
+
+Done!

1.6.2.1.418.g33d56.dirty

These scripts generate projects for the MSVC IDE (.vcproj files) or QMake (.pro files), based on the output of a 'make -n MSVC=1 V=1' run.

This enables us to simply do the necesarry changes in the Makefile, and you can update the other buildsystems by regenerating the files. Keeping the other buildsystems up-to-date with main development.

The generator system is designed to easily drop in pm's for other buildsystems as well, if someone has an itch. However, the focus has been Windows development, so the 'engine' might need patches to support any platform.

Also add some .gitignore entries for MSVC files.

---
 .gitignore                                |   11 +
 compat/vcbuild/README                     |   13 +-
 contrib/buildsystems/Generators.pm        |   42 ++
 contrib/buildsystems/Generators/QMake.pm  |  189 +++++++++
 contrib/buildsystems/Generators/Vcproj.pm |  639 +++++++++++++++++++++++++++++
 contrib/buildsystems/engine.pl            |  353 ++++++++++++++++
 contrib/buildsystems/generate             |   29 ++
 contrib/buildsystems/parse.pl             |  228 ++++++++++
 8 files changed, 1503 insertions(+), 1 deletions(-)
 create mode 100644 contrib/buildsystems/Generators.pm
 create mode 100644 contrib/buildsystems/Generators/QMake.pm
 create mode 100644 contrib/buildsystems/Generators/Vcproj.pm
 create mode 100644 contrib/buildsystems/engine.pl
 create mode 100644 contrib/buildsystems/generate
 create mode 100644 contrib/buildsystems/parse.pl

index 47672b0..51a37b1 100644
@@ -179,3 +179,14 @@ configure
 tags
 TAGS
 cscope*
+*.obj
+*.lib
+*.sln
+*.suo
+*.ncb
+*.vcproj
+*.user
+*.idb
+*.pdb
+Debug/
+Release/

index 354526a..df8a657 100644
@@ -33,7 +33,18 @@ The Steps of Build Git with VS2008
        make common-cmds.h
    to generate the common-cmds.h file needed to compile git.
 
-4. Then build Git with the GNU Make Makefile in the Git projects root
+4. Then either build Git with the GNU Make Makefile in the Git projects
+   root
        make MSVC=1
+   or generate Visual Studio solution/projects (.sln/.vcproj) with the
+   command
+       perl contrib/buildsystems/generate -g Vcproj
+   and open and build the solution with the IDE
+       devenv git.sln /useenv
+   or build with the IDE build engine directly from the command line
+       devenv git.sln /useenv /build "Release|Win32"
+   The /useenv option is required, so Visual Studio picks up the
+   environment variables for the support libraries required to build
+   Git, which you set up in step 1.
 
 Done!

new file mode 100644
index 0000000..bc10f25
@@ -0,0 +1,29 @@
+#!/usr/bin/perl -w
+######################################################################
+# Generate buildsystem files
+#
+# This script generate buildsystem files based on the output of a
+# GNU Make --dry-run, enabling Windows users to develop Git with their
+# trusted IDE with native projects.
+#
+# Note:
+# It is not meant as *the* way of building Git with MSVC, but merely a
+# convenience. The correct way of building Git with MSVC is to use the
+# GNU Make tool to build with the maintained Makefile in the root of
+# the project. If you have the msysgit environment installed and
+# available in your current console, together with the Visual Studio
+# environment you wish to build for, all you have to do is run the
+# command:
+#     make MSVC=1
+#
+# Copyright (C) 2009 Marius Storm-Olsen <[email protected]>
+######################################################################
+use strict;
+use File::Basename;
+use Cwd;
+
+my $git_dir = getcwd();
+$git_dir =~ s=\\=/=g;
+$git_dir = dirname($git_dir) while (!-e "$git_dir/git.c" && "$git_dir" ne "");
+die "Couldn't find Git repo" if ("$git_dir" eq "");
+exec join(" ", ("PERL5LIB=${git_dir}/contrib/buildsystems ${git_dir}/contrib/buildsystems/engine.pl", @ARGV));

1.6.2.1.418.g33d56.dirty

Johannes Sixt said the following on 17.09.2009 22:28:

When using only the vcproj generator, you are correct. However, if you use the qmake generator, and create vcprojs from those, the *.idb and *.pdb files are located in the project directory itself, and not under Debug/. I'm not too worried about this case though, so for me, the three entries *.obj, *.idb and *.pdb can go.

Junio, you want me to push a new patch?

--
.marius

Fair enough.

That's not necessary. I just wanted to make sure that you added the entries deliberately.

With these questions answered, and the most recent change to 04/15 that adjust test-genrandom.c, the MinGW aspect of this series is

Thank you very much for your work and persistency!

-- Hannes

This may help us debug issues on Windows, as we now can build Git natively on Windows with both MinGW and MSVC.

---
I'm just throwing this one out there. If people think manipulating the version here, to ease debugging, I don't mind if this patch is squashed into patch 12.
If people don't like it, just skip this path.

 Makefile |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

index aa918eb..2c20922 100644
@@ -878,6 +878,7 @@ ifneq (,$(findstring CYGWIN,$(uname_S)))
 	UNRELIABLE_FSTAT = UnfortunatelyYes
 endif
 ifdef MSVC
+	GIT_VERSION := $(GIT_VERSION).MSVC
 	pathsep = ;
 	NO_PREAD = YesPlease
 	NO_OPENSSL = YesPlease

1.6.2.1.418.g33d56.dirty

Johannes Sixt said the following on 17.09.2009 22:18:

Ok good, we'll just keep it as is then.
Thanks.

--
.marius

I know I'm a little late with my comments as this patch set has already been merged to master. However, for future reference I'd like to point out that something similar could be archived by using e.g. CMake, and only maintaining the CMake project file. I'm not suggesting to actually switch to CMake at this time, but I wanted to point out that a guy called Pau Garcia i Quiles already seems to have created a preliminary CMakeLists.txt file for Git [1], and also tried to build Git for Windows using his CMake-generated MSVC project files.

[1] "CMake-ifying git", http://www.elpauer.org/?p=324

--
Sebastian

Hi,

We actually discussed this, and I challenged Pau to provide a recipe (a la /src/openssl/release.sh) that builds, installs and commits CMake.

There was no response after that challenge.

Ciao,
Dscho

Johannes Schindelin said the following on 23.09.2009 22:37:

Also, the buildsystem scripts I added to contrib/buildsystem is made in such a way that creating a CMake generator would be very little overhead. And since it's very unlikely that any other buildsystem will replace the default Makefile anytime soon, I think that is the way to go, to generate files for all the other buildsystems based on the output of the Makefile. Not optimal, sure, but follows main development.

Note that the contrib/buildsystem scripts could need a major overhaul by a Perl guru, as they are currently tuned only for Windows consumption (and Msys Perl being the only Perl distribution tested). But they work as intended for the use-case in mind.

--
.marius

[...]

Have you considered simply including MSVC's sys/utime.h here? From a first glance, it seems as if it contains all required symbols.

--
Sebastian

Sebastian Schuberth said the following on 23.09.2009 12:03:

This was a patch which originated from Frank Li's original series. While we might have been able to simply use the MSVC one, I'm sure Frank had a reason for overriding it with the small content which you see in this patch. We'll have to ask Frank about that. We can always remove it now, if the MSVC version works ok and doesn't introduce any compiler errors/warnings.

While I appreciate your comments, they are a tad late, as the whole series is already in master.. :)
See http://repo.or.cz/w/git.git?a=shortlog;h=refs/heads/master

So, give it a try, to see if removing the include override
compat/vcbuild/include/utime.h
doesn't introduce a regression, and send a patch for it.

--
.marius

At beginning, I don't want to copy all msys header files to vcbuilc\include and copy only necessary header file and data structure.
Many structures have been defined at MSVC standard file. So I use simple utimer.h and unistd.h and dirent.h.

Best regards
Frank Li

Indeed, even in recent Visual Studio versions the C compiler only understands C90, not C99. Would it make sense to just force MSVC to compile *.c files with the C++ compiler to fix this, rather than patching files (which might be necessary for future files, too)? See the "/TP" command line option to "CL".

--
Sebastian

I don't think so. There's many other compilers out there that doesn't support C99 features, and most of the git source code does not depend on them. IMO, it is better to fix these issues for all those compilers once and for all, instead of having to find ways of working around them whenever someone changes compiler.

Besides, C++ isn't a strict superset of C. There might be warnings or errors introduced by compiling the code as C++. Implicit conversions from void* springs to mind. Quick tests reveals that MSVC doesn't seem to warn or error about them, but there might be some other differences where we do get into issues.

Erik "kusma" Faye-Lund
[email protected]
(+47) 986 59 656