Fix checkout of large files to network shares under Windows XP
PATCH 1/1 Fix checkout of large files to network shares under Windows XP
This fixes msysGit issue 409, see
http://code.google.com/p/msysgit/issues/detail?id=409
--- compat/mingw.c | 24 ++++++++++++++++++++++++ compat/mingw.h | 3 +++ 2 files changed, 27 insertions(+), 0 deletions(-)
compat/mingw.c +24 -0
@@ -293,6 +293,30 @@ int mingw_open (const char *filename, int oflags, ...)return fd; } +#undef write +ssize_t mingw_write (int fd,const void *buf,size_t count) +{ +ssize_t written =0 ; +size_t total =0 , size = count; + +while (total < count && size >0 ) { + written =write (fd, buf, size); +if (written <0 && errno ==EINVAL ) { +// There seems to be a bug in the Windows XP network stack that +// causes writes with sizes > 64 MB to fail, so we halve the size +// until we succeed or ultimately fail. + size /=2 ; + }else { + buf += written; + total += written; +if (total + size > count) + size = count - total; + } + } + +return written <0 ? written : total; +} +#undef fopenFILE *mingw_fopen (const char *filename,const char *otype) {
compat/mingw.h +3 -0
@@ -178,6 +178,9 @@ int mingw_rmdir(const char *path);int mingw_open (const char *filename,int oflags, ...);#define open mingw_open +ssize_t mingw_write (int fd,const void *buf,size_t count); +#define write mingw_write +FILE *mingw_fopen (const char *filename,const char *otype);#define fopen mingw_fopen
Signature · 1 line
1.7.0.2.msysgit.0.898.gbf4f.dirty
Am 19.04.2010 14:45, schrieb Sebastian Schuberth:
Quoted 19 lines
This fixes msysGit issue 409, see
http://code.google.com/p/msysgit/issues/detail?id=409Signed-off-by Sebastian Schuberth <[email protected]>--- compat/mingw.c | 24 ++++++++++++++++++++++++ compat/mingw.h | 3 +++ 2 files changed, 27 insertions(+), 0 deletions(-)compat/mingw.c +2
-0@@ -293,6 +293,30 @@ int mingw_open (const char *filename, int oflags, ...)return fd; } +#undef write +ssize_t mingw_write (int fd,const void *buf,size_t count)
compat/mingw.c int mingw_open (const char *filename, int oflags, ...) · line 307
Quoted 10 lines
compat/mingw.c +10
-0+{ +ssize_t written =0 ; +size_t total =0 , size = count; + +while (total < count && size >0 ) { + written =write (fd, buf, size); +if (written <0 && errno ==EINVAL ) { +// There seems to be a bug in the Windows XP network stack that +// causes writes with sizes > 64 MB to fail, so we halve the size +// until we succeed or ultimately fail.
C style comments (/*...*/) are preferred over C++ style comments (//...) for git.
Is there a known-good size, or at least a mostly-working one? Would it make sense to start with that size instead of halving and trying until that size is reached?
compat/mingw.c int mingw_open (const char *filename, int oflags, ...) · line 311
Quoted 4 lines
compat/mingw.c +4
-0+ size /=2 ; + }else { + buf += written; + total += written;
What about other errors? You need to break out of the loop instead of adding -1 to buf and total, right?
compat/mingw.c int mingw_open (const char *filename, int oflags, ...) · line 315
Quoted 4 lines
compat/mingw.c +4
-0+if (total + size > count) + size = count - total; + } + }
Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
Quoted 21 lines
compat/mingw.c +4
-0+ +return written <0 ? written : total; +} +#undef fopenFILE *mingw_fopen (const char *filename,const char *otype) {compat/mingw.h +3
-0@@ -178,6 +178,9 @@ int mingw_rmdir(const char *path);int mingw_open (const char *filename,int oflags, ...);#define open mingw_open +ssize_t mingw_write (int fd,const void *buf,size_t count); +#define write mingw_write +FILE *mingw_fopen (const char *filename,const char *otype);#define fopen mingw_fopen
Am 4/19/2010 22:43, schrieb René Scharfe:
Quoted 21 lines
Am 19.04.2010 14:45, schrieb Sebastian Schuberth:
compat/mingw.c int mingw_open (const char *filename, int oflags, ...) · line 307
Quoted 12 lines
compat/mingw.c +12
-0+#undef write +ssize_t mingw_write (int fd,const void *buf,size_t count) +{ +ssize_t written =0 ; +size_t total =0 , size = count; + +while (total < count && size >0 ) { + written =write (fd, buf, size); +if (written <0 && errno ==EINVAL ) { +// There seems to be a bug in the Windows XP network stack that +// causes writes with sizes > 64 MB to fail, so we halve the size +// until we succeed or ultimately fail. C style comments (/*...*/) are preferred over C++ style comments (//...) for git.
Is there a known-good size, or at least a mostly-working one? Would it make sense to start with that size instead of halving and trying until that size is reached?
Quoted 7 lines
compat/mingw.c int mingw_open (const char *filename, int oflags, ...) · line 311
Quoted 4 lines
compat/mingw.c +4
-0+ size /=2 ; + }else { + buf += written; + total += written;What about other errors? You need to break out of the loop instead of adding -1 to buf and total, right?
Thanks for a thorough review. I had the gut feeling that something's wrong with the code due to its structure, but didn't stare at the code long enough to notice this.
I suggest to have this structure
write
if success or failure is not EINVAL
return
do
reduce size
if larger than known (presumed?) maximum
reduce to that maximum
write
while not success and failure is EINVAL
while not failure and exactly reduced size written
write more
I don't think that we will observe any short writes *after* the size was reduced, which Albert is concerned about. Somebody who observes the failure that this works around could instrument the function to see whether short writes are really a problem.
-- Hannes
Quoted 7 lines · René Scharfe
Quoted 4 lines
+ if (written < 0 && errno == EINVAL) {
+ // There seems to be a bug in the Windows XP network stack that
+ // causes writes with sizes > 64 MB to fail, so we halve the size
+ // until we succeed or ultimately fail.C style comments (/*...*/) are preferred over C++ style comments (//...) for git.
Oh well, I've changed that.
Quoted 3 lines
Is there a known-good size, or at least a mostly-working one? Would it make sense to start with that size instead of halving and trying until that size is reached?
As the comment says, the greatest size that worked in my experiments is 64 MB, and other posts on the Internert suggest the same limit, but it's still an unconfirmed / undocumented issue. Anyway, I'm now starting with 64 MB right away if a write failed.
Quoted 7 lines
Quoted 4 lines
+ size /= 2;
+ } else {
+ buf += written;
+ total += written;What about other errors? You need to break out of the loop instead of adding -1 to buf and total, right?
Right, I've fixed that, thanks.
Quoted 8 lines
Quoted 4 lines
+ if (total + size > count)
+ size = count - total;
+ }
+ }Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
I prefer to make the wrapper as transparent as possible. If a direct call to write would not write less than requested, the wrapper should not either.
I've updated work/issue-409 in 4msysgit.git accordingly.
Signature · 1 line
Sebastian Schuberth
Am 4/20/2010 14:42, schrieb Sebastian Schuberth:
Quoted 8 lines · René Scharfe
Quoted 3 lines
Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
I prefer to make the wrapper as transparent as possible. If a direct call to write would not write less than requested, the wrapper should not either.
Sure, but René meant the opposite case: When fewer bytes than requested were written, then you shouldn't retry to write more! That is, you should exit the loop when write(fd, buf, n) does not return n.
I still find your code unnecessarily hard to read. In particular, you should extract the non-problematic case out of the loop. If you followed my suggestion elsewhere in the thread, you wouldn't have to write any conditionals that 'break' out of a loop.
-- Hannes
Quoted 6 lines · Johannes Sixt
Am 4/20/2010 14:42, schrieb Sebastian Schuberth:
Quoted 5 lines · René Scharfe
Quoted 3 lines
Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
Quoted 7 lines
Quoted 3 lines
I prefer to make the wrapper as transparent as possible. If a direct call to write would not write less than requested, the wrapper should not either.
Sure, but René meant the opposite case: When fewer bytes than requested were written, then you shouldn't retry to write more! That is, you should exit the loop when
write(fd, buf, n)does not return n.
I see what you mean, but I do not fully agree. Who guarantees that (on some obscure OS) a following call to write(fd, buf, n) will not return n again, maybe because write() temporarily decided to write fewer bytes than requested to make the next write() call do aligned writes to something? That case then is probably already handled in the caller to write(), but at least my code is not wrong in that respect.
Quoted 4 lines
I still find your code unnecessarily hard to read. In particular, you should extract the non-problematic case out of the loop. If you followed my suggestion elsewhere in the thread, you wouldn't have to write any conditionals that 'break' out of a loop.
I didn't follow your suggestion on purpose because I experimented with it and I found *yours* to be hard to read. It has three calls to write() and more places where errors need to be checked. As I do not have the will to waste more time on style discussions about code that fixes other people's issues, and not the time to test the code on Windows XP over and over again, I hope you are willing to accept code that is different from how you would have written it. So it's take it or leave it (or modify it yourself, if you feel like it).
Signature · 1 line
Sebastian Schuberth
Am 20.04.2010 14:42, schrieb Sebastian Schuberth:
Quoted 7 lines
Quoted 3 lines
Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
I prefer to make the wrapper as transparent as possible. If a direct call to write would not write less than requested, the wrapper should not either.
After the call failed, we don't know how many bytes would have been written had it succeeded.
But I agree with Albert's reasoning to use the knowledge of a working chunk size in order to minimize the number of write(2) calls. Otherwise we'd have to search for a working size again and again, generating lots of failing calls.
Quoted 1 line
I've updated work/issue-409 in 4msysgit.git accordingly.
This patch doesn't help in the test case I cobbled together quickly. It's a Windows XP SP3 client on VMWare mapping a file share exported by a Netapps filer, over a VPN. It's very slow, and I admit that it's a weird setup. I wouldn't actually use it that way, but couldn't find another file share momentarily.
I can check out a 1MB file, but checking out a 32MB file fails. I've added a fprintf() to the loop and I can see that it's halving the size and retries, as intended, until it eventually hits zero.
The file is created using the correct file size (32MB), though.The first failed write(2) call needs to be undone somehow before we can try again, it seems. Do we have to seek back or truncate the file?
Replacing the body of mingw_write() with the following line allows me to check out the 32MB file, by the way:
return write(fd, buf, min(count, 1024 * 1024));
René
Am 20.04.2010 22:49, schrieb René Scharfe:
Quoted 13 lines
Am 20.04.2010 14:42, schrieb Sebastian Schuberth:
Quoted 1 line
I've updated work/issue-409 in 4msysgit.git accordingly.
This patch doesn't help in the test case I cobbled together quickly. It's a Windows XP SP3 client on VMWare mapping a file share exported by a Netapps filer, over a VPN. It's very slow, and I admit that it's a weird setup. I wouldn't actually use it that way, but couldn't find another file share momentarily.
I can check out a 1MB file, but checking out a 32MB file fails. I've added a
fprintf()to the loop and I can see that it's halving the size and retries, as intended, until it eventually hits zero.
Quoted 8 lines
The file is created using the correct file size (32MB), though.The first failed write(2) call needs to be undone somehow before we can try again, it seems. Do we have to seek back or truncate the file?
Replacing the body of
mingw_write()with the following line allows me to check out the 32MB file, by the way:return write(fd, buf, min(count, 1024 * 1024));
OK, I've been searching the web looking for documentation that explains the issue, but haven't found any watertight evidence.
With Process Monitor [1], I found out that local write() calls are translated to WriteFile() calls with a maximum size of 64KB on Windows XP and 256KB on Vista. Writes to a network drive are translated to WriteFile() calls without applying a size cap.
In a forum discussion [2], someone explained that unbuffered writes using WriteFile() have certain size limits, depending on the processor:
x86: 67,076,096 X64: 33,525,760 IA64: 67,051,520
This post is cited by an MS employee in the comment section of the online documentation for WriteFile() [3].
A Knowledge Base article [4] documents a size limit of 67,076,095 for fwrite() to a file on a network drive. fwrite() calls translate to WriteFile() calls, too. The reason for this is said to be an OS problem, but no OS versions are named at all in this article.
Please note that the size limit of fwrite() is suspiciously close to the one for an unbuffered WriteFile().
In my test setup, write() calls to a network drive fail with a size parameter of 32MB, while with 33,525,760 bytes they succeed.
Based on these two observations, I suspect that there's a connection between writes to a network drive and unbuffered writes. Perhaps writes over the net are passed to the NIC driver in one piece which is locked into RAM?
The connection is a bit weak (it would be good to have someone comment on this who actually knows something about this topic and doesn't have to guesswork through a bunch of websites), but I think it's enough to create a patch. Based on the numbers above, I think 31MB is a good size to simply cap writes.
When we learn that other systems need a lower limit still, we can easily reduce our cap, without affecting local performance.
It would be nice to reach chris.gcode, who originally reported this problem [5], and ask him to test. I couldn't find an email address on that webpage, though. His proposed patch there also used an upper limit slightly below 32MB, but tried to compensate for capping by looping until the full requested size was written. That's not really needed.
René
[1] http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
[2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/fef1c9b5-fd92-4ada-8de5-44c2eb30b516
[3] http://msdn.microsoft.com/en-us/library/aa365747(VS.85).aspx#5
[4] http://support.microsoft.com/kb/899149
[5] http://code.google.com/p/msysgit/issues/detail?id=409
-- >8 --
Bigger writes to network drives on Windows XP fail. Cap them at 31MB to allow them to succeed. Callers need to be prepared for write() calls that do less work than requested anyway.
On local drives, write() calls are translated to WriteFile() calls with a cap of 64KB on Windows XP and 256KB on Vista. Thus a cap of 31MB won't affect the number of WriteFile() calls which do the actual work. There's still room for some other version of Windows to use a chunk size of 1MB without increasing the number of system calls.
--- compat/mingw.c | 17 +++++++++++++++++ compat/mingw.h | 3 +++ 2 files changed, 20 insertions(+), 0 deletions(-)
compat/mingw.c +17 -0
@@ -140,6 +140,23 @@ int mingw_open (const char *filename, int oflags, ...)return fd; } +#undef write +ssize_t mingw_write (int fd,const void *buf,size_t count) +{ +/* +* While write() calls to a file on a local disk are translated +* into WriteFile() calls with a maximum size of 64KB on Windows +* XP and 256KB on Vista, no such cap is placed on writes to +* files over the network on Windows XP. Unfortunately, there +* seems to be a limit of 32MB-28KB on X64 and 64MB-32KB on x86; +* bigger writes fail on Windows XP. +* So we cap to a nice 31MB here to avoid write failures over +* the net without changing the number of WriteFile() calls in +* the local case. +*/ +return write (fd, buf,min (count,31 *1024 *1024 )); +} +#undef fopenFILE *mingw_fopen (const char *filename,const char *otype) {
compat/mingw.h +3 -0
@@ -170,6 +170,9 @@ int link(const char *oldpath, const char *newpath);int mingw_open (const char *filename,int oflags, ...);#define open mingw_open +ssize_t mingw_write (int fd,const void *buf,size_t count); +#define write mingw_write +FILE *mingw_fopen (const char *filename,const char *otype);#define fopen mingw_fopen
Signature · 1 line
1.7.1
Am 4/29/2010 22:01, schrieb René Scharfe:
Quoted 7 lines
(quoted diff) +7
-0+#undef write +ssize_t mingw_write(int fd, const void *buf, size_t count) +{ + /* + * While write() calls to a file on a local disk are translated + * into WriteFile() calls with a maximum size of 64KB on Windows + * XP and 256KB on Vista, no such cap is placed on writes to
Review
Quoted 10 lines
(quoted diff) +10
-0+ * files over the network on Windows XP. Unfortunately, there + * seems to be a limit of 32MB-28KB on X64 and 64MB-32KB on x86; + * bigger writes fail on Windows XP. + * So we cap to a nice 31MB here to avoid write failures over + * the net without changing the number of WriteFile() calls in + * the local case. + */ + return write(fd, buf, min(count, 31 * 1024 * 1024)); +} +
Thanks, I have verified that this fixes the problem in my setup as well. I'll queue the patch with the below test case squashed in.
-- Hannes
t/t5705-clone-2gb.sh +9 -3
@@ -12,7 +12,7 @@ test_expect_success 'setup' ' git config pack.compression 0 && git config pack.depth 0 && - blobsize=$((20*1024*1024)) && + blobsize=$((100*1024*1024)) && blobcount=$((2*1024*1024*1024/$blobsize+1)) && i=1 && (while test $i -le $blobcount
@@ -36,9 +36,15 @@ test_expect_success 'setup' ' ' -test_expect_success 'clone' ' +test_expect_success 'clone - bare' ' - git clone --bare --no-hardlinks . clone + git clone --bare --no-hardlinks . clone-bare + +' + +test_expect_success 'clone - with worktree, file:// protocol' ' + + git clone file://. clone-wt '
Quoted 2 lines · René Scharfe
OK, I've been searching the web looking for documentation that explains the issue, but haven't found any watertight evidence.
Thanks René for your research and for finally finishing the fix!
Quoted 5 lines
It would be nice to reach chris.gcode, who originally reported this problem [5], and ask him to test. I couldn't find an email address on that webpage, though. His proposed patch there also used an upper limit slightly below 32MB, but tried to compensate for capping by looping until the full requested size was written. That's not really needed.
I believe to have found his e-mail address on the msysGit mailing list and put him on CC.
Signature · 1 line
Sebastian Schuberth
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
I've cc:'d the mailing lists and the participants of the earlier discussion to share your findings and to see if someone else may have an explanation for this behaviour, or a solution.
The patch in question is c8b296450e5148c576697ea4709072b7855aacd5 and has made it into git versions 1.7.1.1 and 1.7.2 (and up). It caps writes at 31MB to fix a problem with network drives in certain versions of Windows.
Am 14.10.2010 21:26, schrieb Chad Warner:
Quoted 10 lines
I am experiencing issues with the patch you suggested. I am working with a repository via its UNC path and it is failing on large files. I first tried lowing to several different values such as 4MB and still had problems. I eventually got it to work with the following:
return
write(fd, buf, min(count, 1024 * 27));I didn't notice any real delays with having to call write that many more times. However, I really don't know how to go about fixing this issue or validating that this really fixes the problem.
Reducing the write cap to 27KB unconditionally sounds pretty drastic to me; it could hurt the local case. I didn't measure it, though, so I might be wrong. Depending on that measurement we perhaps need to find a different solution.
Also, 27KB is an odd number. I take it that 28KB won't fix the issue for you?
I wonder about the cause of this threshold. Wild guess: network driver bug or other network issue?
Which version of Windows do you use on the client? Which OS runs on the server? Do you have any other information that might help in reproducing the problem?
Thanks,
René
Quoted 3 lines · René Scharfe
Also, 27KB is an odd number. I take it that 28KB won't fix the issue for you?
I guess the beginning of the written buffer is not page aligned, and the system locks pages. Thus depending on the buffer alignment, the written maximum can vary up to the size of one page.
Quoted 3 lines
I wonder about the cause of this threshold. Wild guess: network driver bug or other network issue?
Memory pressure? Git may consume large amount of virtual memory during some operations, but I am not sure how it is translated in the number of physical memory consumed by the process as well as what quotas could be on that system. It would be interesting to see what GetProcessMemoryInfo() returns when write() fails.
Dmitry
On Mon, Apr 19, 2010 at 4:43 PM, René Scharfe
<[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 4 lines
+ if (total + size > count)
+ size = count - total;
+ }
+ }Shouldn't the loop be left in the successful case, too? write(2) is allowed to write less than requested, so the caller already needs to deal with that case anyway.
That's what I thought initially, since the code would be cleaner, but I don't like the fact that you could actually end up making a lot more failed write() calls that way, since you restart the size search on each call to mingw_write().
For example, suppose you were calling mingw_write() with a count that was exactly 11.5 times bigger than whatever maximum size write() was willing to accept. If you only did one write() per mingw_write(), letting the caller restart, this will result in 47 failed writes and 16 successes. Letting mingw_write() do the restart (as in the existing code) will end up with 4 failed writes and 16 successes. Now, I assume (wait, this is Windows-- I'd *like to hope*) that a failed write() is a lot cheaper than a successful one, but this still rubs me the wrong way.
Of course, if we know (or can guess) the maximum size write() will take, that would be best.
--bert
Dscho, I saw that you are planning to queue this to your tree; would it make your job easier if I queue this myself now, or if I refrain from queueing this myself and instead wait for you to tell me to pull or apply?
I can go either way.
Hi,
Quoted 4 lines · Junio C Hamano
Dscho, I saw that you are planning to queue this to your tree; would it make your job easier if I queue this myself now, or if I refrain from queueing this myself and instead wait for you to tell me to pull or apply?
It seems to be contested still, but I have no problem with cooking it in devel directly.
Ciao,
Dscho