git status OOM on mmap of large file
joey@darkstar:~/tmp/t> ls -l big-file -rw-r--r-- 1 joey joey 11811160064 Jan 22 17:48 big-file joey@darkstar:~/tmp/t> git status
fatal: Out of memory, realloc failed
This file is checked into git, but using a smudge/clean filter, so the actual data checked into git is a hash. I did so using git-annex v7 mode, but I suppose git lfs would cause the same problem.
[pid 6573] mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
Why status needs to mmap a large file that is not modified and that is configured to pass through smudge/clean, I don't know. It seems like it should be possible for status work in this situation.
Signature · 1 line
see shy jo
Quoted 5 lines · Joey Hess
joey@darkstar:~/tmp/t> ls -l big-file -rw-r--r-- 1 joey joey 11811160064 Jan 22 17:48 big-file joey@darkstar:~/tmp/t> git status
fatal: Out of memory, realloc failed
Quoted 8 lines
This file is checked into git, but using a smudge/clean filter, so the actual data checked into git is a hash. I did so using git-annex v7 mode, but I suppose git lfs would cause the same problem.
[pid 6573]
mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)= -1 ENOMEM (Cannot allocate memory)Why status needs to mmap a large file that is not modified and that is configured to pass through smudge/clean, I don't know.
I believe that currently, Git stores the smudge/clean output in memory until it writes it out. When using the persistent filter process, it's possible for the process to choose to abort the operation, so we store the data in memory until we get the status.
Theoretically, it should be possible for us to write this to a temporary file, and if necessary, rename into place, although I'm not sure how well that will work on Windows. File modes may also be tricky here. Patches are of course welcome.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Quoted 9 lines · Joey Hess
joey@darkstar:~/tmp/t> ls -l big-file -rw-r--r-- 1 joey joey 11811160064 Jan 22 17:48 big-file joey@darkstar:~/tmp/t> git status
fatal: Out of memory, realloc failedThis file is checked into git, but using a smudge/clean filter, so the actual data checked into git is a hash. I did so using git-annex v7 mode, but I suppose git lfs would cause the same problem.
Quoted 5 lines
[pid 6573]
mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)= -1 ENOMEM (Cannot allocate memory)Why status needs to mmap a large file that is not modified and that is configured to pass through smudge/clean, I don't know. It seems like it should be possible for status work in this situation.
One minor point: I don't think this is us mmap-ing the file. The descriptor is -1, and Git never uses PROT_WRITE. This is likely your libc using mmap to fulfill a malloc() request.
That said, it just turns the question into: why did Git try to malloc that many bytes? If I reproduce your example (using a 100MB file) and set GIT_ALLOC_LIMIT in the environment, the backtrace to die() is:
#1 0x0000555555786d65 in memory_limit_check (size=104857601, gentle=0) at wrapper.c:27 #2 0x0000555555787084 in xrealloc (ptr=0x0, size=104857601) at wrapper.c:137 #3 0x000055555575612e in strbuf_grow (sb=0x7fffffffdbf0, extra=104857600) at strbuf.c:98 #4 0x000055555575731a in strbuf_read (sb=0x7fffffffdbf0, fd=4, hint=104857600) at strbuf.c:429 #5 0x0000555555664a1f in apply_single_file_filter (path=0x5555558c787c "foo.rand", ...) #6 0x0000555555665321 in apply_filter (path=0x5555558c787c "foo.rand", ...)
...
Looking at apply_single_file_filter(), it's not the _original_ file that it's trying to store, but rather the data coming back from the filter. It's just that we use the original file size as a hint!
In this case (and I'd venture to say in most gigantic-file cases) it's much larger than we need, to the point of causing a problem.
In other words, I think this patch fixes your problem:
convert.c +1 -1
@@ -732,7 +732,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
though possibly we should actually continue to use the file size as a hint up to a certain point, which avoids reallocations for more "normal" filters where the input and output sizes are in the same ballpark.
Just off the top of my head, something like:
/* guess that the filtered output will be the same size as the original */ hint = len;
/* allocate 10% extra in case the clean size is slightly larger */ hint *= 1.1;
/*
* in any case, never go higher than half of core.bigfileThreshold.
* We'd like to avoid allocating more bytes than that, and that still
* gives us room for our strbuf to preemptively double if our guess is
* just a little on the low side.
*/
if (hint > big_file_threshold / 2)
hint = big_file_threshold / 2;
But to be honest, I have no idea if that would even produce measurable benefits over simply growing the strbuf from scratch (i.e., hint==0).
-Peff
Quoted 9 lines · brian m. carlson
Quoted 4 lines
[pid 6573]
mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)= -1 ENOMEM (Cannot allocate memory)Why status needs to mmap a large file that is not modified and that is configured to pass through smudge/clean, I don't know.
I believe that currently, Git stores the smudge/clean output in memory until it writes it out. When using the persistent filter process, it's possible for the process to choose to abort the operation, so we store the data in memory until we get the status.
For the clean step, that should be OK, since that filter output is tiny (but see my other message for a silly heuristic and an easy fix). And that should be all that "git status" needs. But...
Quoted 4 lines
Theoretically, it should be possible for us to write this to a temporary file, and if necessary, rename into place, although I'm not sure how well that will work on Windows. File modes may also be tricky here. Patches are of course welcome.
I didn't experiment with the smudge side, but I think it uses the same apply_filter() code. Which means that yes, it would try to store the 11GB in memory before writing it out. And I agree writing it out to a file and moving it directly into place is the sanest option there. If that doesn't work, spooling to a tempfile and then streaming it into place would also work.
-Peff
Quoted 32 lines · Jeff King
Quoted 9 lines · Joey Hess
joey@darkstar:~/tmp/t> ls -l big-file -rw-r--r-- 1 joey joey 11811160064 Jan 22 17:48 big-file joey@darkstar:~/tmp/t> git status
fatal: Out of memory, realloc failedThis file is checked into git, but using a smudge/clean filter, so the actual data checked into git is a hash. I did so using git-annex v7 mode, but I suppose git lfs would cause the same problem.
Quoted 5 lines
[pid 6573]
mmap(NULL, 11811164160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)= -1 ENOMEM (Cannot allocate memory)Why status needs to mmap a large file that is not modified and that is configured to pass through smudge/clean, I don't know. It seems like it should be possible for status work in this situation.
One minor point: I don't think this is us mmap-ing the file. The descriptor is -1, and Git never uses
PROT_WRITE. This is likely your libc using mmap to fulfill amalloc()request.That said, it just turns the question into: why did Git try to malloc that many bytes? If I reproduce your example (using a 100MB file) and set
GIT_ALLOC_LIMITin the environment, the backtrace todie()is:#1 0x0000555555786d65 in memory_limit_check (size=104857601, gentle=0) at wrapper.c:27 #2 0x0000555555787084 in xrealloc (ptr=0x0, size=104857601) at wrapper.c:137 #3 0x000055555575612e in strbuf_grow (sb=0x7fffffffdbf0, extra=104857600) at strbuf.c:98 #4 0x000055555575731a in strbuf_read (sb=0x7fffffffdbf0, fd=4, hint=104857600) at strbuf.c:429 #5 0x0000555555664a1f in apply_single_file_filter (path=0x5555558c787c "foo.rand", ...) #6 0x0000555555665321 in apply_filter (path=0x5555558c787c "foo.rand", ...)
Quoted 5 lines
...
Looking at
apply_single_file_filter(), it's not the _original_ file that it's trying to store, but rather the data coming back from the filter. It's just that we use the original file size as a hint!
Really cool! I guessed as far as malloc() but did not actually test it, let alone examine the problem closely like this.
Signature · 1 line
Duy
Quoted 6 lines · Jeff King
I didn't experiment with the smudge side, but I think it uses the same
apply_filter()code. Which means that yes, it would try to store the 11GB in memory before writing it out. And I agree writing it out to a file and moving it directly into place is the sanest option there. If that doesn't work, spooling to a tempfile and then streaming it into place would also work.
I have seen that buffering on the smudge side, yes. In git-annex I happen to use the smudge filter in an unusual way that avoids that being a problem but I think it would affect git-lfs.
Signature · 1 line
see shy jo
Quoted 3 lines · Jeff King
Looking at
apply_single_file_filter(), it's not the _original_ file that it's trying to store, but rather the data coming back from the filter. It's just that we use the original file size as a hint!
Thanks much for working that out!
Quoted 7 lines
In other words, I think this patch fixes your problem:
convert.c +0
-0
convert.c static int apply_single_file_filter(const char *path, const char *src, size_t le · line 738
Quoted 8 lines
convert.c +1
-1@@ -732,7 +732,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
Yes, confirmed that does fix it.
Quoted 19 lines
though possibly we should actually continue to use the file size as a hint up to a certain point, which avoids reallocations for more "normal" filters where the input and output sizes are in the same ballpark.
Just off the top of my head, something like:
/* guess that the filtered output will be the same size as the original */ hint = len;
/* allocate 10% extra in case the clean size is slightly larger */ hint *= 1.1;
/*
* in any case, never go higher than half of core.bigfileThreshold.
* We'd like to avoid allocating more bytes than that, and that still
* gives us room for our strbuf to preemptively double if our guess is
* just a little on the low side.
*/
if (hint >big_file_threshold/ 2)
Quoted 4 lines
hint = big_file_threshold / 2;But to be honest, I have no idea if that would even produce measurable benefits over simply growing the strbuf from scratch (i.e., hint==0).
Half of 512 MB is still quite a lot of memory to default to using in this situation. Eg smaller VPS's still often only have a GB or two of ram.
When the clean filter is being used in a way that doesn't involve hashes of large files, it will mostly be operating on typically sized source code files. So capping the maximum hint size around the size of a typical source code file would be plenty for both common cases for the clean filter.
I did some benchmarking, using cat as the clean filter:
git status 32 kb file, hint == len
time 3.865 ms (3.829 ms .. 3.943 ms)
0.994 R² (0.987 R² .. 0.999 R²)
mean 3.934 ms (3.889 ms .. 4.021 ms)
std dev 191.8 μs (106.8 μs .. 291.8 μs)
git status 32 kb file, hint == 0
time 3.887 ms (3.751 ms .. 4.064 ms)
0.992 R² (0.986 R² .. 0.998 R²)
mean 4.002 ms (3.931 ms .. 4.138 ms)
std dev 292.2 μs (189.0 μs .. 498.3 μs)
git status 1 mb file, hint == len
time 3.942 ms (3.816 ms .. 4.078 ms)
0.995 R² (0.991 R² .. 0.999 R²)
mean 3.969 ms (3.916 ms .. 4.054 ms)
std dev 220.1 μs (155.1 μs .. 304.3 μs)
git status 1 mb file, hint == 0
time 3.869 ms (3.836 ms .. 3.911 ms)
0.998 R² (0.995 R² .. 1.000 R²)
mean 3.895 ms (3.868 ms .. 3.947 ms)
std dev 112.3 μs (47.93 μs .. 182.7 μs)
git status 1 gb file, hint == len
time 7.173 s (6.834 s .. 7.564 s)
0.999 R² (0.999 R² .. 1.000 R²)
mean 7.560 s (7.369 s .. 7.903 s)
std dev 333.2 ms (27.65 ms .. 412.2 ms)
git status 1 gb file, hint == 0
time 7.652 s (6.307 s .. 8.263 s)
0.996 R² (0.992 R² .. 1.000 R²)
mean 8.082 s (7.843 s .. 8.202 s)
std dev 232.3 ms (2.362 ms .. 277.1 ms)From this, it looks like the file has to be quite large before the preallocation makes a sizable improvement to runtime, and the smudge/clean filters have to be used for actual content filtering (not for hash generation purposes as git-annex and git-lfs use it). An unusual edge case I think. So hint == 0 seems fine.
Signature · 1 line
see shy jo
Quoted 17 lines · Joey Hess
Quoted 17 lines
Just off the top of my head, something like:
/* guess that the filtered output will be the same size as the original */ hint = len;
/* allocate 10% extra in case the clean size is slightly larger */ hint *= 1.1;
/*
* in any case, never go higher than half of core.bigfileThreshold.
* We'd like to avoid allocating more bytes than that, and that still
* gives us room for our strbuf to preemptively double if our guess is
* just a little on the low side.
*/
if (hint >big_file_threshold/ 2)hint = big_file_threshold / 2;
Quoted 5 lines
Quoted 2 lines
But to be honest, I have no idea if that would even produce measurable benefits over simply growing the strbuf from scratch (i.e., hint==0).
Half of 512 MB is still quite a lot of memory to default to using in this situation. Eg smaller VPS's still often only have a GB or two of ram.
I think you'd want to drop core.bigFileThreshold on such a server, just because Git will happily keep 2*(bigFileThreshold-1) in memory to do a diff. But that nit aside...
Quoted 7 lines
I did some benchmarking, using cat as the clean filter:
[...]
From this, it looks like the file has to be quite large before the preallocation makes a sizable improvement to runtime, and the smudge/clean filters have to be used for actual content filtering (not for hash generation purposes as git-annex and git-lfs use it). An unusual edge case I think. So hint == 0 seems fine.
Thanks for these timings! I agree that "hint == 0" is probably reasonable, then.
I suppose there's no reason not to proceed with a patch around this. For most cases it's really only half the solution (since smudging is going to run into the same problem). But fixing that is quite a bit more involved, and the change itself will be largely orthogonal.
-Peff
Quoted 10 lines · Jeff King
Quoted 7 lines
I did some benchmarking, using cat as the clean filter:
[...]
From this, it looks like the file has to be quite large before the preallocation makes a sizable improvement to runtime, and the smudge/clean filters have to be used for actual content filtering (not for hash generation purposes as git-annex and git-lfs use it). An unusual edge case I think. So hint == 0 seems fine.Thanks for these timings! I agree that "hint == 0" is probably reasonable, then.
One other minor point to consider: on some systems over-allocating actually isn't that expensive, because pages are actually allocated until we write to them, and malloc() is perfectly happy to overcommit memory. Your case would only run into problems on Linux when malloc() actually refuses the allocation (so limiting ourselves to "too large but still reasonable" is a valid strategy there).
But I doubt that's something we should be relying on in general. There are many systems that don't overcommit.
-Peff
PATCH 1/1 avoid unncessary malloc of whole file size
In reply to Jeff KingWhen a worktree file is larger than the available memory, and a clean filter is in use, this avoids mallocing a buffer the whole size of the file when reading from the clean filter, which caused commands like git status and git commit to OOM.
Often in this situation the clean filter will produce a short identifier for the file, so such a large buffer is not needed.
When the clean filter does output something around the same size as the worktree file, the buffer will need to be reallocated until it fits, starting at 8192 and doubling in size. Benchmarking indicates that reallocation is not a significant overhead for outputs up to a few MB in size.
--- convert.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
convert.c +1 -1
@@ -732,7 +732,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
Signature · 1 line
2.20.1
Re: [PATCH] avoid unncessary malloc of whole file size
Quoted 5 lines · Joey Hess
When a worktree file is larger than the available memory, and a clean filter is in use, this avoids mallocing a buffer the whole size of the file when reading from the clean filter, which caused commands like git status and git commit to OOM.
Quoted 8 lines
Often in this situation the clean filter will produce a short identifier for the file, so such a large buffer is not needed.
When the clean filter does output something around the same size as the worktree file, the buffer will need to be reallocated until it fits, starting at 8192 and doubling in size. Benchmarking indicates that reallocation is not a significant overhead for outputs up to a few MB in size.
Problem description first, then solultion. "... this avoids ..." is already talking about solution while forcing the readers to know what the problem is.
When a worktree file is ... filter is in use, we allocate a buffer for the whole size of the file when reading from the clean filter. This can force us to overallocate if the clean filter is used to radically shrink a huge file and replace it with a small token (e.g. git-annex or git-lfs) and lead to OOM at the worst case. Reading from the filter and growing the buffer as we go would avoid such an unnecessary OOM.
When the clean filter does output ...
... few MB in size.
perhaps.
Quoted 18 lines
Signed-off-by Joey Hess <[email protected]>--- convert.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)convert.c +1
-1@@ -732,7 +732,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
Re: [PATCH] avoid unncessary malloc of whole file size
Quoted 28 lines · Junio C Hamano
Quoted 5 lines · Joey Hess
When a worktree file is larger than the available memory, and a clean filter is in use, this avoids mallocing a buffer the whole size of the file when reading from the clean filter, which caused commands like git status and git commit to OOM.
Quoted 8 lines
Often in this situation the clean filter will produce a short identifier for the file, so such a large buffer is not needed.
When the clean filter does output something around the same size as the worktree file, the buffer will need to be reallocated until it fits, starting at 8192 and doubling in size. Benchmarking indicates that reallocation is not a significant overhead for outputs up to a few MB in size.
Problem description first, then solultion. "... this avoids ..." is already talking about solution while forcing the readers to know what the problem is.
When a worktree file is ... filter is in use, we allocate a buffer for the whole size of the file when reading from the clean filter. This can force us to overallocate if the clean filter is used to radically shrink a huge file and replace it with a small token (e.g. git-annex or git-lfs) and lead to OOM at the worst case. Reading from the filter and growing the buffer as we go would avoid such an unnecessary OOM.
Quoted 4 lines
When the clean filter does output ...
... few MB in size.perhaps.
Yeah, I agree that organization is nicer. Other than that, the patch looks good to me.
-Peff