convert: avoid malloc of original file size
PATCH 1/1 convert: avoid malloc of original file size
From: Joey Hess <[email protected]>
We write the output of a "clean" filter into a strbuf. Rather than growing the strbuf dynamically as we read its output, we make the initial allocation as large as the original input file. This is a good guess when the filter is just tweaking a few bytes, but it's disastrous when the point of the filter is to condense a very large file into a short identifier (e.g., the way git-lfs and git-annex do). We may ask to allocate many gigabytes, causing the allocation to fail and Git to die().
Instead, let's just let strbuf do its usual growth.
When the clean filter does output something around the same size as the worktree file, the buffer will need to be reallocated until it fits, starting at 8192 and doubling in size. Benchmarking indicates that reallocation is not a significant overhead for outputs up to a few MB in size.
---
This is a resurrection of the patch from:
https://public-inbox.org/git/[email protected]/
It got stalled on discussion of the commit message, which I've rewritten here to match the suggestions in the thread.
As discussed there, I do think this only solves half the problem, as the smudge filter has the same issue in reverse. That's more complicated to fix, and AFAIK nobody is working on it. But I don't think there's any reason not to pick up this part in the meantime.
convert.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
convert.c +1 -1
@@ -731,7 +731,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
Signature · 1 line
2.21.0.787.g929e938557
Quoted 4 lines · Jeff King
As discussed there, I do think this only solves half the problem, as the smudge filter has the same issue in reverse. That's more complicated to fix, and AFAIK nobody is working on it. But I don't think there's any reason not to pick up this part in the meantime.
Yeah, I agree that the reverse direction shares the same issue.
I am not sure 0 is a good initial value in this direction, either; I'd rather clip to min(len, core.bigfilethreshold) or something like that, to avoid regressing the more normal use cases.
But let's queue this and see what happens.
Thanks.
Quoted 17 lines
convert.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)convert.c +1
-1@@ -731,7 +731,7 @@ static int apply_single_file_filter(const char *path, const char *src, size_t leif (start_async (&async))return 0 ;/* error was already reported */ -if (strbuf_read (&nbuf, async.out, len) <0 ) { +if (strbuf_read (&nbuf, async.out,0 ) <0 ) { err =error (_ ("read from external filter '%s' failed" ), cmd); }if (close (async.out)) {
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Jeff King
As discussed there, I do think this only solves half the problem, as the smudge filter has the same issue in reverse. That's more complicated to fix, and AFAIK nobody is working on it. But I don't think there's any reason not to pick up this part in the meantime.
Quoted 5 lines
Yeah, I agree that the reverse direction shares the same issue.
I am not sure 0 is a good initial value in this direction, either; I'd rather clip to
min(len, core.bigfilethreshold)or something like that, to avoid regressing the more normal use cases.
That was my initial thought, too, but Joey's benchmarks show that it doesn't seem to make a big difference either way. In his numbers it did get measurable for a 1GB file, but we'd still not use "hint == len" in that case (we'd probably do one or two doublings to get there).
I also think running a real (non-condensing) filter on a 1GB file is already a pretty unlikely corner case.
Quoted 1 line
But let's queue this and see what happens.
Sounds good to me.
-Peff