Skip to contentSkip to conversation

fast-import: Stream very large blobs directly to pack

PATCH In review Shawn O. Pearce 6 participants 27 messages Jun 15, 2016 Documentation

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fast-import: Stream very large blobs directly to pack

If a blob is larger than the configured big-file-threshold, instead of reading it into a single buffer obtained from malloc, stream it onto the end of the current pack file. Streaming the larger objects into the pack avoids the 4+ GiB memory footprint that occurs when fast-import is processing 2+ GiB blobs.

---
 Documentation/git-fast-import.txt |    6 ++
 fast-import.c                     |  174 +++++++++++++++++++++++++++++++++----
 t/t5705-clone-2gb.sh              |    2 +-
 t/t9300-fast-import.sh            |   46 ++++++++++
 4 files changed, 210 insertions(+), 18 deletions(-)
Documentation/git-fast-import.txt +6 -0
index e6d364f..16d3596 100644
@@ -50,6 +50,12 @@ OPTIONS
 	importers may wish to lower this, such as to ensure the
 	resulting packfiles fit on CDs.
 
+--big-file-threshold=<n>::
+	Maximum size of a blob that fast-import will attempt to
+	create a delta for, expressed in MiB.  The default is 512.
+	Some importers may wish to lower this on systems with
+	constrained memory.
+
 --depth=<n>::
 	Maximum delta depth, for blob and tree deltification.
 	Default is 10.
fast-import.c +157 -17

268 lines not on this page · raw message

t/t5705-clone-2gb.sh +1 -1
index 9f52154..adfaae8 100755
@@ -31,7 +31,7 @@ test_expect_success 'setup' '
 	 echo "data 5" &&
 	 echo ">2gb" &&
 	 cat commit) |
-	git fast-import &&
+	git fast-import --big-file-threshold=2 &&
 	test ! -f exit-status
 
 '
t/t9300-fast-import.sh +46 -0

50 lines not on this page · raw message

Signature · 4 lines
1.7.0.rc0.170.g7207c

-- 
Shawn.
#1 lore raw

fast-import.c static int store_object( · line 1174

Quoted 4 lines · Shawn O. Pearce
fast-import.c +4 -0
+static void stream_blob(
+	uintmax_t len,
+	unsigned char *sha1out,
+	uintmax_t mark)

A funny way to indent and line wrap...

fast-import.c static int store_object( · line 1190

Quoted 6 lines
fast-import.c +6 -0
+{
+ ...
+	/* Determine if we should auto-checkpoint. */
+	if ((pack_size + 60 + len) > max_packsize
+		|| (pack_size + 60 + len) < pack_size)
+		cycle_packfile();

What's "60" in this math?

If the data is not compressible, we could even grow and the end result might be more than (pack_size + len), busting max_packsize. As we are streaming out, we cannot say "oops, let me try again after truncating and closing the current file and then opening a new file", and instead may have to copy the data from the current one to a new one, and truncate the current one. Is this something worth worrying about?

#2 lore raw
Quoted 8 lines · Junio C Hamano

fast-import.c static int store_object( · line 1174

Quoted 4 lines · Shawn O. Pearce
fast-import.c +4 -0
+static void stream_blob(
+	uintmax_t len,
+	unsigned char *sha1out,
+	uintmax_t mark)

A funny way to indent and line wrap...

Consistent with something else in the same file that has a similar task... store_object(). I literally just copied the lines from there and pasted here.

Quoted 6 lines

fast-import.c static int store_object( · line 1190

Quoted 4 lines
fast-import.c +4 -0
+	/* Determine if we should auto-checkpoint. */
+	if ((pack_size + 60 + len) > max_packsize
+		|| (pack_size + 60 + len) < pack_size)
+		cycle_packfile();

What's "60" in this math?

IIRC 60 is 20 bytes for the SHA-1 footer, and another 40 padding for the object header, and the deflate() wrapping.

Again, the 60 was stolen from the existing store_object(), which already has the same assumption. Only there I think we have len as the fully compressed version, so the deflate() wrapping is already being accounted for.

Quoted 2 lines

If the data is not compressible, we could even grow and the end result might be more than (pack_size + len), busting max_packsize.

Yea, that's a good point. I probably should run the length through deflateBound() and use that here in the test. But I didn't think it would make a huge difference. If the file isn't compressible what is the real increment over the uncompressed size? There's a header and footer, and instructions in the stream to indicate literal data, but its not like its doubling in size.

Quoted 5 lines

As we are
streaming out, we cannot say "oops, let me try again after truncating and closing the current file and then opening a new file", and instead may have to copy the data from the current one to a new one, and truncate the current one. Is this something worth worrying about?

Hmm. I'm not that worried about it, but then there's the case of a blob that is larger than the max pack size. We can't store that, period. Should we try to exceed max pack size for that one object? Or should we refuse?

The major reason for this test is to ensure an object offset starts before the 4 GiB boundary so idx v1 can hold the offset. A much less important reason is to try to support older 32 bit filesystems which can't do more than 2 GiB per file.

Signature · 1 line
Shawn.
#3 lore raw

PATCH v2 1/1 fast-import: Stream very large blobs directly to pack

If a blob is larger than the configured big-file-threshold, instead of reading it into a single buffer obtained from malloc, stream it onto the end of the current pack file. Streaming the larger objects into the pack avoids the 4+ GiB memory footprint that occurs when fast-import is processing 2+ GiB blobs.

---

Fixed a missing skip_optional_lf() at the end of the large blob command case.

 Documentation/git-fast-import.txt |    6 ++
 fast-import.c                     |  172 +++++++++++++++++++++++++++++++++----
 t/t5705-clone-2gb.sh              |    2 +-
 t/t9300-fast-import.sh            |   46 ++++++++++
 4 files changed, 208 insertions(+), 18 deletions(-)
Documentation/git-fast-import.txt +6 -0
index e6d364f..16d3596 100644
@@ -50,6 +50,12 @@ OPTIONS
 	importers may wish to lower this, such as to ensure the
 	resulting packfiles fit on CDs.
 
+--big-file-threshold=<n>::
+	Maximum size of a blob that fast-import will attempt to
+	create a delta for, expressed in MiB.  The default is 512.
+	Some importers may wish to lower this on systems with
+	constrained memory.
+
 --depth=<n>::
 	Maximum delta depth, for blob and tree deltification.
 	Default is 10.
fast-import.c +155 -17

266 lines not on this page · raw message

t/t5705-clone-2gb.sh +1 -1
index 9f52154..adfaae8 100755
@@ -31,7 +31,7 @@ test_expect_success 'setup' '
 	 echo "data 5" &&
 	 echo ">2gb" &&
 	 cat commit) |
-	git fast-import &&
+	git fast-import --big-file-threshold=2 &&
 	test ! -f exit-status
 
 '
t/t9300-fast-import.sh +46 -0

50 lines not on this page · raw message

Signature · 4 lines
1.7.0.rc0.170.g7207c

-- 
Shawn.
#4 lore raw

Documentation/git-fast-import.txt OPTIONS · line 58

Quoted 12 lines · Shawn O. Pearce
 Documentation/git-fast-import.txt |    6 ++
 fast-import.c                     |  172 +++++++++++++++++++++++++++++++++----
 t/t5705-clone-2gb.sh              |    2 +-
 t/t9300-fast-import.sh            |   46 ++++++++++
 4 files changed, 208 insertions(+), 18 deletions(-)

+--big-file-threshold=<n>::

Documentation/git-fast-import.txt +5 -0
+	Maximum size of a blob that fast-import will attempt to
+	create a delta for, expressed in MiB.  The default is 512.
+	Some importers may wish to lower this on systems with
+	constrained memory.
+

Shouldn't there be config variable corresponding to this command line option, so that it can be set once for [constrained] system?

Signature · 3 lines
Jakub Narebski
Poland
ShadeHawk on #git
#5 lore raw
Quoted 8 lines · Jakub Narebski
Quoted 5 lines · Shawn O. Pearce
 Documentation/git-fast-import.txt |    6 ++
 fast-import.c                     |  172 +++++++++++++++++++++++++++++++++----
 t/t5705-clone-2gb.sh              |    2 +-
 t/t9300-fast-import.sh            |   46 ++++++++++
 4 files changed, 208 insertions(+), 18 deletions(-)
Quoted 9 lines

Documentation/git-fast-import.txt OPTIONS · line 58

Quoted 6 lines

+--big-file-threshold=<n>::

Documentation/git-fast-import.txt +5 -0
+	Maximum size of a blob that fast-import will attempt to
+	create a delta for, expressed in MiB.  The default is 512.
+	Some importers may wish to lower this on systems with
+	constrained memory.
+

Shouldn't there be config variable corresponding to this command line option, so that it can be set once for [constrained] system?

Implemented as core.bigFileThreshold in this patch... but I didn't document it...

Signature · 1 line
Shawn.
#6 lore raw

Shawn O. Pearce wrote:
...

Quoted 2 lines

Implemented as core.bigFileThreshold in this patch... but I didn't document it...

Bad dog! No biscuit!

#7 lore raw
Quoted 6 lines · A Large Angry SCM

Shawn O. Pearce wrote:
...

Quoted 2 lines

Implemented as core.bigFileThreshold in this patch... but I didn't document it...

Bad dog! No biscuit!

:-)

Patches welcome.

#8 lore raw
In reply to Shawn O. Pearce
Quoted 5 lines · Shawn O. Pearce

If a blob is larger than the configured big-file-threshold, instead of reading it into a single buffer obtained from malloc, stream it onto the end of the current pack file. Streaming the larger objects into the pack avoids the 4+ GiB memory footprint that occurs when fast-import is processing 2+ GiB blobs.

Yeah. I've had that item on my todo list for ages now. This big-file-threshold principle has to be applied to 'git add' too so a big blob is stored in pack file form right away, and used to bypass delta searching in 'git pack-objects', used to skip the diff machinery, and so on.

Nicolas

#9 lore raw
Quoted 8 lines · Nicolas Pitre
Quoted 5 lines · Shawn O. Pearce

If a blob is larger than the configured big-file-threshold, instead of reading it into a single buffer obtained from malloc, stream it onto the end of the current pack file. Streaming the larger objects into the pack avoids the 4+ GiB memory footprint that occurs when fast-import is processing 2+ GiB blobs.

Quoted 5 lines

Yeah. I've had that item on my todo list for ages now. This big-file-threshold principle has to be applied to 'git add' too so a big blob is stored in pack file form right away, and used to bypass delta searching in 'git pack-objects', used to skip the diff machinery, and so on.

Yea, there are a lot of places we should improve for bigger files. gfi just happened to be the first one I got a bug report on from a user...

Signature · 1 line
Shawn.
#10 lore raw
In reply to Shawn O. Pearce

Heya,

Quoted 1 line · Shawn O. Pearce

index 60d0aa2..8114f93 100644

Looks like you based it off an old version of fast-import, at least not on what is in master atm, since that has my fast-import series. Rebasing shouldn't be too hard, you'll just have to move the option parsing to the appropriate function.

Signature · 3 lines
Cheers,

Sverre Rabbelier
#11 lore raw
Quoted 7 lines · Sverre Rabbelier
Quoted 1 line · Shawn O. Pearce

index 60d0aa2..8114f93 100644

Looks like you based it off an old version of fast-import, at least not on what is in master atm, since that has my fast-import series. Rebasing shouldn't be too hard, you'll just have to move the option parsing to the appropriate function.

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

But yea... I guess I also owe Junio a rebased form for master so he has less merge conflicts to resolve.

Signature · 1 line
Shawn.
#12 lore raw

Heya,

Quoted 2 lines · Shawn O. Pearce

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

Ah, sorry, did not notice that.

Speaking of which, how does one figure out what commit a patch is/could be based on? Couldn't we include that in the part behind the triple dash?

Signature · 3 lines
Cheers,

Sverre Rabbelier
#13 lore raw
Quoted 9 lines · Sverre Rabbelier
Quoted 2 lines · Shawn O. Pearce

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

Ah, sorry, did not notice that.

Speaking of which, how does one figure out what commit a patch is/could be based on? Couldn't we include that in the part behind the triple dash?

You have to tell the other guy in your cover letter section. You can try to guess if the index information is accurate by grepping around a log output looking for that blob sha-1 and then finding what branch that's in. Its not pretty.

IIRC other DVCS systems include the ancestor identity in their patch output. Git doesn't bother, because in a multi-patch series the 2nd (and subsequent) patches would have useless id information.

Signature · 1 line
Shawn.
#14 lore raw
In reply to Shawn O. Pearce
Quoted 11 lines · Shawn O. Pearce
Quoted 7 lines · Sverre Rabbelier
Quoted 1 line · Shawn O. Pearce

index 60d0aa2..8114f93 100644

Looks like you based it off an old version of fast-import, at least not on what is in master atm, since that has my fast-import series. Rebasing shouldn't be too hard, you'll just have to move the option parsing to the appropriate function.

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

I personally see that as adding a new feature (especially with new option and config).

Quoted 2 lines

But yea... I guess I also owe Junio a rebased form for master so he has less merge conflicts to resolve.

Thanks.

#15 lore raw
Quoted 7 lines · Junio C Hamano
Quoted 2 lines · Shawn O. Pearce

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

I personally see that as adding a new feature (especially with new option and config).

Sorry, but I take it back. The new codepath triggers even without any explicit request and _fixes_ the situation where old code simply failed, so it is worth queuing for the maintenance track.

Do you want to do the deflatebound thing, or are we Ok without?

#16 lore raw

I'll queued this with a result of my conflict resolution to 'pu' for now but please double check after I push it out.

You may want to add the new option to the output from "cmd -h" and probably description of the configuration in the doc before any of this gets official.

#17 lore raw
Quoted 2 lines · Junio C Hamano

I'll queued this with a result of my conflict resolution to 'pu' for now but please double check after I push it out.

The strtoumax call got messed up. Squash this into your merge:

fast-import.c +1 -1
index e6ebcf6..9c65a24 100644
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {
Quoted 3 lines

You may want to add the new option to the output from "cmd -h" and probably description of the configuration in the doc before any of this gets official.

I'll send an additional patch in a minute with these documentation related updates.

Signature · 1 line
Shawn.
#18 lore raw
Quoted 11 lines · Shawn O. Pearce
Quoted 2 lines · Junio C Hamano

I'll queued this with a result of my conflict resolution to 'pu' for now but please double check after I push it out.

The strtoumax call got messed up. Squash this into your merge:

fast-import.c +0 -0
index e6ebcf6..9c65a24 100644

fast-import.c static int parse_one_option(const char *option) · line 2806

Quoted 8 lines
fast-import.c +1 -1
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {

Ah, of course. Sorry for not being careful and thanks for pointing it out.

#19 lore raw
In reply to Shawn O. Pearce
Quoted 16 lines · Shawn O. Pearce

The strtoumax call got messed up. Squash this into your merge:

fast-import.c +1 -1
index e6ebcf6..9c65a24 100644
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {
Quoted 6 lines
Quoted 3 lines

You may want to add the new option to the output from "cmd -h" and probably description of the configuration in the doc before any of this gets official.

I'll send an additional patch in a minute with these documentation related updates.

Well, well, well....

The documentation says this is counted in bytes, but somehow neither of us found the above " * 1024 * 1024" suspicious.

Shouldn't it be at least like this? It would probably be a good idea to use git_parse_ulong() or somesuch while we are at it.

 fast-import.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
fast-import.c +1 -1
index ca21082..ea1ac0f 100644
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0);
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {
#20 lore raw
Quoted 17 lines · Junio C Hamano

Well, well, well....

The documentation says this is counted in bytes, but somehow neither of us found the above " * 1024 * 1024" suspicious.

Shouldn't it be at least like this? It would probably be a good idea to use git_parse_ulong() or somesuch while we are at it.

 fast-import.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
fast-import.c +0 -0
index ca21082..ea1ac0f 100644

fast-import.c static int parse_one_option(const char *option) · line 2803

Quoted 5 lines
fast-import.c +1 -1
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0);

In my v3 patch I thought I replaced this code with:

+ else if (!prefixcmp(a, "--big-file-threshold=")) {
+ unsigned long v;
+ if (!git_parse_ulong(a + 21, &v))
+ usage(fast_import_usage);
+ big_file_threshold = v;

So we relied on git_parse_ulong to handle unit suffixes as well.

Signature · 1 line
Shawn.
#21 lore raw
Quoted 9 lines · Shawn O. Pearce

In my v3 patch I thought I replaced this code with:

(quoted diff) +5 -0
+               else if (!prefixcmp(a, "--big-file-threshold=")) {
+                       unsigned long v;
+                       if (!git_parse_ulong(a + 21, &v))
+                               usage(fast_import_usage);
+                       big_file_threshold = v;

So we relied on git_parse_ulong to handle unit suffixes as well.

Yeah, you did; but it didn't carried through the merge process across the code restructure to add "option_blah" stuff. Sorry about that.

Looking at the output from

    $ git grep -n -e ' \* 1024 \* 1024' -- '*.c'

I noticed another issue. Don't we need the same thing for max_packsize? Or is that _too much_ of a backward incompatible change that we should wait until 1.7.1?

#22 lore raw
Quoted 10 lines · Junio C Hamano
Quoted 8 lines · Shawn O. Pearce

In my v3 patch I thought I replaced this code with:

(quoted diff) +5 -0
+               else if (!prefixcmp(a, "--big-file-threshold=")) {
+                       unsigned long v;
+                       if (!git_parse_ulong(a + 21, &v))
+                               usage(fast_import_usage);
+                       big_file_threshold = v;
Quoted 4 lines
Quoted 1 line

So we relied on git_parse_ulong to handle unit suffixes as well.

Yeah, you did; but it didn't carried through the merge process across the code restructure to add "option_blah" stuff. Sorry about that.

And this is a fix-up for the mismerge. I didn't touch max-pack-size stuff.

 fast-import.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
fast-import.c +4 -1
index ca21082..a6730d0 100644
@@ -2800,7 +2800,10 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
+		unsigned long v;
+		if (!git_parse_ulong(option + 19, &v))
+			return 0;
+		big_file_threshold = v;
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {
#23 lore raw
Quoted 10 lines · Junio C Hamano

And this is a fix-up for the mismerge. I didn't touch max-pack-size stuff.

 fast-import.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
fast-import.c +0 -0
index ca21082..a6730d0 100644

fast-import.c static int parse_one_option(const char *option) · line 2806

Quoted 8 lines
fast-import.c +4 -1
@@ -2800,7 +2800,10 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
+		unsigned long v;
+		if (!git_parse_ulong(option + 19, &v))
+			return 0;
+		big_file_threshold = v;

Yup, looks good to me.

Signature · 1 line
Shawn.
#24 lore raw
In reply to Junio C Hamano
Quoted 25 lines · Junio C Hamano
Quoted 16 lines · Shawn O. Pearce

The strtoumax call got messed up. Squash this into your merge:

fast-import.c +1 -1
index e6ebcf6..9c65a24 100644
@@ -2800,7 +2800,7 @@ static int parse_one_option(const char *option)
 	if (!prefixcmp(option, "max-pack-size=")) {
 		option_max_pack_size(option + 14);
 	} else if (!prefixcmp(option, "big-file-threshold=")) {
-		big_file_threshold = strtoumax(option + 21, NULL, 0) * 1024 * 1024;
+		big_file_threshold = strtoumax(option + 19, NULL, 0) * 1024 * 1024;
 	} else if (!prefixcmp(option, "depth=")) {
 		option_depth(option + 6);
 	} else if (!prefixcmp(option, "active-branches=")) {
Quoted 6 lines
Quoted 3 lines

You may want to add the new option to the output from "cmd -h" and probably description of the configuration in the doc before any of this gets official.

I'll send an additional patch in a minute with these documentation related updates.

Quoted 7 lines

Well, well, well....

The documentation says this is counted in bytes, but somehow neither of us found the above " * 1024 * 1024" suspicious.

Shouldn't it be at least like this? It would probably be a good idea to use git_parse_ulong() or somesuch while we are at it.

Yes, definitely. I'm about to post a patch moving --max-pack-size in that direction too. I just had to fix a couple other unsuspected issues to get there though. Patches will follow shortly.

Nicolas

#25 lore raw

PATCH 1/1 fast-import: Document the core.bigFileThreshold configuration setting

In reply to Junio C Hamano

---

Quoted 3 lines · Junio C Hamano

You may want to add the new option to the output from "cmd -h" and probably description of the configuration in the doc before any of this gets official.

Doc patch. :-)

Feel free to squash, or to apply as its own change, I don't care either way.

 Documentation/config.txt |   14 ++++++++++++++
 fast-import.c            |    2 +-
 2 files changed, 15 insertions(+), 1 deletions(-)
Documentation/config.txt +14 -0
index f7728ec..b16a20b 100644
@@ -409,6 +409,20 @@ You probably do not need to adjust this value.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.bigFileThreshold::
+	Files larger than this size are stored deflated, without
+	attempting delta compression.  Storing large files without
+	delta compression avoids excessive memory usage, at the
+	slight expense of increased disk usage.
++
+Default is 512 MiB on all platforms.  This should be reasonable
+for most projects as source code and other text files can still
+be delta compressed, but larger binary media files won't be.
++
+Common unit suffixes of 'k', 'm', or 'g' are supported.
++
+Currently only linkgit:git-fast-import[1] honors this setting.
+
 core.excludesfile::
 	In addition to '.gitignore' (per-directory) and
 	'.git/info/exclude', git looks into this file for patterns
fast-import.c +1 -1
index 8055f73..1bcf3bb 100644
@@ -2613,7 +2613,7 @@ static int git_pack_config(const char *k, const char *v, void *cb)
 }
 
 static const char fast_import_usage[] =
-"git fast-import [--date-format=f] [--max-pack-size=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]";
+"git fast-import [--date-format=f] [--max-pack-size=n] [--big-file-threshold=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]";
 
 int main(int argc, const char **argv)
 {
Signature · 4 lines
1.7.0.rc0.170.g7207c

-- 
Shawn.
#26 lore raw
In reply to Junio C Hamano
Quoted 6 lines · Junio C Hamano
Quoted 5 lines · Junio C Hamano
Quoted 2 lines · Shawn O. Pearce

I was intentionally slating this for maint, to fix a bug a user reported when handling large streams.

Quoted 6 lines
Quoted 2 lines

I personally see that as adding a new feature (especially with new option and config).

Sorry, but I take it back. The new codepath triggers even without any explicit request and _fixes_ the situation where old code simply failed, so it is worth queuing for the maintenance track.

Thanks.

Quoted 1 line

Do you want to do the deflatebound thing, or are we Ok without?

No, I think we're OK without it. The pack size limit stuff is only a rough guess anyway. I never meant for us to be sticking to it to within a byte. It was only meant to keep us from shoving a 100 MiB blob onto the end of a pack once we got close to the size requested by the user.

Signature · 1 line
Shawn.