Skip to content

nd/pack-objects-pack-struct updates

Nguyễn Thái Ngọc Duy git 8 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v8 Latest15 patches 4 repliesIn review
  2. v713 patches 27 repliesSuperseded
  3. v611 patches 38 repliesSuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v511 patches8 repliesSuperseded
v411 patches10 repliesSuperseded
v312 patches6 repliesSuperseded
v29 patches1 replySuperseded
v111 patches9 repliesSuperseded

Changes in v8 from the cover letter

- prefer BUG() over die()
- do "1U <<" instead of "1 << " to avoid undefined behavior with signed shifting.
- add more comments based on Jeff's feedback
- plug a leak in try_delta() when delta_size is too large
- be kind and set depth/cache_max_small_delta_size to max limit instead of dying when the user gives a value over limit
- make travis execute pack-objects uncommon code
- use git_env_*() instead of manually handling getenv() values
- fallback code for when a new pack is added when pack-objects is running
- Compressed cached delta size limit is increased from 64k to 1MB
- Cached delta size limit is decreased from 2G to 1MB

Patches in v8 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/15t/README: mention about running the test suite in special modest/README+10 -00 replies
2/15pack-objects: a bit of document about struct object_entrypack-objects.h+45 -00 replies
3/15pack-objects: turn type and in_pack_type to bitfieldsbuiltin/pack-objects.c cache.h object.h +2 more+58 -310 replies
4/15pack-objects: use bitfield for object_entry::dfs_statebuiltin/pack-objects.c pack-objects.h+20 -110 replies
5/15pack-objects: use bitfield for object_entry::depthDocumentation/config.txt Documentation/git-pack-objects.txt Documentation/git-repack.txt +2 more+15 -50 replies
6/15pack-objects: move in_pack_pos out of struct object_entrybuiltin/pack-objects.c pack-bitmap-write.c pack-bitmap.c +2 more+26 -70 replies
7/15pack-objects: move in_pack out of struct object_entrybuiltin/pack-objects.c cache.h pack-objects.c +3 more+128 -120 replies
8/15pack-objects: refer to delta objects by index instead of pointerbuiltin/pack-objects.c pack-objects.h+124 -590 replies
9/15pack-objects: shrink z_delta_size field in struct object_entryDocumentation/config.txt builtin/pack-objects.c pack-objects.h+22 -80 replies
10/15pack-objects: don't check size when the object is badbuiltin/pack-objects.c+2 -20 replies
11/15pack-objects: clarify the use of object_entry::sizebuiltin/pack-objects.c pack-objects.h+11 -40 replies
12/15pack-objects: shrink size field in struct object_entrybuiltin/pack-objects.c pack-objects.c pack-objects.h +1 more+146 -250 replies
13/15pack-objects: shrink delta_size field in struct object_entrybuiltin/pack-objects.c pack-objects.h+38 -110 replies
14/15pack-objects: reorder members to shrink struct object_entrypack-objects.h+21 -70 replies
15/15ci: exercise the whole test suite with uncommon code in pack-objectsci/run-tests.sh+4 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v8 yet.

Read the v8 thread

Cover letter [PATCH v8 00/15] nd/pack-objects-pack-struct updates

v8 changes

- prefer BUG() over die()
- do "1U <<" instead of "1 << " to avoid undefined behavior with signed shifting.
- add more comments based on Jeff's feedback
- plug a leak in try_delta() when delta_size is too large
- be kind and set depth/cache_max_small_delta_size to max limit instead of dying when the user gives a value over limit
- make travis execute pack-objects uncommon code
- use git_env_*() instead of manually handling getenv() values
- fallback code for when a new pack is added when pack-objects is running
- Compressed cached delta size limit is increased from 64k to 1MB
- Cached delta size limit is decreased from 2G to 1MB

Interdiff

builtin/pack-objects.c +24 -11
index c774821930..b5bba2c228 100644
@@ -1439,7 +1439,7 @@ static void check_object(struct object_entry *entry)
 			goto give_up;
 
 		if (type < 0)
-			die("BUG: invalid type %d", type);
+			BUG("invalid type %d", type);
 		entry->in_pack_type = type;
 
 		/*
@@ -1861,6 +1861,11 @@ static pthread_mutex_t progress_mutex;
 
 #endif
 
+/*
+ * Return the size of the object without doing any delta
+ * reconstruction (so non-deltas are true object sizes, but deltas
+ * return the size of the delta data).
+ */
 unsigned long oe_get_size_slow(struct packing_data *pack,
 			       const struct object_entry *e)
 {
@@ -1881,7 +1886,7 @@ unsigned long oe_get_size_slow(struct packing_data *pack,
 
 	p = oe_in_pack(pack, e);
 	if (!p)
-		die("BUG: when e->type is a delta, it must belong to a pack");
+		BUG("when e->type is a delta, it must belong to a pack");
 
 	read_lock();
 	w_curs = NULL;
@@ -2006,8 +2011,10 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,
 	delta_buf = create_delta(src->index, trg->data, trg_size, &delta_size, max_size);
 	if (!delta_buf)
 		return 0;
-	if (delta_size >= (1 << OE_DELTA_SIZE_BITS))
+	if (delta_size >= (1U << OE_DELTA_SIZE_BITS)) {
+		free(delta_buf);
 		return 0;
+	}
 
 	if (DELTA(trg_entry)) {
 		/* Prefer only shallower same-sized deltas. */
@@ -2163,7 +2170,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,
 			unsigned long size;
 
 			size = do_compress(&entry->delta_data, DELTA_SIZE(entry));
-			if (size < (1 << OE_Z_DELTA_BITS)) {
+			if (size < (1U << OE_Z_DELTA_BITS)) {
 				entry->z_delta_size = size;
 				cache_lock();
 				delta_cache_size -= DELTA_SIZE(entry);
@@ -3131,7 +3138,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	};
 
 	if (DFS_NUM_STATES > (1 << OE_DFS_STATE_BITS))
-		die("BUG: too many dfs states, increase OE_DFS_STATE_BITS");
+		BUG("too many dfs states, increase OE_DFS_STATE_BITS");
 
 	check_replace_refs = 0;
 
@@ -3149,12 +3156,16 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	if (pack_to_stdout != !base_name || argc)
 		usage_with_options(pack_usage, pack_objects_options);
 
-	if (depth >= (1 << OE_DEPTH_BITS))
-		die(_("delta chain depth %d is greater than maximum limit %d"),
-		    depth, (1 << OE_DEPTH_BITS) - 1);
-	if (cache_max_small_delta_size >= (1 << OE_Z_DELTA_BITS))
-		die(_("pack.deltaCacheLimit is greater than maximum limit %d"),
-		    (1 << OE_Z_DELTA_BITS) - 1);
+	if (depth >= (1 << OE_DEPTH_BITS)) {
+		warning(_("delta chain depth %d is too deep, forcing %d"),
+			depth, (1 << OE_DEPTH_BITS) - 1);
+		depth = (1 << OE_DEPTH_BITS) - 1;
+	}
+	if (cache_max_small_delta_size >= (1U << OE_Z_DELTA_BITS)) {
+		warning(_("pack.deltaCacheLimit is too high, forcing %d"),
+			(1U << OE_Z_DELTA_BITS) - 1);
+		cache_max_small_delta_size = (1U << OE_Z_DELTA_BITS) - 1;
+	}
 
 	argv_array_push(&rp, "pack-objects");
 	if (thin) {
@@ -3274,6 +3285,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		}
 	}
 
+	prepare_packing_data(&to_pack);
+
 	if (progress)
 		progress_state = start_progress(_("Counting objects"), 0);
 	if (!use_internal_rev_list)
ci/run-tests.sh +4 -1
index 73e273fac7..857d144ee8 100755
@@ -10,7 +10,10 @@ ln -s "$cache_dir/.prove" t/.prove
 make --quiet test
 if test "$jobname" = "linux-gcc"
 then
-	GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test
+	export GIT_TEST_SPLIT_INDEX=YesPlease
+	export GIT_TEST_FULL_IN_PACK_ARRAY=true
+	export GIT_TEST_OE_SIZE=10
+	make --quiet test
 fi
 
 check_unignored_build_artifacts
pack-objects.c +45 -19

90 lines not on this page · raw message

pack-objects.h +29 -17

115 lines not on this page · raw message

t/README +16 -6
index 02bfb3fed5..c01d210c15 100644
@@ -291,16 +291,26 @@ expect the rest to function correctly.
 and know what setup is needed for it.  Or when you want to run
 everything up to a certain test.
 
+
+Running tests with special setups
+---------------------------------
+
+The whole test suite could be run to test some special features
+that cannot be easily covered by a few specific test cases. These
+could be enabled by running the test suite with correct GIT_TEST_
+environment set.
+
+GIT_TEST_SPLIT_INDEX forces split-index mode on the whole test suite.
+
 GIT_TEST_FULL_IN_PACK_ARRAY exercises the uncommon pack-objects code
 path where there are more than 1024 packs even if the actual number of
 packs in repository is below this limit.
 
-GIT_TEST_OE_SIZE_BITS=<bits> exercises the uncommon pack-objects
-code path where we do not cache objecct size in memory and read it
-from existing packs on demand. This normally only happens when the
-object size is over 2GB. This variable forces the code path on any
-object larger than 2^<bits> bytes.
-
+GIT_TEST_OE_SIZE=<n> exercises the uncommon pack-objects code path
+where we do not cache object size in memory and read it from existing
+packs on demand. This normally only happens when the object size is
+over 2GB. This variable forces the code path on any object larger than
+<n> bytes.
 
 Naming Tests
 ------------

Nguyễn Thái Ngọc Duy (15):
t/README: mention about running the test suite in special modes
pack-objects: a bit of document about struct object_entry
pack-objects: turn type and in_pack_type to bitfields
pack-objects: use bitfield for object_entry::dfs_state
pack-objects: use bitfield for object_entry::depth
pack-objects: move in_pack_pos out of struct object_entry
pack-objects: move in_pack out of struct object_entry
pack-objects: refer to delta objects by index instead of pointer
pack-objects: shrink z_delta_size field in struct object_entry
pack-objects: don't check size when the object is bad
pack-objects: clarify the use of object_entry::size
pack-objects: shrink size field in struct object_entry
pack-objects: shrink delta_size field in struct object_entry
pack-objects: reorder members to shrink struct object_entry
ci: exercise the whole test suite with uncommon code in pack-objects

 Documentation/config.txt           |   4 +-
 Documentation/git-pack-objects.txt |   4 +-
 Documentation/git-repack.txt       |   4 +-
 builtin/pack-objects.c             | 366 +++++++++++++++++++----------
 cache.h                            |   3 +
 ci/run-tests.sh                    |   5 +-
 object.h                           |   1 -
 pack-bitmap-write.c                |  14 +-
 pack-bitmap.c                      |   2 +-
 pack-bitmap.h                      |   4 +-
 pack-objects.c                     |  69 ++++++
 pack-objects.h                     | 312 ++++++++++++++++++++++--
 t/README                           |  20 ++
 t/t5300-pack-object.sh             |   5 +
 14 files changed, 650 insertions(+), 163 deletions(-)
Signature · 1 line
2.17.0.rc2.515.g4feb9b7923