Multi-pack-index (MIDX)
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v4 Latest23 patches 7 repliesIn review
- v324 patches 25 repliesSuperseded
- v224 patches 10 repliesSuperseded
Earlier versions · 1
| Version | Posted | Patches | Replies | Status |
|---|---|---|---|---|
| v1 | 23 patches | 51 replies | Superseded |
Patches in v4 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/23 | multi-pack-index: add design documentDocumentation/ | +109 | 0 replies | |
| 2/23 | multi-pack-index: add format detailsDocumentation/ | +49 | 0 replies | |
| 3/23 | multi-pack-index: add builtin.gitignore Documentation/ Makefile +4 more | +76 | 3 replies | |
| 4/23 | multi-pack-index: add 'write' verbDocumentation/ Makefile builtin/ +3 more | +60 | 1 reply | |
| 5/23 | midx: write header information to lockfilemidx.c t/ | +53 | 0 replies | |
| 6/23 | multi-pack-index: load into memoryMakefile midx.c midx.h +5 more | +143 | 0 replies | |
| 7/23 | t5319: expand test datat/ | +84 | 0 replies | |
| 8/23 | packfile: generalize pack directory listpackfile.c packfile.h | +69 | 0 replies | |
| 9/23 | multi-pack-index: read packfile listmidx.c t/ | +55 | 0 replies | |
| 10/23 | multi-pack-index: write pack names in chunkDocumentation/ midx.c midx.h +2 more | +189 | 0 replies | |
| 11/23 | midx: read pack names into arraymidx.c midx.h t/ +1 more | +35 | 0 replies | |
| 12/23 | midx: sort and deduplicate objects from packfilesmidx.c packfile.c packfile.h | +147 | 0 replies | |
| 13/23 | midx: write object ids in a chunkDocumentation/ midx.c midx.h +2 more | +53 | 0 replies | |
| 14/23 | midx: write object id fanout chunkDocumentation/ midx.c midx.h +2 more | +68 | 0 replies | |
| 15/23 | midx: write object offsetsDocumentation/ midx.c midx.h +2 more | +155 | 0 replies | |
| 16/23 | config: create core.multiPackIndex settingDocumentation/ midx.c midx.h +3 more | +85 | 2 replies | |
| 17/23 | midx: read objects from multi-pack-indexmidx.c midx.h packfile.c | +100 | 0 replies | |
| 18/23 | midx: use midx in abbreviation calculationsmidx.c midx.h packfile.c +2 more | +91 | 0 replies | |
| 19/23 | midx: use existing midx when writing new onemidx.c midx.h | +111 | 0 replies | |
| 20/23 | midx: use midx in approximate_object_countpackfile.c | +3 | 0 replies | |
| 21/23 | midx: prevent duplicate packfile loadspackfile.c | +9 | 0 replies | |
| 22/23 | packfile: skip loading index if in multi-pack-indexpackfile.c | +17 | 0 replies | |
| 23/23 | midx: clear midx on repackbuiltin/ midx.c midx.h +1 more | +31 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.
Cover letter [PATCH v4 00/23] Multi-pack-index (MIDX)
v3 had a lot of interesting feedback, most of which was non-functional, but made a big impact on the shape of the patch, especially the test script.
These are the important changes:
* 'git multi-pack-index' will report usage if the 'write' verb is not provided, or if extra parameters are provided. A later series will create the 'verify' verb.
* t5319-multi-pack-index.sh has a reoganized way to generate object data, so it has fewer code clones.
* 'test-tool read-midx' uses '-' instead of '_'.
* The global 'core_multi_pack_index' is replaced with a one-time call to git_config_bool() per repository that loads a multi-pack-index.
* 'struct multi_pack_index' is now defined in midx.h and kept anonymous to object-store.h.
* Added a test that 'git repack' removes the multi-pack-index.
* Fixed a doc bug when linking to the technical docs.
I included the diff between the latest ds/multi-pack-index and this series as part of this message.
You can see the CI builds for Linux, Mac, and Windows linked from the GitHub pull request [1].
Thanks,
-Stolee
[1] https://github.com/gitgitgadget/git/pull/5
Derrick Stolee (23):
multi-pack-index: add design document
multi-pack-index: add format details
multi-pack-index: add builtin
multi-pack-index: add 'write' verb
midx: write header information to lockfile
multi-pack-index: load into memory
t5319: expand test data
packfile: generalize pack directory list
multi-pack-index: read packfile list
multi-pack-index: write pack names in chunk
midx: read pack names into array
midx: sort and deduplicate objects from packfiles
midx: write object ids in a chunk
midx: write object id fanout chunk
midx: write object offsets
config: create core.multiPackIndex setting
midx: read objects from multi-pack-index
midx: use midx in abbreviation calculations
midx: use existing midx when writing new one
midx: use midx in approximate_object_count
midx: prevent duplicate packfile loads
packfile: skip loading index if in multi-pack-index
midx: clear midx on repack
.gitignore | 3 +- Documentation/config.txt | 5 + Documentation/git-multi-pack-index.txt | 56 ++ Documentation/technical/multi-pack-index.txt | 109 +++ Documentation/technical/pack-format.txt | 77 ++ Makefile | 3 + builtin.h | 1 + builtin/multi-pack-index.c | 47 + builtin/repack.c | 9 + command-list.txt | 1 + git.c | 1 + midx.c | 918 +++++++++++++++++++ midx.h | 44 + object-store.h | 9 + packfile.c | 169 +++- packfile.h | 9 + sha1-name.c | 70 ++ t/helper/test-read-midx.c | 51 ++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5319-multi-pack-index.sh | 179 ++++ 21 files changed, 1720 insertions(+), 43 deletions(-) create mode 100644 Documentation/git-multi-pack-index.txt create mode 100644 Documentation/technical/multi-pack-index.txt create mode 100644 builtin/multi-pack-index.c create mode 100644 midx.c create mode 100644 midx.h create mode 100644 t/helper/test-read-midx.c create mode 100755 t/t5319-multi-pack-index.sh
base-commit: 53f9a3e157dbbc901a02ac2c73346d375e24978c
Signature · 1 line
2.18.0.118.gd4f65b8d14
Documentation/config.txt +2 -1
@@ -910,7 +910,8 @@ core.commitGraph:: core.multiPackIndex:: Use the multi-pack-index file to track multiple packfiles using a - single index. See link:technical/multi-pack-index[1]. + single index. See link:technical/multi-pack-index.html[the + multi-pack-index design document]. core.sparseCheckout:: Enable "sparse checkout" feature. See section "Sparse checkout" in
Documentation/git-multi-pack-index.txt +3 -3
@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes SYNOPSIS -------- [verse] -'git multi-pack-index' [--object-dir <dir>] <verb> +'git multi-pack-index' [--object-dir=<dir>] <verb> DESCRIPTION -----------
@@ -18,7 +18,7 @@ Write or verify a multi-pack-index (MIDX) file. OPTIONS ------- ---object-dir <dir>:: +--object-dir=<dir>:: Use given directory for the location of Git objects. We check `<dir>/packs/multi-pack-index` for the current MIDX file, and `<dir>/packs` for the pack-files to index.
@@ -37,7 +37,7 @@ EXAMPLES $ git multi-pack-index write ----------------------------------------------- -* Write a MIDX file for the packfiles in an alternate. +* Write a MIDX file for the packfiles in an alternate object store. + ----------------------------------------------- $ git multi-pack-index --object-dir <alt> write
builtin/multi-pack-index.c +11 -10
@@ -5,7 +5,7 @@#include "midx.h" static char const *const builtin_multi_pack_index_usage[] = { -N_ ("git multi-pack-index [--object-dir <dir>] [write]" ), +N_ ("git multi-pack-index [--object-dir=<dir>] write" ),NULL };
@@ -18,14 +18,10 @@ int cmd_multi_pack_index(int argc, const char **argv, {static struct option builtin_multi_pack_index_options[] = {OPT_FILENAME (0 ,"object-dir" , &opts.object_dir, -N_ ("The object directory containing set of packfile and pack-index pairs" )), +N_ ("object directory containing set of packfile and pack-index pairs" )),OPT_END (), }; -if (argc ==2 && !strcmp (argv[1 ],"-h" )) -usage_with_options (builtin_multi_pack_index_usage, - builtin_multi_pack_index_options); -git_config (git_default_config,NULL ); argc = parse_options(argc, argv, prefix,
@@ -36,11 +32,16 @@ int cmd_multi_pack_index(int argc, const char **argv, opts.object_dir =get_object_directory ();if (argc ==0 ) -usage_with_options (builtin_multi_pack_index_usage, - builtin_multi_pack_index_options); + goto usage; + +if (!strcmp (argv[0 ],"write" )) { +if (argc >1 ) + goto usage; -if (!strcmp (argv[0 ],"write" ))return write_midx_file (opts.object_dir); + } -return 0 ; +usage: +usage_with_options (builtin_multi_pack_index_usage, + builtin_multi_pack_index_options); }
builtin/repack.c +7 -6
@@ -335,12 +335,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)for_each_string_list_item (item, &names) {for (ext =0 ; ext <ARRAY_SIZE (exts); ext++) {char *fname, *fname_old; - fname =mkpathdup ("%s/pack-%s%s" , packdir, - item->string, exts[ext].name); -if (!file_exists (fname)) { -free (fname); -continue ; - }if (!midx_cleared) {/* if we move a packfile, it will invalidated the midx */
@@ -348,6 +342,13 @@ int cmd_repack(int argc, const char **argv, const char *prefix) midx_cleared =1 ; } + fname =mkpathdup ("%s/pack-%s%s" , packdir, + item->string, exts[ext].name); +if (!file_exists (fname)) { +free (fname); +continue ; + } + fname_old =mkpathdup ("%s/old-%s%s" , packdir, item->string, exts[ext].name);if (file_exists (fname_old))
cache.h +0 -1
@@ -814,7 +814,6 @@ extern char *git_replace_ref_base;extern int fsync_object_files;extern int core_preload_index;extern int core_commit_graph; -extern int core_multi_pack_index;extern int core_apply_sparse_checkout;extern int precomposed_unicode;extern int protect_hfs;
config.c +0 -5
@@ -1313,11 +1313,6 @@ static int git_default_core_config(const char *var, const char *value)return 0 ; } -if (!strcmp (var,"core.multipackindex" )) { - core_multi_pack_index =git_config_bool (var, value); -return 0 ; - } -if (!strcmp (var,"core.sparsecheckout" )) { core_apply_sparse_checkout =git_config_bool (var, value);return 0 ;
environment.c +0 -1
@@ -67,7 +67,6 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;char *notes_ref_name;int grafts_replace_parents =1 ;int core_commit_graph; -int core_multi_pack_index;int core_apply_sparse_checkout;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */
midx.c +9 -8
@@ -1,4 +1,5 @@#include "cache.h" +#include "config.h" #include "csum-file.h" #include "dir.h" #include "lockfile.h"
@@ -60,7 +61,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir) midx_size =xsize_t (st.st_size);if (midx_size <MIDX_MIN_SIZE ) { -close (fd);error (_ ("multi-pack-index file %s is too small" ), midx_name); goto cleanup_fail; }
@@ -69,8 +69,7 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir) midx_map =xmmap (NULL , midx_size,PROT_READ ,MAP_PRIVATE , fd,0 ); - m =xcalloc (1 ,sizeof (*m) +strlen (object_dir) +1 ); -strcpy (m->object_dir, object_dir); +FLEX_ALLOC_MEM (m, object_dir, object_dir,strlen (object_dir)); m->fd = fd; m->data = midx_map; m->data_len = midx_size;
@@ -171,7 +170,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir)return m; cleanup_fail: -/* no need to check for NULL when freeing */ free (m);free (midx_name);if (midx_map)
@@ -324,8 +322,10 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir) {struct multi_pack_index *m = r->objects->multi_pack_index;struct multi_pack_index *m_search; +int config_value; -if (!core_multi_pack_index) +if (repo_config_get_bool (r,"core.multipackindex" , &config_value) || + !config_value)return 0 ;for (m_search = m; m_search; m_search = m_search->next)
@@ -382,7 +382,8 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,ALLOC_GROW (packs->names, packs->nr +1 , packs->alloc_names); packs->list[packs->nr] =add_packed_git (full_path, - full_path_len,0 ); + full_path_len, +0 );if (!packs->list[packs->nr]) { warning(_("failed to add packfile '%s'"),
@@ -661,8 +662,8 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,struct pack_midx_entry *next = list;if (oidcmp (&obj->oid, &next->oid) >=0 )BUG ("OIDs not in order: %s >= %s" , -oid_to_hex (&obj->oid), -oid_to_hex (&next->oid)); +oid_to_hex (&obj->oid), +oid_to_hex (&next->oid)); }hashwrite (f, obj->oid.hash, (int )hash_len);
midx.h +25 -1
@@ -3,7 +3,31 @@#include "repository.h" -struct multi_pack_index ; +struct multi_pack_index { +struct multi_pack_index *next; + +int fd; + +const unsigned char *data; +size_t data_len; + +uint32_t signature; +unsigned char version; +unsigned char hash_len; +unsigned char num_chunks; +uint32_t num_packs; +uint32_t num_objects; + +const unsigned char *chunk_pack_names; +const uint32_t *chunk_oid_fanout; +const unsigned char *chunk_oid_lookup; +const unsigned char *chunk_object_offsets; +const unsigned char *chunk_large_offsets; + +const char **pack_names; +struct packed_git **packs; +char object_dir[FLEX_ARRAY ]; +};struct multi_pack_index *load_multi_pack_index (const char *object_dir);int bsearch_midx (const struct object_id *oid,struct multi_pack_index *m,uint32_t *result);
object-store.h +1 -25
@@ -84,31 +84,7 @@ struct packed_git {char pack_name[FLEX_ARRAY ];/* more */ }; -struct multi_pack_index { -struct multi_pack_index *next; - -int fd; - -const unsigned char *data; -size_t data_len; - -uint32_t signature; -unsigned char version; -unsigned char hash_len; -unsigned char num_chunks; -uint32_t num_packs; -uint32_t num_objects; - -const unsigned char *chunk_pack_names; -const uint32_t *chunk_oid_fanout; -const unsigned char *chunk_oid_lookup; -const unsigned char *chunk_object_offsets; -const unsigned char *chunk_large_offsets; - -const char **pack_names; -struct packed_git **packs; -char object_dir[FLEX_ARRAY ]; -}; +struct multi_pack_index ;struct raw_object_store {/*
t/helper/test-read-midx.c +8 -11
@@ -1,6 +1,3 @@ -/* -* test-mktemp.c: code to exercise the creation of temporary files -*/ #include "test-tool.h" #include "cache.h" #include "midx.h"
@@ -13,7 +10,7 @@ static int read_midx_file(const char *object_dir)struct multi_pack_index *m =load_multi_pack_index (object_dir);if (!m) -return 0 ; +return 1 ; printf("header: %08x %d %d %d\n", m->signature,
@@ -24,15 +21,15 @@ static int read_midx_file(const char *object_dir)printf ("chunks:" );if (m->chunk_pack_names) -printf (" pack_names" ); +printf (" pack-names" );if (m->chunk_oid_fanout) -printf (" oid_fanout" ); +printf (" oid-fanout" );if (m->chunk_oid_lookup) -printf (" oid_lookup" ); +printf (" oid-lookup" );if (m->chunk_object_offsets) -printf (" object_offsets" ); +printf (" object-offsets" );if (m->chunk_large_offsets) -printf (" large_offsets" ); +printf (" large-offsets" );printf ("\nnum_objects: %d\n" , m->num_objects);
@@ -40,7 +37,7 @@ static int read_midx_file(const char *object_dir)for (i =0 ; i < m->num_packs; i++)printf ("%s\n" , m->pack_names[i]); -printf ("object_dir: %s\n" , m->object_dir); +printf ("object-dir: %s\n" , m->object_dir);return 0 ; }
@@ -48,7 +45,7 @@ static int read_midx_file(const char *object_dir)int cmd__read_midx (int argc,const char **argv) {if (argc !=2 ) -usage ("read-midx <object_dir>" ); +usage ("read-midx <object-dir>" );return read_midx_file (argv[1 ]); }
t/t5319-multi-pack-index.sh +85 -97
243 lines not on this page · raw message