Reftable support git-core
Subsystems (2)
cover Reftable support git-core
This adds the reftable library, and hooks it up as a ref backend.
Han-Wen Nienhuys (5):
setup.c: enable repo detection for reftable
create .git/refs in files-backend.c
Document how ref iterators and symrefs interact
Add reftable library
Reftable support for git-core
Makefile | 23 +- builtin/init-db.c | 2 - refs.c | 18 +- refs.h | 2 + refs/files-backend.c | 4 + refs/refs-internal.h | 4 + refs/reftable-backend.c | 780 ++++++++++++++++++++++++++++ reftable/README.md | 12 + reftable/VERSION | 293 +++++++++++ reftable/basics.c | 180 +++++++ reftable/basics.h | 38 ++ reftable/block.c | 382 ++++++++++++++ reftable/block.h | 71 +++ reftable/block_test.c | 145 ++++++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/constants.h | 27 + reftable/dump.c | 102 ++++ reftable/file.c | 98 ++++ reftable/iter.c | 211 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 262 ++++++++++ reftable/merged.h | 34 ++ reftable/merged_test.c | 247 +++++++++ reftable/pq.c | 116 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 672 ++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1030 +++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/record_test.c | 313 +++++++++++ reftable/reftable.h | 396 ++++++++++++++ reftable/reftable_test.c | 434 ++++++++++++++++ reftable/slice.c | 179 +++++++ reftable/slice.h | 39 ++ reftable/slice_test.c | 38 ++ reftable/stack.c | 931 +++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/stack_test.c | 265 ++++++++++ reftable/test_framework.c | 60 +++ reftable/test_framework.h | 63 +++ reftable/tree.c | 60 +++ reftable/tree.h | 24 + reftable/tree_test.c | 54 ++ reftable/writer.c | 586 +++++++++++++++++++++ reftable/writer.h | 46 ++ setup.c | 20 +- 47 files changed, 8530 insertions(+), 12 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/block_test.c create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/record_test.c create mode 100644 reftable/reftable.h create mode 100644 reftable/reftable_test.c create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/slice_test.c create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c create mode 100644 reftable/test_framework.c create mode 100644 reftable/test_framework.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/tree_test.c create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h
base-commit: 232378479ee6c66206d47a9be175e3a39682aea6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Signature · 1 line
gitgitgadget
PATCH 1/5 setup.c: enable repo detection for reftable
From: Han-Wen Nienhuys <[email protected]>
* only check R_OK for .git/refs.
In the reftable format, tables are stored under $GITDIR/reftable/ and the list of tables is managed in a file $GITDIR/refs. Generally, this file does not have the +x bit set
* allow missing HEAD if there is a reftable/
Change-Id I5d22317a15a84c8529aa503ae357a4afba247fe9
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- setup.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
setup.c +14 -6
@@ -304,22 +304,23 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)* - either an objects/ directory _or_ the proper * GIT_OBJECT_DIRECTORY environment variable * - a refs/ directory -* - either a HEAD symlink or a HEAD file that is formatted as -* a proper "ref:", or a regular file HEAD that has a properly -* formatted sha1 object name. +* - a reftable/ director or, either a HEAD ref. +* The HEAD ref should be a HEAD symlink or a HEAD file that is formatted as +* a proper "ref:", or a regular file HEAD that has a properly formatted sha1 +* object name. */ int is_git_directory (const char *suspect) {struct strbuf path =STRBUF_INIT ;int ret =0 ; +int have_head =0 ;size_t len;/* Check worktree-related signatures */ strbuf_addstr (&path, suspect);strbuf_complete (&path,'/' );strbuf_addstr (&path,"HEAD" ); -if (validate_headref (path.buf)) - goto done; + have_head = !validate_headref (path.buf);strbuf_reset (&path);get_common_dir (&path, suspect);
@@ -339,9 +340,16 @@ int is_git_directory(const char *suspect)strbuf_setlen (&path, len);strbuf_addstr (&path,"/refs" ); -if (access (path.buf,X_OK )) +if (access (path.buf,R_OK )) goto done; +if (!have_head) { +strbuf_setlen (&path, len); +strbuf_addstr (&path,"/reftable" ); +if (access (path.buf,X_OK )) + goto done; + } + ret =1 ; done:strbuf_release (&path);
Signature · 1 line
gitgitgadget
PATCH 2/5 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which creates a file in that place.
Signed-off-by Han-Wen Nienhuys <[email protected]>
Change-Id I2fc47c89f5ec605734007ceff90321c02474aa92
--- builtin/init-db.c | 2 -- refs/files-backend.c | 4 ++++ 2 files changed, 4 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +4 -0
@@ -3158,6 +3158,10 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +// XXX adjust_shared_perm ? +/* * Create .git/refs/{heads,tags} */
Signature · 1 line
gitgitgadget
PATCH 3/5 Document how ref iterators and symrefs interact
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Change-Id Ie3ee63c52254c000ef712986246ca28f312b4301
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 3 +++ 1 file changed, 3 insertions(+)
refs/refs-internal.h +3 -0
@@ -269,6 +269,9 @@ int refs_rename_ref_available(struct ref_store *refs,* to the next entry, ref_iterator_advance() aborts the iteration, * frees the ref_iterator, and returns ITER_ERROR. * +* Ref iterators cannot return symref targets, so symbolic refs must be +* dereferenced during the iteration. +* * The reference currently being looked at can be peeled by calling * ref_iterator_peel(). This function is often faster than peel_ref(), * so it should be preferred when iterating over references.
Signature · 1 line
gitgitgadget
PATCH 4/5 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
Change-Id Id396ff42be8b42b9e11f194a32e2f95b8250c109
--- reftable/README.md | 12 + reftable/VERSION | 293 +++++++++++ reftable/basics.c | 180 +++++++ reftable/basics.h | 38 ++ reftable/block.c | 382 ++++++++++++++ reftable/block.h | 71 +++ reftable/block_test.c | 145 ++++++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/constants.h | 27 + reftable/dump.c | 102 ++++ reftable/file.c | 98 ++++ reftable/iter.c | 211 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 262 ++++++++++ reftable/merged.h | 34 ++ reftable/merged_test.c | 247 +++++++++ reftable/pq.c | 116 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 672 ++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1030 +++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/record_test.c | 313 +++++++++++ reftable/reftable.h | 396 ++++++++++++++ reftable/reftable_test.c | 434 ++++++++++++++++ reftable/slice.c | 179 +++++++ reftable/slice.h | 39 ++ reftable/slice_test.c | 38 ++ reftable/stack.c | 931 +++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/stack_test.c | 265 ++++++++++ reftable/test_framework.c | 60 +++ reftable/test_framework.h | 63 +++ reftable/tree.c | 60 +++ reftable/tree.h | 24 + reftable/tree_test.c | 54 ++ reftable/writer.c | 586 +++++++++++++++++++++ reftable/writer.h | 46 ++ 39 files changed, 7689 insertions(+) create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/block_test.c create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/record_test.c create mode 100644 reftable/reftable.h create mode 100644 reftable/reftable_test.c create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/slice_test.c create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c create mode 100644 reftable/test_framework.c create mode 100644 reftable/test_framework.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/tree_test.c create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h
reftable/README.md new file +12 -0
@@ -0,0 +1,12 @@ +The source code in this directory comes from https://github.com/google/reftable +and can be updated by doing: + + rm -rf reftable-repo && \ + git clone https://github.com/google/reftable reftable-repo && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +293 -0
293 lines not on this page · raw message
reftable/basics.c new file +180 -0
180 lines not on this page · raw message
reftable/basics.h new file +38 -0
@@ -0,0 +1,38 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include <stdint.h> + +#include "reftable.h" + +#define true 1 +#define false 0 +#define ARRAYSIZE (a) sizeof(a) / sizeof(a[0]) + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +382 -0
382 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/block_test.c new file +145 -0
145 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/constants.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +102 -0
102 lines not on this page · raw message
reftable/file.c new file +98 -0
98 lines not on this page · raw message
reftable/iter.c new file +211 -0
211 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +262 -0
262 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/merged_test.c new file +247 -0
247 lines not on this page · raw message
reftable/pq.c new file +116 -0
116 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +672 -0
672 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1030 -0
1,030 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/record_test.c new file +313 -0
313 lines not on this page · raw message
reftable/reftable.h new file +396 -0
396 lines not on this page · raw message
reftable/reftable_test.c new file +434 -0
434 lines not on this page · raw message
reftable/slice.c new file +179 -0
179 lines not on this page · raw message
reftable/stack.c new file +931 -0
931 lines not on this page · raw message
reftable/stack_test.c new file +265 -0
265 lines not on this page · raw message
reftable/test_framework.c new file +60 -0
60 lines not on this page · raw message
reftable/test_framework.h new file +63 -0
63 lines not on this page · raw message
reftable/tree.c new file +60 -0
60 lines not on this page · raw message
reftable/tree_test.c new file +54 -0
54 lines not on this page · raw message
reftable/writer.c new file +586 -0
586 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
PATCH 5/5 Reftable support for git-core
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
$ ~/vc/git/git init
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
$ ~/vc/git/git add a
$ ~/vc/git/git commit -mx
fatal: not a git repository (or any of the parent directories): .git
[master (root-commit) 373d969] x
1 file changed, 1 insertion(+)
create mode 100644 a
$ ~/vc/git/git show-ref
373d96972fca9b63595740bba3898a762778ba20 HEAD
373d96972fca9b63595740bba3898a762778ba20 refs/heads/master
$ ls -l .git/reftable/
total 12
-rw------- 1 hanwen primarygroup 126 Jan 23 20:08 000000000001-000000000001.ref
-rw------- 1 hanwen primarygroup 4277 Jan 23 20:08 000000000002-000000000002.ref
$ go run ~/vc/reftable/cmd/dump.go -table /tmp/qz/.git/reftable/000000000002-000000000002.ref
** DEBUG **
name /tmp/qz/.git/reftable/000000000002-000000000002.ref, sz 4209: 'r' reftable.readerOffsets{Present:true, Offset:0x0, IndexOffset:0x0}, 'o' reftable.readerOffsets{Present:false, Offset:0x0, IndexOffset:0x0} 'g' reftable.readerOffsets{Present:true, Offset:0x1000, IndexOffset:0x0}
** REFS **
reftable.RefRecord{RefName:"refs/heads/master", UpdateIndex:0x2, Value:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, TargetValue:[]uint8(nil), Target:""}
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}TODO:
* resolve the design problem with reflog expiry.
Change-Id I225ee6317b7911edf9aa95f43299f6c7c4511914
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- Makefile | 23 +- refs.c | 18 +- refs.h | 2 + refs/refs-internal.h | 1 + refs/reftable-backend.c | 780 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 820 insertions(+), 4 deletions(-) create mode 100644 refs/reftable-backend.c
Makefile +22 -1
@@ -813,6 +813,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += command-list.h
@@ -958,6 +959,7 @@ LIB_OBJS += rebase-interactive.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
@@ -1162,7 +1164,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION)
@@ -2343,11 +2345,27 @@ VCSSVN_OBJS += vcs-svn/fast_export.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL
@@ -2484,6 +2502,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
refs.c +15 -3
@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1839,10 +1839,22 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,static struct ref_store *ref_store_init (const char *gitdir,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct strbuf refs_path =STRBUF_INIT ; + +// XXX this should probably come from a git config setting and not +// default to reftable. +const char *be_name ="reftable" ; +struct ref_storage_be *be;struct ref_store *refs; +strbuf_addstr (&refs_path, gitdir); +strbuf_addstr (&refs_path,"/refs" ); + +if (!is_directory (refs_path.buf)) { + be_name ="reftable" ; + } +strbuf_release (&refs_path); + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
refs.h +2 -0
@@ -456,6 +456,8 @@ int refs_for_each_reflog_ent_reverse(struct ref_store *refs,const char *refname, each_reflog_ent_fn fn, void *cb_data); + +// XXX which ordering are these? Newest or oldest first? int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
refs/refs-internal.h +1 -0
@@ -659,6 +659,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +780 -0
780 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Quoted 1 line · Han-Wen Nienhuys via GitGitGadget
This adds the reftable library, and hooks it up as a ref backend.
Just a quick impression before getting into details of individual steps.
* With this series, the reftable backend seems to take over as the default and only backend. We would need to design and decide how repositories would specify which backend it uses (I personally do not think we need to allow more than one backend to be active at the same time) before we take this series out of RFC status.
* What's reftable/VERSION file? Does it really have what you intended to add?
* Mixed indentation and many whitespace breakages make the code distracting to review.
* Comparison with 0 is written as "if (!strcmp(a, b))" in this codebase, and never "if (0 != strcmp(a, b))".
* We unfortunately do not use var defn "for (int i = 0; ..." yet.
* We do not use // comments.
Thanks.
Quoted 101 lines
Han-Wen Nienhuys (5):
setup.c: enable repo detection for reftable
create .git/refs infiles-backend.c
Document how ref iterators and symrefs interact
Add reftable library
Reftable support for git-coreMakefile | 23 +- builtin/init-db.c | 2 - refs.c | 18 +- refs.h | 2 + refs/files-backend.c | 4 + refs/refs-internal.h | 4 + refs/reftable-backend.c | 780 ++++++++++++++++++++++++++++ reftable/README.md | 12 + reftable/VERSION | 293 +++++++++++ reftable/basics.c | 180 +++++++ reftable/basics.h | 38 ++ reftable/block.c | 382 ++++++++++++++ reftable/block.h | 71 +++ reftable/block_test.c | 145 ++++++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/constants.h | 27 + reftable/dump.c | 102 ++++ reftable/file.c | 98 ++++ reftable/iter.c | 211 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 262 ++++++++++ reftable/merged.h | 34 ++ reftable/merged_test.c | 247 +++++++++ reftable/pq.c | 116 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 672 ++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1030 +++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/record_test.c | 313 +++++++++++ reftable/reftable.h | 396 ++++++++++++++ reftable/reftable_test.c | 434 ++++++++++++++++ reftable/slice.c | 179 +++++++ reftable/slice.h | 39 ++ reftable/slice_test.c | 38 ++ reftable/stack.c | 931 +++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/stack_test.c | 265 ++++++++++ reftable/test_framework.c | 60 +++ reftable/test_framework.h | 63 +++ reftable/tree.c | 60 +++ reftable/tree.h | 24 + reftable/tree_test.c | 54 ++ reftable/writer.c | 586 +++++++++++++++++++++ reftable/writer.h | 46 ++ setup.c | 20 +- 47 files changed, 8530 insertions(+), 12 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/block_test.c create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/record_test.c create mode 100644 reftable/reftable.h create mode 100644 reftable/reftable_test.c create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/slice_test.c create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c create mode 100644 reftable/test_framework.c create mode 100644 reftable/test_framework.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/tree_test.c create mode 100644 reftable/writer.c create mode 100644 reftable/writer.hbase-commit:
232378479ee6c66206d47a9be175e3a39682aea6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Quoted 10 lines · Junio C Hamano
Quoted 1 line
This adds the reftable library, and hooks it up as a ref backend.
Just a quick impression before getting into details of individual steps.
* With this series, the reftable backend seems to take over as the default and only backend. We would need to design and decide how repositories would specify which backend it uses (I personally do not think we need to allow more than one backend to be active at the same time) before we take this series out of RFC status.
Yes, obviously. Would you have concrete ideas on how this should work?
Quoted 2 lines
* What's reftable/VERSION file? Does it really have what you intended to add?
Fixed, and explained in reftable/VERSION.
Quoted 2 lines
* Mixed indentation and many whitespace breakages make the code distracting to review.
Do you mean in the changes to refs/reftable-backend.c? Or the imported code?
The imported code is uniformly formatted with clang-format.
Is there a clang-format setting for uniformly formatting to Linux/Git style? I really love automated formatting, so we don't have to spend time debating irrelevant details.
Quoted 8 lines
* Comparison with 0 is written as "if (!strcmp(a, b))" in this codebase, and never "if (0 !=
strcmp(a, b))".* We unfortunately do not use var defn "for (int i = 0; ..." yet.
* We do not use // comments.
fixed.
Quoted 10 lines · Han-Wen Nienhuys
Quoted 2 lines
* Mixed indentation and many whitespace breakages make the code distracting to review.
Do you mean in the changes to
refs/reftable-backend.c? Or the imported code?The imported code is uniformly formatted with clang-format.
Is there a clang-format setting for uniformly formatting to Linux/Git style? I really love automated formatting, so we don't have to spend time debating irrelevant details.
Ah, there is a .clang-format entry in the git repository already. Thanks!
Hi Han-Wen,
Quoted 8 lines · Han-Wen Nienhuys via GitGitGadget
This adds the reftable library, and hooks it up as a ref backend.
Han-Wen Nienhuys (5):
setup.c: enable repo detection for reftable
create .git/refs infiles-backend.c
Document how ref iterators and symrefs interact
Add reftable library
Reftable support for git-core
I am most of the time just a curious reader on this list but as someone who has no idea what the reftable library does (except that it can serve as a "ref backend"), I would expect much more elaborate commit messages. In particular, patch 4/5 (i.e. the commit message of the "add reftable library" commit) should describe the purpose of the reftable library at least briefly; and patch 5/5 should not contain shell commands and output without context but a short description why the patch is doing what it is doing. For example, if the use of the reftable library makes certain git commands a lot faster, it should state that.
Thanks.
Stephan
I added some more background in one of the commit messages. You can find extensive discussion here:
https://github.com/google/reftable#background-reading
Quoted 26 lines · Stephan Beyer
Hi Han-Wen,
Quoted 8 lines · Han-Wen Nienhuys via GitGitGadget
This adds the reftable library, and hooks it up as a ref backend.
Han-Wen Nienhuys (5):
setup.c: enable repo detection for reftable
create .git/refs infiles-backend.c
Document how ref iterators and symrefs interact
Add reftable library
Reftable support for git-coreI am most of the time just a curious reader on this list but as someone who has no idea what the reftable library does (except that it can serve as a "ref backend"), I would expect much more elaborate commit messages. In particular, patch 4/5 (i.e. the commit message of the "add reftable library" commit) should describe the purpose of the reftable library at least briefly; and patch 5/5 should not contain shell commands and output without context but a short description why the patch is doing what it is doing. For example, if the use of the reftable library makes certain git commands a lot faster, it should state that.
Thanks.
Stephan
v2 cover Reftable support git-core
In reply to Han-Wen Nienhuys via GitGitGadgetThis adds the reftable library, and hooks it up as a ref backend.
At this point, I am mainly interested in feedback on the spots marked with XXX in the Git source code, in particular, how to handle reflog expiry in this backend.
v2
* address Jun's nits.
* address Dscho's portability comments
* more background in commit messages.
Han-Wen Nienhuys (5):
setup.c: enable repo detection for reftable
create .git/refs in files-backend.c
Document how ref iterators and symrefs interact
Add reftable library
Reftable support for git-core
Makefile | 23 +- builtin/init-db.c | 2 - refs.c | 18 +- refs.h | 2 + refs/files-backend.c | 4 + refs/refs-internal.h | 4 + refs/reftable-backend.c | 812 +++++++++++++++++++++++++++ reftable/LICENSE | 31 ++ reftable/README.md | 17 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 38 ++ reftable/block.c | 401 ++++++++++++++ reftable/block.h | 71 +++ reftable/block_test.c | 151 +++++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 230 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 288 ++++++++++ reftable/merged.h | 34 ++ reftable/merged_test.c | 258 +++++++++ reftable/pq.c | 124 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 710 ++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1110 +++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/record_test.c | 332 +++++++++++ reftable/reftable.h | 399 +++++++++++++ reftable/reftable_test.c | 481 ++++++++++++++++ reftable/slice.c | 199 +++++++ reftable/slice.h | 39 ++ reftable/slice_test.c | 38 ++ reftable/stack.c | 985 ++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/stack_test.c | 281 ++++++++++ reftable/system.h | 36 ++ reftable/test_framework.c | 67 +++ reftable/test_framework.h | 64 +++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/tree_test.c | 61 ++ reftable/writer.c | 624 +++++++++++++++++++++ reftable/writer.h | 46 ++ setup.c | 20 +- 50 files changed, 8782 insertions(+), 12 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/block_test.c create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/record_test.c create mode 100644 reftable/reftable.h create mode 100644 reftable/reftable_test.c create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/slice_test.c create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c create mode 100644 reftable/system.h create mode 100644 reftable/test_framework.c create mode 100644 reftable/test_framework.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/tree_test.c create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h
base-commit: bc7a3d4dc04dd719e7c8c35ebd7a6e6651c5c5b6
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Range-diff vs v1:
1: fd2baf1628 = 1: 174b98f6db setup.c: enable repo detection for reftable
2: bc643d0b0c = 2: d7d642dcf6 create .git/refs in files-backend.c
3: 1a01e0b1b5 = 3: 9cf185b51f Document how ref iterators and symrefs interact
4: 3c86bd1d7e ! 4: 2106ff286b Add reftable library
@@ -2,23 +2,92 @@
Add reftable library
+ Reftable is a new format for storing the ref database. It provides the
+ following benefits:
+
+ * Simple and fast atomic ref transactions, including multiple refs and reflogs.
+ * Compact storage of ref data.
+ * Fast look ups of ref data.
+ * Case-sensitive ref names on Windows/OSX, regardless of file system
+ * Eliminates file/directory conflicts in ref names
+
+ Further context and motivation can be found in background reading:
+
+ * Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
+
+ * Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
+
+ * First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
+
+ * Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
+
+ * First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
+
+ * libgit2 support issue: https://github.com/libgit2/libgit2/issues
+
+ * GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
+
+ * go-git support issue: https://github.com/src-d/go-git/issues/1059
+
Signed-off-by: Han-Wen Nienhuys <[email protected]>
Change-Id: Id396ff42be8b42b9e11f194a32e2f95b8250c109
+ diff --git a/reftable/LICENSE b/reftable/LICENSE
+ new file mode 100644
+ --- /dev/null
+ +++ b/reftable/LICENSE
+@@
++BSD License
++
++Copyright (c) 2020, Google LLC
++All rights reserved.
++
++Redistribution and use in source and binary forms, with or without
++modification, are permitted provided that the following conditions are
++met:
++
++* Redistributions of source code must retain the above copyright notice,
++this list of conditions and the following disclaimer.
++
++* Redistributions in binary form must reproduce the above copyright
++notice, this list of conditions and the following disclaimer in the
++documentation and/or other materials provided with the distribution.
++
++* Neither the name of Google LLC nor the names of its contributors may
++be used to endorse or promote products derived from this software
++without specific prior written permission.
++
++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
++"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
++LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
++A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
++OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
++SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
++LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
++DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
++THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
++(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
++OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
diff --git a/reftable/README.md b/reftable/README.md
new file mode 100644
--- /dev/null
+++ b/reftable/README.md
@@
-+The source code in this directory comes from https://github.com/google/reftable
-+and can be updated by doing:
+
-+ rm -rf reftable-repo && \
-+ git clone https://github.com/google/reftable reftable-repo && \
++The source code in this directory comes from https://github.com/google/reftable.
++
++The VERSION file keeps track of the current version of the reftable library.
++
++To update the library, do:
++
++ ((cd reftable-repo && git fetch origin && git checkout origin/master ) ||
++ git clone https://github.com/google/reftable reftable-repo) && \
+ cp reftable-repo/c/*.[ch] reftable/ && \
-+ cp reftable-repo/LICENSE reftable/
++ cp reftable-repo/LICENSE reftable/ &&
+ git --git-dir reftable-repo/.git show --no-patch origin/master \
-+ > reftable/VERSION
++ > reftable/VERSION && \
++ echo '/* empty */' > reftable/config.h
+
+Bugfixes should be accompanied by a test and applied to upstream project at
+https://github.com/google/reftable.
@@ -28,299 +97,11 @@
--- /dev/null
+++ b/reftable/VERSION
@@
-+commit 887250932a9c39a15aee26aef59df2300c77c03f
++commit c616d53b88657c3a5fe4d2e7243a48effc34c626
+Author: Han-Wen Nienhuys <[email protected]>
-+Date: Wed Jan 22 19:50:16 2020 +0100
-+
-+ C: implement reflog expiry
-+
-+diff --git a/c/reftable.h b/c/reftable.h
-+index 2f44973..d760af9 100644
-+--- a/c/reftable.h
-++++ b/c/reftable.h
-+@@ -359,8 +359,17 @@ void stack_destroy(struct stack *st);
-+ /* reloads the stack if necessary. */
-+ int stack_reload(struct stack *st);
-+
-+-/* compacts all reftables into a giant table. */
-+-int stack_compact_all(struct stack *st);
-++/* Policy for expiring reflog entries. */
-++struct log_expiry_config {
-++ /* Drop entries older than this timestamp */
-++ uint64_t time;
-++
-++ /* Drop older entries */
-++ uint64_t min_update_index;
-++};
-++
-++/* compacts all reftables into a giant table. Expire reflog entries if config is non-NULL */
-++int stack_compact_all(struct stack *st, struct log_expiry_config *config);
-+
-+ /* heuristically compact unbalanced table stack. */
-+ int stack_auto_compact(struct stack *st);
-+diff --git a/c/stack.c b/c/stack.c
-+index d67828f..641ac52 100644
-+--- a/c/stack.c
-++++ b/c/stack.c
-+@@ -119,7 +119,7 @@ static struct reader **stack_copy_readers(struct stack *st, int cur_len) {
-+ return cur;
-+ }
-+
-+-static int stack_reload_once(struct stack *st, char **names) {
-++static int stack_reload_once(struct stack *st, char **names, bool reuse_open) {
-+ int cur_len = st->merged == NULL ? 0 : st->merged->stack_len;
-+ struct reader **cur = stack_copy_readers(st, cur_len);
-+ int err = 0;
-+@@ -136,7 +136,7 @@ static int stack_reload_once(struct stack *st, char **names) {
-+
-+ // this is linear; we assume compaction keeps the number of tables
-+ // under control so this is not quadratic.
-+- for (int j = 0; j < cur_len; j++) {
-++ for (int j = 0; reuse_open && j < cur_len; j++) {
-+ if (cur[j] != NULL && 0 == strcmp(cur[j]->name, name)) {
-+ rd = cur[j];
-+ cur[j] = NULL;
-+@@ -207,7 +207,7 @@ static int tv_cmp(struct timeval *a, struct timeval *b) {
-+ return udiff;
-+ }
-+
-+-int stack_reload(struct stack *st) {
-++static int stack_reload_maybe_reuse(struct stack *st, bool reuse_open) {
-+ struct timeval deadline = {};
-+ int err = gettimeofday(&deadline, NULL);
-+ int64_t delay = 0;
-+@@ -238,7 +238,7 @@ int stack_reload(struct stack *st) {
-+ free_names(names);
-+ return err;
-+ }
-+- err = stack_reload_once(st, names);
-++ err = stack_reload_once(st, names, reuse_open);
-+ if (err == 0) {
-+ free_names(names);
-+ break;
-+@@ -269,6 +269,10 @@ int stack_reload(struct stack *st) {
-+ return 0;
-+ }
-+
-++int stack_reload(struct stack *st) {
-++ return stack_reload_maybe_reuse(st, true);
-++}
-++
-+ // -1 = error
-+ // 0 = up to date
-+ // 1 = changed.
-+@@ -471,7 +475,7 @@ uint64_t stack_next_update_index(struct stack *st) {
-+ }
-+
-+ static int stack_compact_locked(struct stack *st, int first, int last,
-+- struct slice *temp_tab) {
-++ struct slice *temp_tab, struct log_expiry_config *config) {
-+ struct slice next_name = {};
-+ int tab_fd = -1;
-+ struct writer *wr = NULL;
-+@@ -488,7 +492,7 @@ static int stack_compact_locked(struct stack *st, int first, int last,
-+ tab_fd = mkstemp((char *)slice_as_string(temp_tab));
-+ wr = new_writer(fd_writer, &tab_fd, &st->config);
-+
-+- err = stack_write_compact(st, wr, first, last);
-++ err = stack_write_compact(st, wr, first, last, config);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+@@ -515,7 +519,7 @@ static int stack_compact_locked(struct stack *st, int first, int last,
-+ }
-+
-+ int stack_write_compact(struct stack *st, struct writer *wr, int first,
-+- int last) {
-++ int last, struct log_expiry_config *config) {
-+ int subtabs_len = last - first + 1;
-+ struct reader **subtabs = calloc(sizeof(struct reader *), last - first + 1);
-+ struct merged_table *mt = NULL;
-+@@ -580,6 +584,16 @@ int stack_write_compact(struct stack *st, struct writer *wr, int first,
-+ continue;
-+ }
-+
-++ // XXX collect stats?
-++
-++ if (config != NULL && config->time > 0 && log.time < config->time) {
-++ continue;
-++ }
-++
-++ if (config != NULL && config->min_update_index > 0 && log.update_index < config->min_update_index) {
-++ continue;
-++ }
-++
-+ err = writer_add_log(wr, &log);
-+ if (err < 0) {
-+ break;
-+@@ -599,7 +613,7 @@ int stack_write_compact(struct stack *st, struct writer *wr, int first,
-+ }
-+
-+ // < 0: error. 0 == OK, > 0 attempt failed; could retry.
-+-static int stack_compact_range(struct stack *st, int first, int last) {
-++static int stack_compact_range(struct stack *st, int first, int last, struct log_expiry_config *expiry) {
-+ struct slice temp_tab_name = {};
-+ struct slice new_table_name = {};
-+ struct slice lock_file_name = {};
-+@@ -612,7 +626,7 @@ static int stack_compact_range(struct stack *st, int first, int last) {
-+ char **delete_on_success = calloc(sizeof(char *), compact_count + 1);
-+ char **subtable_locks = calloc(sizeof(char *), compact_count + 1);
-+
-+- if (first >= last) {
-++ if (first > last || (expiry == NULL && first == last)) {
-+ err = 0;
-+ goto exit;
-+ }
-+@@ -676,7 +690,7 @@ static int stack_compact_range(struct stack *st, int first, int last) {
-+ }
-+ have_lock = false;
-+
-+- err = stack_compact_locked(st, first, last, &temp_tab_name);
-++ err = stack_compact_locked(st, first, last, &temp_tab_name, expiry);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+@@ -739,10 +753,12 @@ static int stack_compact_range(struct stack *st, int first, int last) {
-+ have_lock = false;
-+
-+ for (char **p = delete_on_success; *p; p++) {
-+- unlink(*p);
-++ if (0 != strcmp(*p, slice_as_string(&new_table_path))) {
-++ unlink(*p);
-++ }
-+ }
-+
-+- err = stack_reload(st);
-++ err = stack_reload_maybe_reuse(st, first < last);
-+ exit:
-+ for (char **p = subtable_locks; *p; p++) {
-+ unlink(*p);
-+@@ -764,12 +780,12 @@ static int stack_compact_range(struct stack *st, int first, int last) {
-+ return err;
-+ }
-+
-+-int stack_compact_all(struct stack *st) {
-+- return stack_compact_range(st, 0, st->merged->stack_len - 1);
-++int stack_compact_all(struct stack *st, struct log_expiry_config *config) {
-++ return stack_compact_range(st, 0, st->merged->stack_len - 1, config);
-+ }
-+
-+-static int stack_compact_range_stats(struct stack *st, int first, int last) {
-+- int err = stack_compact_range(st, first, last);
-++static int stack_compact_range_stats(struct stack *st, int first, int last, struct log_expiry_config *config) {
-++ int err = stack_compact_range(st, first, last, config);
-+ if (err > 0) {
-+ st->stats.failures++;
-+ }
-+@@ -856,7 +872,7 @@ int stack_auto_compact(struct stack *st) {
-+ struct segment seg = suggest_compaction_segment(sizes, st->merged->stack_len);
-+ free(sizes);
-+ if (segment_size(&seg) > 0) {
-+- return stack_compact_range_stats(st, seg.start, seg.end - 1);
-++ return stack_compact_range_stats(st, seg.start, seg.end - 1, NULL);
-+ }
-+
-+ return 0;
-+diff --git a/c/stack.h b/c/stack.h
-+index 8cb2cb0..092a03b 100644
-+--- a/c/stack.h
-++++ b/c/stack.h
-+@@ -25,7 +25,7 @@ int read_lines(const char *filename, char ***lines);
-+ int stack_try_add(struct stack *st,
-+ int (*write_table)(struct writer *wr, void *arg), void *arg);
-+ int stack_write_compact(struct stack *st, struct writer *wr, int first,
-+- int last);
-++ int last, struct log_expiry_config *config);
-+ int fastlog2(uint64_t sz);
-+
-+ struct segment {
-+diff --git a/c/stack_test.c b/c/stack_test.c
-+index c49d8b8..8c0cde0 100644
-+--- a/c/stack_test.c
-++++ b/c/stack_test.c
-+@@ -121,7 +121,7 @@ void test_stack_add(void) {
-+ assert_err(err);
-+ }
-+
-+- err = stack_compact_all(st);
-++ err = stack_compact_all(st, NULL);
-+ assert_err(err);
-+
-+ for (int i = 0; i < N; i++) {
-+@@ -186,7 +186,73 @@ void test_suggest_compaction_segment(void) {
-+ }
-+ }
-+
-++void test_reflog_expire(void) {
-++ char dir[256] = "/tmp/stack.XXXXXX";
-++ assert(mkdtemp(dir));
-++ printf("%s\n", dir);
-++ char fn[256] = "";
-++ strcat(fn, dir);
-++ strcat(fn, "/refs");
-++
-++ struct write_options cfg = {};
-++ struct stack *st = NULL;
-++ int err = new_stack(&st, dir, fn, cfg);
-++ assert_err(err);
-++
-++ struct log_record logs[20] = {};
-++ int N = ARRAYSIZE(logs) -1;
-++ for (int i = 1; i <= N; i++) {
-++ char buf[256];
-++ sprintf(buf, "branch%02d", i);
-++
-++ logs[i].ref_name = strdup(buf);
-++ logs[i].update_index = i;
-++ logs[i].time = i;
-++ logs[i].new_hash = malloc(SHA1_SIZE);
-++ logs[i].email = strdup("identity@invalid");
-++ set_test_hash(logs[i].new_hash, i);
-++ }
-++
-++ for (int i = 1; i <= N; i++) {
-++ int err = stack_add(st, &write_test_log, &logs[i]);
-++ assert_err(err);
-++ }
-++
-++ err = stack_compact_all(st, NULL);
-++ assert_err(err);
-++
-++ struct log_expiry_config expiry = {
-++ .time = 10,
-++ };
-++ err = stack_compact_all(st, &expiry);
-++ assert_err(err);
-++
-++ struct log_record log = {};
-++ err = stack_read_log(st, logs[9].ref_name, &log);
-++ assert(err == 1);
-++
-++ err = stack_read_log(st, logs[11].ref_name, &log);
-++ assert_err(err);
-++
-++ expiry.min_update_index = 15;
-++ err = stack_compact_all(st, &expiry);
-++ assert_err(err);
-++
-++ err = stack_read_log(st, logs[14].ref_name, &log);
-++ assert(err == 1);
-++
-++ err = stack_read_log(st, logs[16].ref_name, &log);
-++ assert_err(err);
-++
-++ // cleanup
-++ stack_destroy(st);
-++ for (int i = 0; i < N; i++) {
-++ log_record_clear(&logs[i]);
-++ }
-++}
-++
-+ int main() {
-++ add_test_case("test_reflog_expire", test_reflog_expire);
-+ add_test_case("test_suggest_compaction_segment",
-+ &test_suggest_compaction_segment);
-+ add_test_case("test_sizes_to_segments", &test_sizes_to_segments);
++Date: Mon Jan 27 15:05:43 2020 +0100
++
++ C: ban // comments
diff --git a/reftable/basics.c b/reftable/basics.c
new file mode 100644
@@ -337,175 +118,191 @@
+
+#include "basics.h"
+
-+#include <stdio.h>
-+#include <stdlib.h>
-+#include <string.h>
++#include "system.h"
+
-+void put_u24(byte *out, uint32_t i) {
-+ out[0] = (byte)((i >> 16) & 0xff);
-+ out[1] = (byte)((i >> 8) & 0xff);
-+ out[2] = (byte)((i)&0xff);
++void put_u24(byte *out, uint32_t i)
++{
++ out[0] = (byte)((i >> 16) & 0xff);
++ out[1] = (byte)((i >> 8) & 0xff);
++ out[2] = (byte)((i)&0xff);
+}
+
-+uint32_t get_u24(byte *in) {
-+ return (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 | (uint32_t)(in[2]);
++uint32_t get_u24(byte *in)
++{
++ return (uint32_t)(in[0]) << 16 | (uint32_t)(in[1]) << 8 |
++ (uint32_t)(in[2]);
+}
+
-+void put_u32(byte *out, uint32_t i) {
-+ out[0] = (byte)((i >> 24) & 0xff);
-+ out[1] = (byte)((i >> 16) & 0xff);
-+ out[2] = (byte)((i >> 8) & 0xff);
-+ out[3] = (byte)((i)&0xff);
++void put_u32(byte *out, uint32_t i)
++{
++ out[0] = (byte)((i >> 24) & 0xff);
++ out[1] = (byte)((i >> 16) & 0xff);
++ out[2] = (byte)((i >> 8) & 0xff);
++ out[3] = (byte)((i)&0xff);
+}
+
-+uint32_t get_u32(byte *in) {
-+ return (uint32_t)(in[0]) << 24 | (uint32_t)(in[1]) << 16 |
-+ (uint32_t)(in[2]) << 8 | (uint32_t)(in[3]);
++uint32_t get_u32(byte *in)
++{
++ return (uint32_t)(in[0]) << 24 | (uint32_t)(in[1]) << 16 |
++ (uint32_t)(in[2]) << 8 | (uint32_t)(in[3]);
+}
+
-+void put_u64(byte *out, uint64_t v) {
-+ for (int i = sizeof(uint64_t); i--;) {
-+ out[i] = (byte)(v & 0xff);
-+ v >>= 8;
-+ }
++void put_u64(byte *out, uint64_t v)
++{
++ int i = 0;
++ for (i = sizeof(uint64_t); i--;) {
++ out[i] = (byte)(v & 0xff);
++ v >>= 8;
++ }
+}
+
-+uint64_t get_u64(byte *out) {
-+ uint64_t v = 0;
-+ for (int i = 0; i < sizeof(uint64_t); i++) {
-+ v = (v << 8) | (byte)(out[i] & 0xff);
-+ }
-+ return v;
++uint64_t get_u64(byte *out)
++{
++ uint64_t v = 0;
++ int i = 0;
++ for (i = 0; i < sizeof(uint64_t); i++) {
++ v = (v << 8) | (byte)(out[i] & 0xff);
++ }
++ return v;
+}
+
-+void put_u16(byte *out, uint16_t i) {
-+ out[0] = (byte)((i >> 8) & 0xff);
-+ out[1] = (byte)((i)&0xff);
++void put_u16(byte *out, uint16_t i)
++{
++ out[0] = (byte)((i >> 8) & 0xff);
++ out[1] = (byte)((i)&0xff);
+}
+
-+uint16_t get_u16(byte *in) {
-+ return (uint32_t)(in[0]) << 8 | (uint32_t)(in[1]);
++uint16_t get_u16(byte *in)
++{
++ return (uint32_t)(in[0]) << 8 | (uint32_t)(in[1]);
+}
+
+/*
+ find smallest index i in [0, sz) at which f(i) is true, assuming
+ that f is ascending. Return sz if f(i) is false for all indices.
+*/
-+int binsearch(int sz, int (*f)(int k, void *args), void *args) {
-+ int lo = 0;
-+ int hi = sz;
-+
-+ /* invariant: (hi == sz) || f(hi) == true
-+ (lo == 0 && f(0) == true) || fi(lo) == false
-+ */
-+ while (hi - lo > 1) {
-+ int mid = lo + (hi - lo) / 2;
-+
-+ int val = f(mid, args);
-+ if (val) {
-+ hi = mid;
-+ } else {
-+ lo = mid;
-+ }
-+ }
-+
-+ if (lo == 0) {
-+ if (f(0, args)) {
-+ return 0;
-+ } else {
-+ return 1;
-+ }
-+ }
-+
-+ return hi;
-+}
-+
-+void free_names(char **a) {
-+ char **p = a;
-+ if (p == NULL) {
-+ return;
-+ }
-+ while (*p) {
-+ free(*p);
-+ p++;
-+ }
-+ free(a);
-+}
-+
-+int names_length(char **names) {
-+ int len = 0;
-+ for (char **p = names; *p; p++) {
-+ len++;
-+ }
-+ return len;
++int binsearch(int sz, int (*f)(int k, void *args), void *args)
++{
++ int lo = 0;
++ int hi = sz;
++
++ /* invariant: (hi == sz) || f(hi) == true
++ (lo == 0 && f(0) == true) || fi(lo) == false
++ */
++ while (hi - lo > 1) {
++ int mid = lo + (hi - lo) / 2;
++
++ int val = f(mid, args);
++ if (val) {
++ hi = mid;
++ } else {
++ lo = mid;
++ }
++ }
++
++ if (lo == 0) {
++ if (f(0, args)) {
++ return 0;
++ } else {
++ return 1;
++ }
++ }
++
++ return hi;
++}
++
++void free_names(char **a)
++{
++ char **p = a;
++ if (p == NULL) {
++ return;
++ }
++ while (*p) {
++ free(*p);
++ p++;
++ }
++ free(a);
++}
++
++int names_length(char **names)
++{
++ int len = 0;
++ for (char **p = names; *p; p++) {
++ len++;
++ }
++ return len;
+}
+
+/* parse a newline separated list of names. Empty names are discarded. */
-+void parse_names(char *buf, int size, char ***namesp) {
-+ char **names = NULL;
-+ int names_cap = 0;
-+ int names_len = 0;
-+
-+ char *p = buf;
-+ char *end = buf + size;
-+ while (p < end) {
-+ char *next = strchr(p, '\n');
-+ if (next != NULL) {
-+ *next = 0;
-+ } else {
-+ next = end;
-+ }
-+ if (p < next) {
-+ if (names_len == names_cap) {
-+ names_cap = 2 * names_cap + 1;
-+ names = realloc(names, names_cap * sizeof(char *));
-+ }
-+ names[names_len++] = strdup(p);
-+ }
-+ p = next + 1;
-+ }
-+
-+ if (names_len == names_cap) {
-+ names_cap = 2 * names_cap + 1;
-+ names = realloc(names, names_cap * sizeof(char *));
-+ }
-+
-+ names[names_len] = NULL;
-+ *namesp = names;
-+}
-+
-+int names_equal(char **a, char **b) {
-+ while (*a && *b) {
-+ if (0 != strcmp(*a, *b)) {
-+ return 0;
-+ }
-+
-+ a++;
-+ b++;
-+ }
-+
-+ return *a == *b;
-+}
-+
-+const char *error_str(int err) {
-+ switch (err) {
-+ case IO_ERROR:
-+ return "I/O error";
-+ case FORMAT_ERROR:
-+ return "FORMAT_ERROR";
-+ case NOT_EXIST_ERROR:
-+ return "NOT_EXIST_ERROR";
-+ case LOCK_ERROR:
-+ return "LOCK_ERROR";
-+ case API_ERROR:
-+ return "API_ERROR";
-+ case ZLIB_ERROR:
-+ return "ZLIB_ERROR";
-+ case -1:
-+ return "general error";
-+ default:
-+ return "unknown error code";
-+ }
++void parse_names(char *buf, int size, char ***namesp)
++{
++ char **names = NULL;
++ int names_cap = 0;
++ int names_len = 0;
++
++ char *p = buf;
++ char *end = buf + size;
++ while (p < end) {
++ char *next = strchr(p, '\n');
++ if (next != NULL) {
++ *next = 0;
++ } else {
++ next = end;
++ }
++ if (p < next) {
++ if (names_len == names_cap) {
++ names_cap = 2 * names_cap + 1;
++ names = realloc(names,
++ names_cap * sizeof(char *));
++ }
++ names[names_len++] = strdup(p);
++ }
++ p = next + 1;
++ }
++
++ if (names_len == names_cap) {
++ names_cap = 2 * names_cap + 1;
++ names = realloc(names, names_cap * sizeof(char *));
++ }
++
++ names[names_len] = NULL;
++ *namesp = names;
++}
++
++int names_equal(char **a, char **b)
++{
++ while (*a && *b) {
++ if (strcmp(*a, *b)) {
++ return 0;
++ }
++
++ a++;
++ b++;
++ }
++
++ return *a == *b;
++}
++
++const char *error_str(int err)
++{
++ switch (err) {
++ case IO_ERROR:
++ return "I/O error";
++ case FORMAT_ERROR:
++ return "FORMAT_ERROR";
++ case NOT_EXIST_ERROR:
++ return "NOT_EXIST_ERROR";
++ case LOCK_ERROR:
++ return "LOCK_ERROR";
++ case API_ERROR:
++ return "API_ERROR";
++ case ZLIB_ERROR:
++ return "ZLIB_ERROR";
++ case -1:
++ return "general error";
++ default:
++ return "unknown error code";
++ }
+}
diff --git a/reftable/basics.h b/reftable/basics.h
@@ -524,7 +321,7 @@
+#ifndef BASICS_H
+#define BASICS_H
+
-+#include <stdint.h>
++#include "system.h"
+
+#include "reftable.h"
+
@@ -567,10 +364,7 @@
+
+#include "block.h"
+
-+#include <assert.h>
-+#include <stdio.h>
-+#include <stdlib.h>
-+#include <string.h>
++#include "system.h"
+
+#include "blocksource.h"
+#include "constants.h"
@@ -579,365 +373,387 @@
+#include "zlib.h"
+
+int block_writer_register_restart(struct block_writer *w, int n, bool restart,
-+ struct slice key);
++ struct slice key);
+
+void block_writer_init(struct block_writer *bw, byte typ, byte *buf,
-+ uint32_t block_size, uint32_t header_off,
-+ int hash_size) {
-+ bw->buf = buf;
-+ bw->hash_size = hash_size;
-+ bw->block_size = block_size;
-+ bw->header_off = header_off;
-+ bw->buf[header_off] = typ;
-+ bw->next = header_off + 4;
-+ bw->restart_interval = 16;
-+ bw->entries = 0;
++ uint32_t block_size, uint32_t header_off, int hash_size)
++{
++ bw->buf = buf;
++ bw->hash_size = hash_size;
++ bw->block_size = block_size;
++ bw->header_off = header_off;
++ bw->buf[header_off] = typ;
++ bw->next = header_off + 4;
++ bw->restart_interval = 16;
++ bw->entries = 0;
+}
+
-+byte block_writer_type(struct block_writer *bw) {
-+ return bw->buf[bw->header_off];
++byte block_writer_type(struct block_writer *bw)
++{
++ return bw->buf[bw->header_off];
+}
+
+/* adds the record to the block. Returns -1 if it does not fit, 0 on
+ success */
-+int block_writer_add(struct block_writer *w, struct record rec) {
-+ struct slice empty = {};
-+ struct slice last =
-+ w->entries % w->restart_interval == 0 ? empty : w->last_key;
-+ struct slice out = {
-+ .buf = w->buf + w->next,
-+ .len = w->block_size - w->next,
-+ };
-+
-+ struct slice start = out;
-+
-+ bool restart = false;
-+ struct slice key = {};
-+ int n = 0;
-+
-+ record_key(rec, &key);
-+ n = encode_key(&restart, out, last, key, record_val_type(rec));
-+ if (n < 0) {
-+ goto err;
-+ }
-+ out.buf += n;
-+ out.len -= n;
-+
-+ n = record_encode(rec, out, w->hash_size);
-+ if (n < 0) {
-+ goto err;
-+ }
-+
-+ out.buf += n;
-+ out.len -= n;
-+
-+ if (block_writer_register_restart(w, start.len - out.len, restart, key) < 0) {
-+ goto err;
-+ }
-+
-+ free(slice_yield(&key));
-+ return 0;
++int block_writer_add(struct block_writer *w, struct record rec)
++{
++ struct slice empty = {};
++ struct slice last = w->entries % w->restart_interval == 0 ? empty :
++ w->last_key;
++ struct slice out = {
++ .buf = w->buf + w->next,
++ .len = w->block_size - w->next,
++ };
++
++ struct slice start = out;
++
++ bool restart = false;
++ struct slice key = {};
++ int n = 0;
++
++ record_key(rec, &key);
++ n = encode_key(&restart, out, last, key, record_val_type(rec));
++ if (n < 0) {
++ goto err;
++ }
++ out.buf += n;
++ out.len -= n;
++
++ n = record_encode(rec, out, w->hash_size);
++ if (n < 0) {
++ goto err;
++ }
++
++ out.buf += n;
++ out.len -= n;
++
++ if (block_writer_register_restart(w, start.len - out.len, restart,
++ key) < 0) {
++ goto err;
++ }
++
++ free(slice_yield(&key));
++ return 0;
+
+err:
-+ free(slice_yield(&key));
-+ return -1;
++ free(slice_yield(&key));
++ return -1;
+}
+
+int block_writer_register_restart(struct block_writer *w, int n, bool restart,
-+ struct slice key) {
-+ int rlen = w->restart_len;
-+ if (rlen >= MAX_RESTARTS) {
-+ restart = false;
-+ }
-+
-+ if (restart) {
-+ rlen++;
-+ }
-+ if (2 + 3 * rlen + n > w->block_size - w->next) {
-+ return -1;
-+ }
-+ if (restart) {
-+ if (w->restart_len == w->restart_cap) {
-+ w->restart_cap = w->restart_cap * 2 + 1;
-+ w->restarts = realloc(w->restarts, sizeof(uint32_t) * w->restart_cap);
-+ }
-+
-+ w->restarts[w->restart_len++] = w->next;
-+ }
-+
-+ w->next += n;
-+ slice_copy(&w->last_key, key);
-+ w->entries++;
-+ return 0;
-+}
-+
-+int block_writer_finish(struct block_writer *w) {
-+ for (int i = 0; i < w->restart_len; i++) {
-+ put_u24(w->buf + w->next, w->restarts[i]);
-+ w->next += 3;
-+ }
-+
-+ put_u16(w->buf + w->next, w->restart_len);
-+ w->next += 2;
-+ put_u24(w->buf + 1 + w->header_off, w->next);
-+
-+ if (block_writer_type(w) == BLOCK_TYPE_LOG) {
-+ int block_header_skip = 4 + w->header_off;
-+ struct slice compressed = {};
-+ uLongf dest_len = 0, src_len = 0;
-+ slice_resize(&compressed, w->next - block_header_skip);
-+
-+ dest_len = compressed.len;
-+ src_len = w->next - block_header_skip;
-+
-+ if (Z_OK != compress2(compressed.buf, &dest_len, w->buf + block_header_skip,
-+ src_len, 9)) {
-+ free(slice_yield(&compressed));
-+ return ZLIB_ERROR;
-+ }
-+ memcpy(w->buf + block_header_skip, compressed.buf, dest_len);
-+ w->next = dest_len + block_header_skip;
-+ }
-+ return w->next;
-+}
-+
-+byte block_reader_type(struct block_reader *r) {
-+ return r->block.data[r->header_off];
++ struct slice key)
++{
++ int rlen = w->restart_len;
++ if (rlen >= MAX_RESTARTS) {
++ restart = false;
++ }
++
++ if (restart) {
++ rlen++;
++ }
++ if (2 + 3 * rlen + n > w->block_size - w->next) {
++ return -1;
++ }
++ if (restart) {
++ if (w->restart_len == w->restart_cap) {
++ w->restart_cap = w->restart_cap * 2 + 1;
++ w->restarts = realloc(
++ w->restarts, sizeof(uint32_t) * w->restart_cap);
++ }
++
++ w->restarts[w->restart_len++] = w->next;
++ }
++
++ w->next += n;
++ slice_copy(&w->last_key, key);
++ w->entries++;
++ return 0;
++}
++
++int block_writer_finish(struct block_writer *w)
++{
++ int i = 0;
++ for (i = 0; i < w->restart_len; i++) {
++ put_u24(w->buf + w->next, w->restarts[i]);
++ w->next += 3;
++ }
++
++ put_u16(w->buf + w->next, w->restart_len);
++ w->next += 2;
++ put_u24(w->buf + 1 + w->header_off, w->next);
++
++ if (block_writer_type(w) == BLOCK_TYPE_LOG) {
++ int block_header_skip = 4 + w->header_off;
++ struct slice compressed = {};
++ uLongf dest_len = 0, src_len = 0;
++ slice_resize(&compressed, w->next - block_header_skip);
++
++ dest_len = compressed.len;
++ src_len = w->next - block_header_skip;
++
++ if (Z_OK != compress2(compressed.buf, &dest_len,
++ w->buf + block_header_skip, src_len, 9)) {
++ free(slice_yield(&compressed));
++ return ZLIB_ERROR;
++ }
++ memcpy(w->buf + block_header_skip, compressed.buf, dest_len);
++ w->next = dest_len + block_header_skip;
++ }
++ return w->next;
++}
++
++byte block_reader_type(struct block_reader *r)
++{
++ return r->block.data[r->header_off];
+}
+
+int block_reader_init(struct block_reader *br, struct block *block,
-+ uint32_t header_off, uint32_t table_block_size,
-+ int hash_size) {
-+ uint32_t full_block_size = table_block_size;
-+ byte typ = block->data[header_off];
-+ uint32_t sz = get_u24(block->data + header_off + 1);
-+
-+ if (!is_block_type(typ)) {
-+ return FORMAT_ERROR;
-+ }
-+
-+ if (typ == BLOCK_TYPE_LOG) {
-+ struct slice uncompressed = {};
-+ int block_header_skip = 4 + header_off;
-+ uLongf dst_len = sz - block_header_skip;
-+ uLongf src_len = block->len - block_header_skip;
-+
-+ slice_resize(&uncompressed, sz);
-+ memcpy(uncompressed.buf, block->data, block_header_skip);
-+
-+ if (Z_OK != uncompress2(uncompressed.buf + block_header_skip, &dst_len,
-+ block->data + block_header_skip, &src_len)) {
-+ free(slice_yield(&uncompressed));
-+ return ZLIB_ERROR;
-+ }
-+
-+ block_source_return_block(block->source, block);
-+ block->data = uncompressed.buf;
-+ block->len = dst_len; /* XXX: 4 bytes missing? */
-+ block->source = malloc_block_source();
-+ full_block_size = src_len + block_header_skip;
-+ } else if (full_block_size == 0) {
-+ full_block_size = sz;
-+ } else if (sz < full_block_size && sz < block->len && block->data[sz] != 0) {
-+ // If the block is smaller than the full block size,
-+ // it is padded (data followed by '\0') or the next
-+ // block is unaligned.
-+ full_block_size = sz;
-+ }
-+
-+ {
-+ uint16_t restart_count = get_u16(block->data + sz - 2);
-+ uint32_t restart_start = sz - 2 - 3 * restart_count;
-+
-+ byte *restart_bytes = block->data + restart_start;
-+
-+ // transfer ownership.
-+ br->block = *block;
-+ block->data = NULL;
-+ block->len = 0;
-+
-+ br->hash_size = hash_size;
-+ br->block_len = restart_start;
-+ br->full_block_size = full_block_size;
-+ br->header_off = header_off;
-+ br->restart_count = restart_count;
-+ br->restart_bytes = restart_bytes;
-+ }
-+
-+ return 0;
-+}
-+
-+static uint32_t block_reader_restart_offset(struct block_reader *br, int i) {
-+ return get_u24(br->restart_bytes + 3 * i);
-+}
-+
-+void block_reader_start(struct block_reader *br, struct block_iter *it) {
-+ it->br = br;
-+ slice_resize(&it->last_key, 0);
-+ it->next_off = br->header_off + 4;
++ uint32_t header_off, uint32_t table_block_size,
++ int hash_size)
++{
++ uint32_t full_block_size = table_block_size;
++ byte typ = block->data[header_off];
++ uint32_t sz = get_u24(block->data + header_off + 1);
++
++ if (!is_block_type(typ)) {
++ return FORMAT_ERROR;
++ }
++
++ if (typ == BLOCK_TYPE_LOG) {
++ struct slice uncompressed = {};
++ int block_header_skip = 4 + header_off;
++ uLongf dst_len = sz - block_header_skip;
++ uLongf src_len = block->len - block_header_skip;
++
++ slice_resize(&uncompressed, sz);
++ memcpy(uncompressed.buf, block->data, block_header_skip);
++
++ if (Z_OK !=
++ uncompress2(uncompressed.buf + block_header_skip, &dst_len,
++ block->data + block_header_skip, &src_len)) {
++ free(slice_yield(&uncompressed));
++ return ZLIB_ERROR;
++ }
++
++ block_source_return_block(block->source, block);
++ block->data = uncompressed.buf;
++ block->len = dst_len; /* XXX: 4 bytes missing? */
++ block->source = malloc_block_source();
++ full_block_size = src_len + block_header_skip;
++ } else if (full_block_size == 0) {
++ full_block_size = sz;
++ } else if (sz < full_block_size && sz < block->len &&
++ block->data[sz] != 0) {
++ /* If the block is smaller than the full block size, it is
++ padded (data followed by '\0') or the next block is
++ unaligned. */
++ full_block_size = sz;
++ }
++
++ {
++ uint16_t restart_count = get_u16(block->data + sz - 2);
++ uint32_t restart_start = sz - 2 - 3 * restart_count;
++
++ byte *restart_bytes = block->data + restart_start;
++
++ /* transfer ownership. */
++ br->block = *block;
++ block->data = NULL;
++ block->len = 0;
++
++ br->hash_size = hash_size;
++ br->block_len = restart_start;
++ br->full_block_size = full_block_size;
++ br->header_off = header_off;
++ br->restart_count = restart_count;
++ br->restart_bytes = restart_bytes;
++ }
++
++ return 0;
++}
++
++static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
++{
++ return get_u24(br->restart_bytes + 3 * i);
++}
++
++void block_reader_start(struct block_reader *br, struct block_iter *it)
++{
++ it->br = br;
++ slice_resize(&it->last_key, 0);
++ it->next_off = br->header_off + 4;
+}
+
+struct restart_find_args {
-+ struct slice key;
-+ struct block_reader *r;
-+ int error;
++ struct slice key;
++ struct block_reader *r;
++ int error;
+};
+
-+static int restart_key_less(int idx, void *args) {
-+ struct restart_find_args *a = (struct restart_find_args *)args;
-+ uint32_t off = block_reader_restart_offset(a->r, idx);
-+ struct slice in = {
-+ .buf = a->r->block.data + off,
-+ .len = a->r->block_len - off,
-+ };
-+
-+ /* the restart key is verbatim in the block, so this could avoid the
-+ alloc for decoding the key */
-+ struct slice rkey = {};
-+ struct slice last_key = {};
-+ byte unused_extra;
-+ int n = decode_key(&rkey, &unused_extra, last_key, in);
-+ if (n < 0) {
-+ a->error = 1;
-+ return -1;
-+ }
-+
-+ {
-+ int result = slice_compare(a->key, rkey);
-+ free(slice_yield(&rkey));
-+ return result;
-+ }
-+}
-+
-+void block_iter_copy_from(struct block_iter *dest, struct block_iter *src) {
-+ dest->br = src->br;
-+ dest->next_off = src->next_off;
-+ slice_copy(&dest->last_key, src->last_key);
-+}
-+
-+// return < 0 for error, 0 for OK, > 0 for EOF.
-+int block_iter_next(struct block_iter *it, struct record rec) {
-+ if (it->next_off >= it->br->block_len) {
-+ return 1;
-+ }
-+
-+ {
-+ struct slice in = {
-+ .buf = it->br->block.data + it->next_off,
-+ .len = it->br->block_len - it->next_off,
-+ };
-+ struct slice start = in;
-+ struct slice key = {};
-+ byte extra;
-+ int n = decode_key(&key, &extra, it->last_key, in);
-+ if (n < 0) {
-+ return -1;
-+ }
-+
-+ in.buf += n;
-+ in.len -= n;
-+ n = record_decode(rec, key, extra, in, it->br->hash_size);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ in.buf += n;
-+ in.len -= n;
-+
-+ slice_copy(&it->last_key, key);
-+ it->next_off += start.len - in.len;
-+ free(slice_yield(&key));
-+ return 0;
-+ }
-+}
-+
-+int block_reader_first_key(struct block_reader *br, struct slice *key) {
-+ struct slice empty = {};
-+ int off = br->header_off + 4;
-+ struct slice in = {
-+ .buf = br->block.data + off,
-+ .len = br->block_len - off,
-+ };
-+
-+ byte extra = 0;
-+ int n = decode_key(key, &extra, empty, in);
-+ if (n < 0) {
-+ return n;
-+ }
-+ return 0;
-+}
-+
-+int block_iter_seek(struct block_iter *it, struct slice want) {
-+ return block_reader_seek(it->br, it, want);
-+}
-+
-+void block_iter_close(struct block_iter *it) {
-+ free(slice_yield(&it->last_key));
++static int restart_key_less(int idx, void *args)
++{
++ struct restart_find_args *a = (struct restart_find_args *)args;
++ uint32_t off = block_reader_restart_offset(a->r, idx);
++ struct slice in = {
++ .buf = a->r->block.data + off,
++ .len = a->r->block_len - off,
++ };
++
++ /* the restart key is verbatim in the block, so this could avoid the
++ alloc for decoding the key */
++ struct slice rkey = {};
++ struct slice last_key = {};
++ byte unused_extra;
++ int n = decode_key(&rkey, &unused_extra, last_key, in);
++ if (n < 0) {
++ a->error = 1;
++ return -1;
++ }
++
++ {
++ int result = slice_compare(a->key, rkey);
++ free(slice_yield(&rkey));
++ return result;
++ }
++}
++
++void block_iter_copy_from(struct block_iter *dest, struct block_iter *src)
++{
++ dest->br = src->br;
++ dest->next_off = src->next_off;
++ slice_copy(&dest->last_key, src->last_key);
++}
++
++/* return < 0 for error, 0 for OK, > 0 for EOF. */
++int block_iter_next(struct block_iter *it, struct record rec)
++{
++ if (it->next_off >= it->br->block_len) {
++ return 1;
++ }
++
++ {
++ struct slice in = {
++ .buf = it->br->block.data + it->next_off,
++ .len = it->br->block_len - it->next_off,
++ };
++ struct slice start = in;
++ struct slice key = {};
++ byte extra;
++ int n = decode_key(&key, &extra, it->last_key, in);
++ if (n < 0) {
++ return -1;
++ }
++
++ in.buf += n;
++ in.len -= n;
++ n = record_decode(rec, key, extra, in, it->br->hash_size);
++ if (n < 0) {
++ return -1;
++ }
++ in.buf += n;
++ in.len -= n;
++
++ slice_copy(&it->last_key, key);
++ it->next_off += start.len - in.len;
++ free(slice_yield(&key));
++ return 0;
++ }
++}
++
++int block_reader_first_key(struct block_reader *br, struct slice *key)
++{
++ struct slice empty = {};
++ int off = br->header_off + 4;
++ struct slice in = {
++ .buf = br->block.data + off,
++ .len = br->block_len - off,
++ };
++
++ byte extra = 0;
++ int n = decode_key(key, &extra, empty, in);
++ if (n < 0) {
++ return n;
++ }
++ return 0;
++}
++
++int block_iter_seek(struct block_iter *it, struct slice want)
++{
++ return block_reader_seek(it->br, it, want);
++}
++
++void block_iter_close(struct block_iter *it)
++{
++ free(slice_yield(&it->last_key));
+}
+
+int block_reader_seek(struct block_reader *br, struct block_iter *it,
-+ struct slice want) {
-+ struct restart_find_args args = {
-+ .key = want,
-+ .r = br,
-+ };
-+
-+ int i = binsearch(br->restart_count, &restart_key_less, &args);
-+ if (args.error) {
-+ return -1;
-+ }
-+
-+ it->br = br;
-+ if (i > 0) {
-+ i--;
-+ it->next_off = block_reader_restart_offset(br, i);
-+ } else {
-+ it->next_off = br->header_off + 4;
-+ }
-+
-+ {
-+ struct record rec = new_record(block_reader_type(br));
-+ struct slice key = {};
-+ int result = 0;
-+ int err = 0;
-+ struct block_iter next = {};
-+ while (true) {
-+ block_iter_copy_from(&next, it);
-+
-+ err = block_iter_next(&next, rec);
-+ if (err < 0) {
-+ result = -1;
-+ goto exit;
-+ }
-+
-+ record_key(rec, &key);
-+ if (err > 0 || slice_compare(key, want) >= 0) {
-+ result = 0;
-+ goto exit;
-+ }
-+
-+ block_iter_copy_from(it, &next);
-+ }
-+
-+ exit:
-+ free(slice_yield(&key));
-+ free(slice_yield(&next.last_key));
-+ record_clear(rec);
-+ free(record_yield(&rec));
-+
-+ return result;
-+ }
-+}
-+
-+void block_writer_reset(struct block_writer *bw) {
-+ bw->restart_len = 0;
-+ bw->last_key.len = 0;
-+}
-+
-+void block_writer_clear(struct block_writer *bw) {
-+ free(bw->restarts);
-+ bw->restarts = NULL;
-+ free(slice_yield(&bw->last_key));
-+ // the block is not owned.
++ struct slice want)
++{
++ struct restart_find_args args = {
++ .key = want,
++ .r = br,
++ };
++
++ int i = binsearch(br->restart_count, &restart_key_less, &args);
++ if (args.error) {
++ return -1;
++ }
++
++ it->br = br;
++ if (i > 0) {
++ i--;
++ it->next_off = block_reader_restart_offset(br, i);
++ } else {
++ it->next_off = br->header_off + 4;
++ }
++
++ {
++ struct record rec = new_record(block_reader_type(br));
++ struct slice key = {};
++ int result = 0;
++ int err = 0;
++ struct block_iter next = {};
++ while (true) {
++ block_iter_copy_from(&next, it);
++
++ err = block_iter_next(&next, rec);
++ if (err < 0) {
++ result = -1;
++ goto exit;
++ }
++
++ record_key(rec, &key);
++ if (err > 0 || slice_compare(key, want) >= 0) {
++ result = 0;
++ goto exit;
++ }
++
++ block_iter_copy_from(it, &next);
++ }
++
++ exit:
++ free(slice_yield(&key));
++ free(slice_yield(&next.last_key));
++ record_clear(rec);
++ free(record_yield(&rec));
++
++ return result;
++ }
++}
++
++void block_writer_reset(struct block_writer *bw)
++{
++ bw->restart_len = 0;
++ bw->last_key.len = 0;
++}
++
++void block_writer_clear(struct block_writer *bw)
++{
++ free(bw->restarts);
++ bw->restarts = NULL;
++ free(slice_yield(&bw->last_key));
++ /* the block is not owned. */
+}
diff --git a/reftable/block.h b/reftable/block.h
@@ -961,22 +777,22 @@
+#include "reftable.h"
+
+struct block_writer {
-+ byte *buf;
-+ uint32_t block_size;
-+ uint32_t header_off;
-+ int restart_interval;
-+ int hash_size;
-+
-+ uint32_t next;
-+ uint32_t *restarts;
-+ uint32_t restart_len;
-+ uint32_t restart_cap;
-+ struct slice last_key;
-+ int entries;
++ byte *buf;
++ uint32_t block_size;
++ uint32_t header_off;
++ int restart_interval;
++ int hash_size;
++
++ uint32_t next;
++ uint32_t *restarts;
++ uint32_t restart_len;
++ uint32_t restart_cap;
++ struct slice last_key;
++ int entries;
+};
+
+void block_writer_init(struct block_writer *bw, byte typ, byte *buf,
-+ uint32_t block_size, uint32_t header_off, int hash_size);
++ uint32_t block_size, uint32_t header_off, int hash_size);
+byte block_writer_type(struct block_writer *bw);
+int block_writer_add(struct block_writer *w, struct record rec);
+int block_writer_finish(struct block_writer *w);
@@ -984,29 +800,29 @@
+void block_writer_clear(struct block_writer *bw);
+
+struct block_reader {
-+ uint32_t header_off;
-+ struct block block;
-+ int hash_size;
-+
-+ // size of the data, excluding restart data.
-+ uint32_t block_len;
-+ byte *restart_bytes;
-+ uint32_t full_block_size;
-+ uint16_t restart_count;
++ uint32_t header_off;
++ struct block block;
++ int hash_size;
++
++ /* size of the data, excluding restart data. */
++ uint32_t block_len;
++ byte *restart_bytes;
++ uint32_t full_block_size;
++ uint16_t restart_count;
+};
+
+struct block_iter {
-+ struct block_reader *br;
-+ struct slice last_key;
-+ uint32_t next_off;
++ struct block_reader *br;
++ struct slice last_key;
++ uint32_t next_off;
+};
+
+int block_reader_init(struct block_reader *br, struct block *bl,
-+ uint32_t header_off, uint32_t table_block_size,
-+ int hash_size);
++ uint32_t header_off, uint32_t table_block_size,
++ int hash_size);
+void block_reader_start(struct block_reader *br, struct block_iter *it);
+int block_reader_seek(struct block_reader *br, struct block_iter *it,
-+ struct slice want);
++ struct slice want);
+byte block_reader_type(struct block_reader *r);
+int block_reader_first_key(struct block_reader *br, struct slice *key);
+
@@ -1032,7 +848,7 @@
+
+#include "block.h"
+
-+#include <string.h>
++#include "system.h"
+
+#include "basics.h"
+#include "constants.h"
@@ -1041,131 +857,137 @@
+#include "test_framework.h"
+
+struct binsearch_args {
-+ int key;
-+ int *arr;
++ int key;
++ int *arr;
+};
+
-+static int binsearch_func(int i, void *void_args) {
-+ struct binsearch_args *args = (struct binsearch_args *)void_args;
-+
-+ return args->key < args->arr[i];
-+}
-+
-+void test_binsearch() {
-+ int arr[] = {2, 4, 6, 8, 10};
-+ int sz = ARRAYSIZE(arr);
-+ struct binsearch_args args = {
-+ .arr = arr,
-+ };
-+
-+ for (int i = 1; i < 11; i++) {
-+ args.key = i;
-+ int res = binsearch(sz, &binsearch_func, &args);
-+
-+ if (res < sz) {
-+ assert(args.key < arr[res]);
-+ if (res > 0) {
-+ assert(args.key >= arr[res - 1]);
-+ }
-+ } else {
-+ assert(args.key == 10 || args.key == 11);
-+ }
-+ }
-+}
-+
-+void test_block_read_write() {
-+ const int header_off = 21; // random
-+ const int N = 30;
-+ char *names[N];
-+ const int block_size = 1024;
-+ struct block block = {};
-+ block.data = calloc(block_size, 1);
-+ block.len = block_size;
-+
-+ struct block_writer bw = {};
-+ block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size, header_off,
-+ SHA1_SIZE);
-+ struct ref_record ref = {};
-+ struct record rec = {};
-+ record_from_ref(&rec, &ref);
-+
-+ for (int i = 0; i < N; i++) {
-+ char name[100];
-+ sprintf(name, "branch%02d", i);
-+
-+ byte hash[SHA1_SIZE];
-+ memset(hash, i, sizeof(hash));
-+
-+ ref.ref_name = name;
-+ ref.value = hash;
-+ names[i] = strdup(name);
-+ int n = block_writer_add(&bw, rec);
-+ ref.ref_name = NULL;
-+ ref.value = NULL;
-+ assert(n == 0);
-+ }
-+
-+ int n = block_writer_finish(&bw);
-+ assert(n > 0);
-+
-+ block_writer_clear(&bw);
-+
-+ struct block_reader br = {};
-+ block_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);
-+
-+ struct block_iter it = {};
-+ block_reader_start(&br, &it);
-+
-+ int j = 0;
-+ while (true) {
-+ int r = block_iter_next(&it, rec);
-+ assert(r >= 0);
-+ if (r > 0) {
-+ break;
-+ }
-+ assert_streq(names[j], ref.ref_name);
-+ j++;
-+ }
-+
-+ record_clear(rec);
-+ block_iter_close(&it);
-+
-+ struct slice want = {};
-+ for (int i = 0; i < N; i++) {
-+ slice_set_string(&want, names[i]);
-+
-+ struct block_iter it = {};
-+ int n = block_reader_seek(&br, &it, want);
-+ assert(n == 0);
-+
-+ n = block_iter_next(&it, rec);
-+ assert(n == 0);
-+
-+ assert_streq(names[i], ref.ref_name);
-+
-+ want.len--;
-+ n = block_reader_seek(&br, &it, want);
-+ assert(n == 0);
-+
-+ n = block_iter_next(&it, rec);
-+ assert(n == 0);
-+ assert_streq(names[10 * (i / 10)], ref.ref_name);
-+
-+ block_iter_close(&it);
-+ }
-+
-+ record_clear(rec);
-+ free(block.data);
-+ free(slice_yield(&want));
-+ for (int i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+}
-+
-+int main() {
-+ add_test_case("binsearch", &test_binsearch);
-+ add_test_case("block_read_write", &test_block_read_write);
-+ test_main();
++static int binsearch_func(int i, void *void_args)
++{
++ struct binsearch_args *args = (struct binsearch_args *)void_args;
++
++ return args->key < args->arr[i];
++}
++
++void test_binsearch()
++{
++ int arr[] = { 2, 4, 6, 8, 10 };
++ int sz = ARRAYSIZE(arr);
++ struct binsearch_args args = {
++ .arr = arr,
++ };
++
++ int i = 0;
++ for (i = 1; i < 11; i++) {
++ args.key = i;
++ int res = binsearch(sz, &binsearch_func, &args);
++
++ if (res < sz) {
++ assert(args.key < arr[res]);
++ if (res > 0) {
++ assert(args.key >= arr[res - 1]);
++ }
++ } else {
++ assert(args.key == 10 || args.key == 11);
++ }
++ }
++}
++
++void test_block_read_write()
++{
++ const int header_off = 21; /* random */
++ const int N = 30;
++ char *names[N];
++ const int block_size = 1024;
++ struct block block = {};
++ block.data = calloc(block_size, 1);
++ block.len = block_size;
++
++ struct block_writer bw = {};
++ block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
++ header_off, SHA1_SIZE);
++ struct ref_record ref = {};
++ struct record rec = {};
++ record_from_ref(&rec, &ref);
++
++ int i = 0;
++ for (i = 0; i < N; i++) {
++ char name[100];
++ snprintf(name, sizeof(name), "branch%02d", i);
++
++ byte hash[SHA1_SIZE];
++ memset(hash, i, sizeof(hash));
++
++ ref.ref_name = name;
++ ref.value = hash;
++ names[i] = strdup(name);
++ int n = block_writer_add(&bw, rec);
++ ref.ref_name = NULL;
++ ref.value = NULL;
++ assert(n == 0);
++ }
++
++ int n = block_writer_finish(&bw);
++ assert(n > 0);
++
++ block_writer_clear(&bw);
++
++ struct block_reader br = {};
++ block_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);
++
++ struct block_iter it = {};
++ block_reader_start(&br, &it);
++
++ int j = 0;
++ while (true) {
++ int r = block_iter_next(&it, rec);
++ assert(r >= 0);
++ if (r > 0) {
++ break;
++ }
++ assert_streq(names[j], ref.ref_name);
++ j++;
++ }
++
++ record_clear(rec);
++ block_iter_close(&it);
++
++ struct slice want = {};
++ for (i = 0; i < N; i++) {
++ slice_set_string(&want, names[i]);
++
++ struct block_iter it = {};
++ int n = block_reader_seek(&br, &it, want);
++ assert(n == 0);
++
++ n = block_iter_next(&it, rec);
++ assert(n == 0);
++
++ assert_streq(names[i], ref.ref_name);
++
++ want.len--;
++ n = block_reader_seek(&br, &it, want);
++ assert(n == 0);
++
++ n = block_iter_next(&it, rec);
++ assert(n == 0);
++ assert_streq(names[10 * (i / 10)], ref.ref_name);
++
++ block_iter_close(&it);
++ }
++
++ record_clear(rec);
++ free(block.data);
++ free(slice_yield(&want));
++ for (i = 0; i < N; i++) {
++ free(names[i]);
++ }
++}
++
++int main()
++{
++ add_test_case("binsearch", &test_binsearch);
++ add_test_case("block_read_write", &test_block_read_write);
++ test_main();
+}
diff --git a/reftable/blocksource.h b/reftable/blocksource.h
@@ -1188,7 +1010,7 @@
+
+uint64_t block_source_size(struct block_source source);
+int block_source_read_block(struct block_source source, struct block *dest,
-+ uint64_t off, uint32_t size);
++ uint64_t off, uint32_t size);
+void block_source_return_block(struct block_source source, struct block *ret);
+void block_source_close(struct block_source source);
+
@@ -1197,6 +1019,13 @@
diff --git a/reftable/bytes.c b/reftable/bytes.c
new file mode 100644
+ diff --git a/reftable/config.h b/reftable/config.h
+ new file mode 100644
+ --- /dev/null
+ +++ b/reftable/config.h
+@@
++/* empty */
+
diff --git a/reftable/constants.h b/reftable/constants.h
new file mode 100644
--- /dev/null
@@ -1235,107 +1064,102 @@
--- /dev/null
+++ b/reftable/dump.c
@@
-+// Copyright 2020 Google Inc. All rights reserved.
-+//
-+// Licensed under the Apache License, Version 2.0 (the "License");
-+// you may not use this file except in compliance with the License.
-+// You may obtain a copy of the License at
-+//
-+// http://www.apache.org/licenses/LICENSE-2.0
-+//
-+// Unless required by applicable law or agreed to in writing, software
-+// distributed under the License is distributed on an "AS IS" BASIS,
-+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-+// See the License for the specific language governing permissions and
-+// limitations under the License.
++/*
++Copyright 2020 Google LLC
+
-+#include <stdio.h>
-+#include <string.h>
-+#include <unistd.h>
++Use of this source code is governed by a BSD-style
++license that can be found in the LICENSE file or at
++https://developers.google.com/open-source/licenses/bsd
++*/
++
++#include "system.h"
+
+#include "reftable.h"
+
-+static int dump_table(const char *tablename) {
-+ struct block_source src = {};
-+ int err = block_source_from_file(&src, tablename);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ struct reader *r = NULL;
-+ err = new_reader(&r, src, tablename);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ {
-+ struct iterator it = {};
-+ err = reader_seek_ref(r, &it, "");
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ struct ref_record ref = {};
-+ while (1) {
-+ err = iterator_next_ref(it, &ref);
-+ if (err > 0) {
-+ break;
-+ }
-+ if (err < 0) {
-+ return err;
-+ }
-+ ref_record_print(&ref, 20);
-+ }
-+ iterator_destroy(&it);
-+ ref_record_clear(&ref);
-+ }
-+
-+ {
-+ struct iterator it = {};
-+ err = reader_seek_log(r, &it, "");
-+ if (err < 0) {
-+ return err;
-+ }
-+ struct log_record log = {};
-+ while (1) {
-+ err = iterator_next_log(it, &log);
-+ if (err > 0) {
-+ break;
-+ }
-+ if (err < 0) {
-+ return err;
-+ }
-+ log_record_print(&log, 20);
-+ }
-+ iterator_destroy(&it);
-+ log_record_clear(&log);
-+ }
-+ return 0;
-+}
-+
-+int main(int argc, char *argv[]) {
-+ int opt;
-+ const char *table = NULL;
-+ while ((opt = getopt(argc, argv, "t:")) != -1) {
-+ switch (opt) {
-+ case 't':
-+ table = strdup(optarg);
-+ break;
-+ case '?':
-+ printf("usage: %s [-table tablefile]\n", argv[0]);
-+ return 2;
-+ break;
-+ }
-+ }
-+
-+ if (table != NULL) {
-+ int err = dump_table(table);
-+ if (err < 0) {
-+ fprintf(stderr, "%s: %s: %s\n", argv[0], table, error_str(err));
-+ return 1;
-+ }
-+ }
-+ return 0;
++static int dump_table(const char *tablename)
++{
++ struct block_source src = {};
++ int err = block_source_from_file(&src, tablename);
++ if (err < 0) {
++ return err;
++ }
++
++ struct reader *r = NULL;
++ err = new_reader(&r, src, tablename);
++ if (err < 0) {
++ return err;
++ }
++
++ {
++ struct iterator it = {};
++ err = reader_seek_ref(r, &it, "");
++ if (err < 0) {
++ return err;
++ }
++
++ struct ref_record ref = {};
++ while (1) {
++ err = iterator_next_ref(it, &ref);
++ if (err > 0) {
++ break;
++ }
++ if (err < 0) {
++ return err;
++ }
++ ref_record_print(&ref, 20);
++ }
++ iterator_destroy(&it);
++ ref_record_clear(&ref);
++ }
++
++ {
++ struct iterator it = {};
++ err = reader_seek_log(r, &it, "");
++ if (err < 0) {
++ return err;
++ }
++ struct log_record log = {};
++ while (1) {
++ err = iterator_next_log(it, &log);
++ if (err > 0) {
++ break;
++ }
++ if (err < 0) {
++ return err;
++ }
++ log_record_print(&log, 20);
++ }
++ iterator_destroy(&it);
++ log_record_clear(&log);
++ }
++ return 0;
++}
++
++int main(int argc, char *argv[])
++{
++ int opt;
++ const char *table = NULL;
++ while ((opt = getopt(argc, argv, "t:")) != -1) {
++ switch (opt) {
++ case 't':
++ table = strdup(optarg);
++ break;
++ case '?':
++ printf("usage: %s [-table tablefile]\n", argv[0]);
++ return 2;
++ break;
++ }
++ }
++
++ if (table != NULL) {
++ int err = dump_table(table);
++ if (err < 0) {
++ fprintf(stderr, "%s: %s: %s\n", argv[0], table,
++ error_str(err));
++ return 1;
++ }
++ }
++ return 0;
+}
diff --git a/reftable/file.c b/reftable/file.c
@@ -1351,15 +1175,7 @@
+https://developers.google.com/open-source/licenses/bsd
+*/
+
-+#include <assert.h>
-+#include <errno.h>
-+#include <fcntl.h>
-+#include <stdio.h>
-+#include <stdlib.h>
-+#include <string.h>
-+#include <sys/stat.h>
-+#include <sys/types.h>
-+#include <unistd.h>
++#include "system.h"
+
+#include "block.h"
+#include "iter.h"
@@ -1368,78 +1184,85 @@
+#include "tree.h"
+
+struct file_block_source {
-+ int fd;
-+ uint64_t size;
++ int fd;
++ uint64_t size;
+};
+
-+static uint64_t file_size(void *b) {
-+ return ((struct file_block_source *)b)->size;
++static uint64_t file_size(void *b)
++{
++ return ((struct file_block_source *)b)->size;
+}
+
-+static void file_return_block(void *b, struct block *dest) {
-+ memset(dest->data, 0xff, dest->len);
-+ free(dest->data);
++static void file_return_block(void *b, struct block *dest)
++{
++ memset(dest->data, 0xff, dest->len);
++ free(dest->data);
+}
+
-+static void file_close(void *b) {
-+ int fd = ((struct file_block_source *)b)->fd;
-+ if (fd > 0) {
-+ close(fd);
-+ ((struct file_block_source *)b)->fd = 0;
-+ }
++static void file_close(void *b)
++{
++ int fd = ((struct file_block_source *)b)->fd;
++ if (fd > 0) {
++ close(fd);
++ ((struct file_block_source *)b)->fd = 0;
++ }
+
-+ free(b);
++ free(b);
+}
+
+static int file_read_block(void *v, struct block *dest, uint64_t off,
-+ uint32_t size) {
-+ struct file_block_source *b = (struct file_block_source *)v;
-+ assert(off + size <= b->size);
-+ dest->data = malloc(size);
-+ if (pread(b->fd, dest->data, size, off) != size) {
-+ return -1;
-+ }
-+ dest->len = size;
-+ return size;
++ uint32_t size)
++{
++ struct file_block_source *b = (struct file_block_source *)v;
++ assert(off + size <= b->size);
++ dest->data = malloc(size);
++ if (pread(b->fd, dest->data, size, off) != size) {
++ return -1;
++ }
++ dest->len = size;
++ return size;
+}
+
+struct block_source_vtable file_vtable = {
-+ .size = &file_size,
-+ .read_block = &file_read_block,
-+ .return_block = &file_return_block,
-+ .close = &file_close,
++ .size = &file_size,
++ .read_block = &file_read_block,
++ .return_block = &file_return_block,
++ .close = &file_close,
+};
+
-+int block_source_from_file(struct block_source *bs, const char *name) {
-+ struct stat st = {};
-+ int err = 0;
-+ int fd = open(name, O_RDONLY);
-+ if (fd < 0) {
-+ if (errno == ENOENT) {
-+ return NOT_EXIST_ERROR;
-+ }
-+ return -1;
-+ }
-+
-+ err = fstat(fd, &st);
-+ if (err < 0) {
-+ return -1;
-+ }
-+
-+ {
-+ struct file_block_source *p = calloc(sizeof(struct file_block_source), 1);
-+ p->size = st.st_size;
-+ p->fd = fd;
-+
-+ bs->ops = &file_vtable;
-+ bs->arg = p;
-+ }
-+ return 0;
-+}
-+
-+int fd_writer(void *arg, byte *data, int sz) {
-+ int *fdp = (int *)arg;
-+ return write(*fdp, data, sz);
++int block_source_from_file(struct block_source *bs, const char *name)
++{
++ struct stat st = {};
++ int err = 0;
++ int fd = open(name, O_RDONLY);
++ if (fd < 0) {
++ if (errno == ENOENT) {
++ return NOT_EXIST_ERROR;
++ }
++ return -1;
++ }
++
++ err = fstat(fd, &st);
++ if (err < 0) {
++ return -1;
++ }
++
++ {
++ struct file_block_source *p =
++ calloc(sizeof(struct file_block_source), 1);
++ p->size = st.st_size;
++ p->fd = fd;
++
++ bs->ops = &file_vtable;
++ bs->arg = p;
++ }
++ return 0;
++}
++
++int fd_writer(void *arg, byte *data, int sz)
++{
++ int *fdp = (int *)arg;
++ return write(*fdp, data, sz);
+}
diff --git a/reftable/iter.c b/reftable/iter.c
@@ -1457,206 +1280,225 @@
+
+#include "iter.h"
+
-+#include <stdlib.h>
-+#include <string.h>
++#include "system.h"
+
+#include "block.h"
+#include "constants.h"
+#include "reader.h"
+#include "reftable.h"
+
-+bool iterator_is_null(struct iterator it) { return it.ops == NULL; }
++bool iterator_is_null(struct iterator it)
++{
++ return it.ops == NULL;
++}
+
-+static int empty_iterator_next(void *arg, struct record rec) { return 1; }
++static int empty_iterator_next(void *arg, struct record rec)
++{
++ return 1;
++}
+
-+static void empty_iterator_close(void *arg) {}
++static void empty_iterator_close(void *arg)
++{
++}
+
+struct iterator_vtable empty_vtable = {
-+ .next = &empty_iterator_next,
-+ .close = &empty_iterator_close,
++ .next = &empty_iterator_next,
++ .close = &empty_iterator_close,
+};
+
-+void iterator_set_empty(struct iterator *it) {
-+ it->iter_arg = NULL;
-+ it->ops = &empty_vtable;
++void iterator_set_empty(struct iterator *it)
++{
++ it->iter_arg = NULL;
++ it->ops = &empty_vtable;
+}
+
-+int iterator_next(struct iterator it, struct record rec) {
-+ return it.ops->next(it.iter_arg, rec);
++int iterator_next(struct iterator it, struct record rec)
++{
++ return it.ops->next(it.iter_arg, rec);
+}
+
-+void iterator_destroy(struct iterator *it) {
-+ if (it->ops == NULL) {
-+ return;
-+ }
-+ it->ops->close(it->iter_arg);
-+ it->ops = NULL;
-+ free(it->iter_arg);
-+ it->iter_arg = NULL;
++void iterator_destroy(struct iterator *it)
++{
++ if (it->ops == NULL) {
++ return;
++ }
++ it->ops->close(it->iter_arg);
++ it->ops = NULL;
++ free(it->iter_arg);
++ it->iter_arg = NULL;
+}
+
-+int iterator_next_ref(struct iterator it, struct ref_record *ref) {
-+ struct record rec = {};
-+ record_from_ref(&rec, ref);
-+ return iterator_next(it, rec);
++int iterator_next_ref(struct iterator it, struct ref_record *ref)
++{
++ struct record rec = {};
++ record_from_ref(&rec, ref);
++ return iterator_next(it, rec);
+}
+
-+int iterator_next_log(struct iterator it, struct log_record *log) {
-+ struct record rec = {};
-+ record_from_log(&rec, log);
-+ return iterator_next(it, rec);
++int iterator_next_log(struct iterator it, struct log_record *log)
++{
++ struct record rec = {};
++ record_from_log(&rec, log);
++ return iterator_next(it, rec);
+}
+
-+static void filtering_ref_iterator_close(void *iter_arg) {
-+ struct filtering_ref_iterator *fri =
-+ (struct filtering_ref_iterator *)iter_arg;
-+ free(slice_yield(&fri->oid));
-+ iterator_destroy(&fri->it);
++static void filtering_ref_iterator_close(void *iter_arg)
++{
++ struct filtering_ref_iterator *fri =
++ (struct filtering_ref_iterator *)iter_arg;
++ free(slice_yield(&fri->oid));
++ iterator_destroy(&fri->it);
+}
+
-+static int filtering_ref_iterator_next(void *iter_arg, struct record rec) {
-+ struct filtering_ref_iterator *fri =
-+ (struct filtering_ref_iterator *)iter_arg;
-+ struct ref_record *ref = (struct ref_record *)rec.data;
++static int filtering_ref_iterator_next(void *iter_arg, struct record rec)
++{
++ struct filtering_ref_iterator *fri =
++ (struct filtering_ref_iterator *)iter_arg;
++ struct ref_record *ref = (struct ref_record *)rec.data;
+
-+ while (true) {
-+ int err = iterator_next_ref(fri->it, ref);
-+ if (err != 0) {
-+ return err;
-+ }
++ while (true) {
++ int err = iterator_next_ref(fri->it, ref);
++ if (err != 0) {
++ return err;
++ }
+
-+ if (fri->double_check) {
-+ struct iterator it = {};
++ if (fri->double_check) {
++ struct iterator it = {};
+
-+ int err = reader_seek_ref(fri->r, &it, ref->ref_name);
-+ if (err == 0) {
-+ err = iterator_next_ref(it, ref);
-+ }
++ int err = reader_seek_ref(fri->r, &it, ref->ref_name);
++ if (err == 0) {
++ err = iterator_next_ref(it, ref);
++ }
+
-+ iterator_destroy(&it);
++ iterator_destroy(&it);
+
-+ if (err < 0) {
-+ return err;
-+ }
++ if (err < 0) {
++ return err;
++ }
+
-+ if (err > 0) {
-+ continue;
-+ }
-+ }
++ if (err > 0) {
++ continue;
++ }
++ }
+
-+ if ((ref->target_value != NULL &&
-+ 0 == memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||
-+ (ref->value != NULL &&
-+ 0 == memcmp(fri->oid.buf, ref->value, fri->oid.len))) {
-+ return 0;
-+ }
-+ }
++ if ((ref->target_value != NULL &&
++ !memcmp(fri->oid.buf, ref->target_value, fri->oid.len)) ||
++ (ref->value != NULL &&
++ !memcmp(fri->oid.buf, ref->value, fri->oid.len))) {
++ return 0;
++ }
++ }
+}
+
+struct iterator_vtable filtering_ref_iterator_vtable = {
-+ .next = &filtering_ref_iterator_next,
-+ .close = &filtering_ref_iterator_close,
++ .next = &filtering_ref_iterator_next,
++ .close = &filtering_ref_iterator_close,
+};
+
+void iterator_from_filtering_ref_iterator(struct iterator *it,
-+ struct filtering_ref_iterator *fri) {
-+ it->iter_arg = fri;
-+ it->ops = &filtering_ref_iterator_vtable;
-+}
-+
-+static void indexed_table_ref_iter_close(void *p) {
-+ struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;
-+ block_iter_close(&it->cur);
-+ reader_return_block(it->r, &it->block_reader.block);
-+ free(slice_yield(&it->oid));
-+}
-+
-+static int indexed_table_ref_iter_next_block(
-+ struct indexed_table_ref_iter *it) {
-+ if (it->offset_idx == it->offset_len) {
-+ it->finished = true;
-+ return 1;
-+ }
-+
-+ reader_return_block(it->r, &it->block_reader.block);
-+
-+ {
-+ uint64_t off = it->offsets[it->offset_idx++];
-+ int err =
-+ reader_init_block_reader(it->r, &it->block_reader, off, BLOCK_TYPE_REF);
-+ if (err < 0) {
-+ return err;
-+ }
-+ if (err > 0) {
-+ // indexed block does not exist.
-+ return FORMAT_ERROR;
-+ }
-+ }
-+ block_reader_start(&it->block_reader, &it->cur);
-+ return 0;
-+}
-+
-+static int indexed_table_ref_iter_next(void *p, struct record rec) {
-+ struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;
-+ struct ref_record *ref = (struct ref_record *)rec.data;
-+
-+ while (true) {
-+ int err = block_iter_next(&it->cur, rec);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ if (err > 0) {
-+ err = indexed_table_ref_iter_next_block(it);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ if (it->finished) {
-+ return 1;
-+ }
-+ continue;
-+ }
-+
-+ if (0 == memcmp(it->oid.buf, ref->target_value, it->oid.len) ||
-+ 0 == memcmp(it->oid.buf, ref->value, it->oid.len)) {
-+ return 0;
-+ }
-+ }
++ struct filtering_ref_iterator *fri)
++{
++ it->iter_arg = fri;
++ it->ops = &filtering_ref_iterator_vtable;
++}
++
++static void indexed_table_ref_iter_close(void *p)
++{
++ struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;
++ block_iter_close(&it->cur);
++ reader_return_block(it->r, &it->block_reader.block);
++ free(slice_yield(&it->oid));
++}
++
++static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
++{
++ if (it->offset_idx == it->offset_len) {
++ it->finished = true;
++ return 1;
++ }
++
++ reader_return_block(it->r, &it->block_reader.block);
++
++ {
++ uint64_t off = it->offsets[it->offset_idx++];
++ int err = reader_init_block_reader(it->r, &it->block_reader,
++ off, BLOCK_TYPE_REF);
++ if (err < 0) {
++ return err;
++ }
++ if (err > 0) {
++ /* indexed block does not exist. */
++ return FORMAT_ERROR;
++ }
++ }
++ block_reader_start(&it->block_reader, &it->cur);
++ return 0;
++}
++
++static int indexed_table_ref_iter_next(void *p, struct record rec)
++{
++ struct indexed_table_ref_iter *it = (struct indexed_table_ref_iter *)p;
++ struct ref_record *ref = (struct ref_record *)rec.data;
++
++ while (true) {
++ int err = block_iter_next(&it->cur, rec);
++ if (err < 0) {
++ return err;
++ }
++
++ if (err > 0) {
++ err = indexed_table_ref_iter_next_block(it);
++ if (err < 0) {
++ return err;
++ }
++
++ if (it->finished) {
++ return 1;
++ }
++ continue;
++ }
++
++ if (!memcmp(it->oid.buf, ref->target_value, it->oid.len) ||
++ !memcmp(it->oid.buf, ref->value, it->oid.len)) {
++ return 0;
++ }
++ }
+}
+
+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,
-+ struct reader *r, byte *oid, int oid_len,
-+ uint64_t *offsets, int offset_len) {
-+ struct indexed_table_ref_iter *itr =
-+ calloc(sizeof(struct indexed_table_ref_iter), 1);
-+ int err = 0;
++ struct reader *r, byte *oid, int oid_len,
++ uint64_t *offsets, int offset_len)
++{
++ struct indexed_table_ref_iter *itr =
++ calloc(sizeof(struct indexed_table_ref_iter), 1);
++ int err = 0;
+
-+ itr->r = r;
-+ slice_resize(&itr->oid, oid_len);
-+ memcpy(itr->oid.buf, oid, oid_len);
++ itr->r = r;
++ slice_resize(&itr->oid, oid_len);
++ memcpy(itr->oid.buf, oid, oid_len);
+
-+ itr->offsets = offsets;
-+ itr->offset_len = offset_len;
++ itr->offsets = offsets;
++ itr->offset_len = offset_len;
+
-+ err = indexed_table_ref_iter_next_block(itr);
-+ if (err < 0) {
-+ free(itr);
-+ } else {
-+ *dest = itr;
-+ }
-+ return err;
++ err = indexed_table_ref_iter_next_block(itr);
++ if (err < 0) {
++ free(itr);
++ } else {
++ *dest = itr;
++ }
++ return err;
+}
+
+struct iterator_vtable indexed_table_ref_iter_vtable = {
-+ .next = &indexed_table_ref_iter_next,
-+ .close = &indexed_table_ref_iter_close,
++ .next = &indexed_table_ref_iter_next,
++ .close = &indexed_table_ref_iter_close,
+};
+
+void iterator_from_indexed_table_ref_iter(struct iterator *it,
-+ struct indexed_table_ref_iter *itr) {
-+ it->iter_arg = itr;
-+ it->ops = &indexed_table_ref_iter_vtable;
++ struct indexed_table_ref_iter *itr)
++{
++ it->iter_arg = itr;
++ it->ops = &indexed_table_ref_iter_vtable;
+}
diff --git a/reftable/iter.h b/reftable/iter.h
@@ -1680,8 +1522,8 @@
+#include "slice.h"
+
+struct iterator_vtable {
-+ int (*next)(void *iter_arg, struct record rec);
-+ void (*close)(void *iter_arg);
++ int (*next)(void *iter_arg, struct record rec);
++ void (*close)(void *iter_arg);
+};
+
+void iterator_set_empty(struct iterator *it);
@@ -1689,35 +1531,35 @@
+bool iterator_is_null(struct iterator it);
+
+struct filtering_ref_iterator {
-+ struct reader *r;
-+ struct slice oid;
-+ bool double_check;
-+ struct iterator it;
++ struct reader *r;
++ struct slice oid;
++ bool double_check;
++ struct iterator it;
+};
+
+void iterator_from_filtering_ref_iterator(struct iterator *,
-+ struct filtering_ref_iterator *);
++ struct filtering_ref_iterator *);
+
+struct indexed_table_ref_iter {
-+ struct reader *r;
-+ struct slice oid;
-+
-+ // mutable
-+ uint64_t *offsets;
-+
-+ // Points to the next offset to read.
-+ int offset_idx;
-+ int offset_len;
-+ struct block_reader block_reader;
-+ struct block_iter cur;
-+ bool finished;
++ struct reader *r;
++ struct slice oid;
++
++ /* mutable */
++ uint64_t *offsets;
++
++ /* Points to the next offset to read. */
++ int offset_idx;
++ int offset_len;
++ struct block_reader block_reader;
++ struct block_iter cur;
++ bool finished;
+};
+
+void iterator_from_indexed_table_ref_iter(struct iterator *it,
-+ struct indexed_table_ref_iter *itr);
++ struct indexed_table_ref_iter *itr);
+int new_indexed_table_ref_iter(struct indexed_table_ref_iter **dest,
-+ struct reader *r, byte *oid, int oid_len,
-+ uint64_t *offsets, int offset_len);
++ struct reader *r, byte *oid, int oid_len,
++ uint64_t *offsets, int offset_len);
+
+#endif
@@ -1736,257 +1578,283 @@
+
+#include "merged.h"
+
-+#include <stdlib.h>
++#include "system.h"
+
+#include "constants.h"
+#include "iter.h"
+#include "pq.h"
+#include "reader.h"
+
-+static int merged_iter_init(struct merged_iter *mi) {
-+ for (int i = 0; i < mi->stack_len; i++) {
-+ struct record rec = new_record(mi->typ);
-+ int err = iterator_next(mi->stack[i], rec);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ if (err > 0) {
-+ iterator_destroy(&mi->stack[i]);
-+ record_clear(rec);
-+ free(record_yield(&rec));
-+ } else {
-+ struct pq_entry e = {
-+ .rec = rec,
-+ .index = i,
-+ };
-+ merged_iter_pqueue_add(&mi->pq, e);
-+ }
-+ }
-+
-+ return 0;
-+}
-+
-+static void merged_iter_close(void *p) {
-+ struct merged_iter *mi = (struct merged_iter *)p;
-+ merged_iter_pqueue_clear(&mi->pq);
-+ for (int i = 0; i < mi->stack_len; i++) {
-+ iterator_destroy(&mi->stack[i]);
-+ }
-+ free(mi->stack);
-+}
-+
-+static int merged_iter_advance_subiter(struct merged_iter *mi, int idx) {
-+ if (iterator_is_null(mi->stack[idx])) {
-+ return 0;
-+ }
-+
-+ {
-+ struct record rec = new_record(mi->typ);
-+ struct pq_entry e = {
-+ .rec = rec,
-+ .index = idx,
-+ };
-+ int err = iterator_next(mi->stack[idx], rec);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ if (err > 0) {
-+ iterator_destroy(&mi->stack[idx]);
-+ record_clear(rec);
-+ free(record_yield(&rec));
-+ return 0;
-+ }
-+
-+ merged_iter_pqueue_add(&mi->pq, e);
-+ }
-+ return 0;
-+}
-+
-+static int merged_iter_next(struct merged_iter *mi, struct record rec) {
-+ struct slice entry_key = {};
-+ struct pq_entry entry = merged_iter_pqueue_remove(&mi->pq);
-+ int err = merged_iter_advance_subiter(mi, entry.index);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ record_key(entry.rec, &entry_key);
-+ while (!merged_iter_pqueue_is_empty(mi->pq)) {
-+ struct pq_entry top = merged_iter_pqueue_top(mi->pq);
-+ struct slice k = {};
-+ int err = 0, cmp = 0;
-+
-+ record_key(top.rec, &k);
-+
-+ cmp = slice_compare(k, entry_key);
-+ free(slice_yield(&k));
-+
-+ if (cmp > 0) {
-+ break;
-+ }
-+
-+ merged_iter_pqueue_remove(&mi->pq);
-+ err = merged_iter_advance_subiter(mi, top.index);
-+ if (err < 0) {
-+ return err;
-+ }
-+ record_clear(top.rec);
-+ free(record_yield(&top.rec));
-+ }
-+
-+ record_copy_from(rec, entry.rec, mi->hash_size);
-+ record_clear(entry.rec);
-+ free(record_yield(&entry.rec));
-+ free(slice_yield(&entry_key));
-+ return 0;
-+}
-+
-+static int merged_iter_next_void(void *p, struct record rec) {
-+ struct merged_iter *mi = (struct merged_iter *)p;
-+ if (merged_iter_pqueue_is_empty(mi->pq)) {
-+ return 1;
-+ }
-+
-+ return merged_iter_next(mi, rec);
++static int merged_iter_init(struct merged_iter *mi)
++{
++ int i = 0;
++ for (i = 0; i < mi->stack_len; i++) {
++ struct record rec = new_record(mi->typ);
++ int err = iterator_next(mi->stack[i], rec);
++ if (err < 0) {
++ return err;
++ }
++
++ if (err > 0) {
++ iterator_destroy(&mi->stack[i]);
++ record_clear(rec);
++ free(record_yield(&rec));
++ } else {
++ struct pq_entry e = {
++ .rec = rec,
++ .index = i,
++ };
++ merged_iter_pqueue_add(&mi->pq, e);
++ }
++ }
++
++ return 0;
++}
++
++static void merged_iter_close(void *p)
++{
++ struct merged_iter *mi = (struct merged_iter *)p;
++ int i = 0;
++ merged_iter_pqueue_clear(&mi->pq);
++ for (i = 0; i < mi->stack_len; i++) {
++ iterator_destroy(&mi->stack[i]);
++ }
++ free(mi->stack);
++}
++
++static int merged_iter_advance_subiter(struct merged_iter *mi, int idx)
++{
++ if (iterator_is_null(mi->stack[idx])) {
++ return 0;
++ }
++
++ {
++ struct record rec = new_record(mi->typ);
++ struct pq_entry e = {
++ .rec = rec,
++ .index = idx,
++ };
++ int err = iterator_next(mi->stack[idx], rec);
++ if (err < 0) {
++ return err;
++ }
++
++ if (err > 0) {
++ iterator_destroy(&mi->stack[idx]);
++ record_clear(rec);
++ free(record_yield(&rec));
++ return 0;
++ }
++
++ merged_iter_pqueue_add(&mi->pq, e);
++ }
++ return 0;
++}
++
++static int merged_iter_next(struct merged_iter *mi, struct record rec)
++{
++ struct slice entry_key = {};
++ struct pq_entry entry = merged_iter_pqueue_remove(&mi->pq);
++ int err = merged_iter_advance_subiter(mi, entry.index);
++ if (err < 0) {
++ return err;
++ }
++
++ record_key(entry.rec, &entry_key);
++ while (!merged_iter_pqueue_is_empty(mi->pq)) {
++ struct pq_entry top = merged_iter_pqueue_top(mi->pq);
++ struct slice k = {};
++ int err = 0, cmp = 0;
++
++ record_key(top.rec, &k);
++
++ cmp = slice_compare(k, entry_key);
++ free(slice_yield(&k));
++
++ if (cmp > 0) {
++ break;
++ }
++
++ merged_iter_pqueue_remove(&mi->pq);
++ err = merged_iter_advance_subiter(mi, top.index);
++ if (err < 0) {
++ return err;
++ }
++ record_clear(top.rec);
++ free(record_yield(&top.rec));
++ }
++
++ record_copy_from(rec, entry.rec, mi->hash_size);
++ record_clear(entry.rec);
++ free(record_yield(&entry.rec));
++ free(slice_yield(&entry_key));
++ return 0;
++}
++
++static int merged_iter_next_void(void *p, struct record rec)
++{
++ struct merged_iter *mi = (struct merged_iter *)p;
++ if (merged_iter_pqueue_is_empty(mi->pq)) {
++ return 1;
++ }
++
++ return merged_iter_next(mi, rec);
+}
+
+struct iterator_vtable merged_iter_vtable = {
-+ .next = &merged_iter_next_void,
-+ .close = &merged_iter_close,
++ .next = &merged_iter_next_void,
++ .close = &merged_iter_close,
+};
+
+static void iterator_from_merged_iter(struct iterator *it,
-+ struct merged_iter *mi) {
-+ it->iter_arg = mi;
-+ it->ops = &merged_iter_vtable;
-+}
-+
-+int new_merged_table(struct merged_table **dest, struct reader **stack, int n) {
-+ uint64_t last_max = 0;
-+ uint64_t first_min = 0;
-+ for (int i = 0; i < n; i++) {
-+ struct reader *r = stack[i];
-+ if (i > 0 && last_max >= reader_min_update_index(r)) {
-+ return FORMAT_ERROR;
-+ }
-+ if (i == 0) {
-+ first_min = reader_min_update_index(r);
-+ }
-+
-+ last_max = reader_max_update_index(r);
-+ }
-+
-+ {
-+ struct merged_table m = {
-+ .stack = stack,
-+ .stack_len = n,
-+ .min = first_min,
-+ .max = last_max,
-+ .hash_size = SHA1_SIZE,
-+ };
-+
-+ *dest = calloc(sizeof(struct merged_table), 1);
-+ **dest = m;
-+ }
-+ return 0;
-+}
-+
-+void merged_table_close(struct merged_table *mt) {
-+ for (int i = 0; i < mt->stack_len; i++) {
-+ reader_free(mt->stack[i]);
-+ }
-+ free(mt->stack);
-+ mt->stack = NULL;
-+ mt->stack_len = 0;
++ struct merged_iter *mi)
++{
++ it->iter_arg = mi;
++ it->ops = &merged_iter_vtable;
++}
++
++int new_merged_table(struct merged_table **dest, struct reader **stack, int n)
++{
++ uint64_t last_max = 0;
++ uint64_t first_min = 0;
++ int i = 0;
++ for (i = 0; i < n; i++) {
++ struct reader *r = stack[i];
++ if (i > 0 && last_max >= reader_min_update_index(r)) {
++ return FORMAT_ERROR;
++ }
++ if (i == 0) {
++ first_min = reader_min_update_index(r);
++ }
++
++ last_max = reader_max_update_index(r);
++ }
++
++ {
++ struct merged_table m = {
++ .stack = stack,
++ .stack_len = n,
++ .min = first_min,
++ .max = last_max,
++ .hash_size = SHA1_SIZE,
++ };
++
++ *dest = calloc(sizeof(struct merged_table), 1);
++ **dest = m;
++ }
++ return 0;
++}
++
++void merged_table_close(struct merged_table *mt)
++{
++ int i = 0;
++ for (i = 0; i < mt->stack_len; i++) {
++ reader_free(mt->stack[i]);
++ }
++ free(mt->stack);
++ mt->stack = NULL;
++ mt->stack_len = 0;
+}
+
+/* clears the list of subtable, without affecting the readers themselves. */
-+void merged_table_clear(struct merged_table *mt) {
-+ free(mt->stack);
-+ mt->stack = NULL;
-+ mt->stack_len = 0;
++void merged_table_clear(struct merged_table *mt)
++{
++ free(mt->stack);
++ mt->stack = NULL;
++ mt->stack_len = 0;
+}
+
-+void merged_table_free(struct merged_table *mt) {
-+ if (mt == NULL) {
-+ return;
-+ }
-+ merged_table_clear(mt);
-+ free(mt);
++void merged_table_free(struct merged_table *mt)
++{
++ if (mt == NULL) {
++ return;
++ }
++ merged_table_clear(mt);
++ free(mt);
+}
+
-+uint64_t merged_max_update_index(struct merged_table *mt) { return mt->max; }
++uint64_t merged_max_update_index(struct merged_table *mt)
++{
++ return mt->max;
++}
+
-+uint64_t merged_min_update_index(struct merged_table *mt) { return mt->min; }
++uint64_t merged_min_update_index(struct merged_table *mt)
++{
++ return mt->min;
++}
+
+static int merged_table_seek_record(struct merged_table *mt,
-+ struct iterator *it, struct record rec) {
-+ struct iterator *iters = calloc(sizeof(struct iterator), mt->stack_len);
-+ struct merged_iter merged = {
-+ .stack = iters,
-+ .typ = record_type(rec),
-+ .hash_size = mt->hash_size,
-+ };
-+ int n = 0;
-+ int err = 0;
-+ for (int i = 0; i < mt->stack_len && err == 0; i++) {
-+ int e = reader_seek(mt->stack[i], &iters[n], rec);
-+ if (e < 0) {
-+ err = e;
-+ }
-+ if (e == 0) {
-+ n++;
-+ }
-+ }
-+ if (err < 0) {
-+ for (int i = 0; i < n; i++) {
-+ iterator_destroy(&iters[i]);
-+ }
-+ free(iters);
-+ return err;
-+ }
-+
-+ merged.stack_len = n, err = merged_iter_init(&merged);
-+ if (err < 0) {
-+ merged_iter_close(&merged);
-+ return err;
-+ }
-+
-+ {
-+ struct merged_iter *p = malloc(sizeof(struct merged_iter));
-+ *p = merged;
-+ iterator_from_merged_iter(it, p);
-+ }
-+ return 0;
++ struct iterator *it, struct record rec)
++{
++ struct iterator *iters = calloc(sizeof(struct iterator), mt->stack_len);
++ struct merged_iter merged = {
++ .stack = iters,
++ .typ = record_type(rec),
++ .hash_size = mt->hash_size,
++ };
++ int n = 0;
++ int err = 0;
++ int i = 0;
++ for (i = 0; i < mt->stack_len && err == 0; i++) {
++ int e = reader_seek(mt->stack[i], &iters[n], rec);
++ if (e < 0) {
++ err = e;
++ }
++ if (e == 0) {
++ n++;
++ }
++ }
++ if (err < 0) {
++ int i = 0;
++ for (i = 0; i < n; i++) {
++ iterator_destroy(&iters[i]);
++ }
++ free(iters);
++ return err;
++ }
++
++ merged.stack_len = n, err = merged_iter_init(&merged);
++ if (err < 0) {
++ merged_iter_close(&merged);
++ return err;
++ }
++
++ {
++ struct merged_iter *p = malloc(sizeof(struct merged_iter));
++ *p = merged;
++ iterator_from_merged_iter(it, p);
++ }
++ return 0;
+}
+
+int merged_table_seek_ref(struct merged_table *mt, struct iterator *it,
-+ const char *name) {
-+ struct ref_record ref = {
-+ .ref_name = (char *)name,
-+ };
-+ struct record rec = {};
-+ record_from_ref(&rec, &ref);
-+ return merged_table_seek_record(mt, it, rec);
++ const char *name)
++{
++ struct ref_record ref = {
++ .ref_name = (char *)name,
++ };
++ struct record rec = {};
++ record_from_ref(&rec, &ref);
++ return merged_table_seek_record(mt, it, rec);
+}
+
+int merged_table_seek_log_at(struct merged_table *mt, struct iterator *it,
-+ const char *name, uint64_t update_index) {
-+ struct log_record log = {
-+ .ref_name = (char *)name,
-+ .update_index = update_index,
-+ };
-+ struct record rec = {};
-+ record_from_log(&rec, &log);
-+ return merged_table_seek_record(mt, it, rec);
++ const char *name, uint64_t update_index)
++{
++ struct log_record log = {
++ .ref_name = (char *)name,
++ .update_index = update_index,
++ };
++ struct record rec = {};
++ record_from_log(&rec, &log);
++ return merged_table_seek_record(mt, it, rec);
+}
+
+int merged_table_seek_log(struct merged_table *mt, struct iterator *it,
-+ const char *name) {
-+ uint64_t max = ~((uint64_t)0);
-+ return merged_table_seek_log_at(mt, it, name, max);
++ const char *name)
++{
++ uint64_t max = ~((uint64_t)0);
++ return merged_table_seek_log_at(mt, it, name, max);
+}
diff --git a/reftable/merged.h b/reftable/merged.h
@@ -2009,20 +1877,20 @@
+#include "reftable.h"
+
+struct merged_table {
-+ struct reader **stack;
-+ int stack_len;
-+ int hash_size;
++ struct reader **stack;
++ int stack_len;
++ int hash_size;
+
-+ uint64_t min;
-+ uint64_t max;
++ uint64_t min;
++ uint64_t max;
+};
+
+struct merged_iter {
-+ struct iterator *stack;
-+ int hash_size;
-+ int stack_len;
-+ byte typ;
-+ struct merged_iter_pqueue pq;
++ struct iterator *stack;
++ int hash_size;
++ int stack_len;
++ byte typ;
++ struct merged_iter_pqueue pq;
+} merged_iter;
+
+void merged_table_clear(struct merged_table *mt);
@@ -2044,7 +1912,7 @@
+
+#include "merged.h"
+
-+#include <string.h>
++#include "system.h"
+
+#include "basics.h"
+#include "block.h"
@@ -2055,231 +1923,242 @@
+#include "reftable.h"
+#include "test_framework.h"
+
-+void test_pq(void) {
-+ char *names[54] = {};
-+ int N = ARRAYSIZE(names) - 1;
-+
-+ for (int i = 0; i < N; i++) {
-+ char name[100];
-+ sprintf(name, "%02d", i);
-+ names[i] = strdup(name);
-+ }
-+
-+ struct merged_iter_pqueue pq = {};
-+
-+ int i = 1;
-+ do {
-+ struct record rec = new_record(BLOCK_TYPE_REF);
-+ record_as_ref(rec)->ref_name = names[i];
-+
-+ struct pq_entry e = {
-+ .rec = rec,
-+ };
-+ merged_iter_pqueue_add(&pq, e);
-+ merged_iter_pqueue_check(pq);
-+ i = (i * 7) % N;
-+ } while (i != 1);
-+
-+ const char *last = NULL;
-+ while (!merged_iter_pqueue_is_empty(pq)) {
-+ struct pq_entry e = merged_iter_pqueue_remove(&pq);
-+ merged_iter_pqueue_check(pq);
-+ struct ref_record *ref = record_as_ref(e.rec);
-+
-+ if (last != NULL) {
-+ assert(strcmp(last, ref->ref_name) < 0);
-+ }
-+ last = ref->ref_name;
-+ ref->ref_name = NULL;
-+ free(ref);
-+ }
-+
-+ for (int i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+
-+ merged_iter_pqueue_clear(&pq);
-+}
-+
-+void write_test_table(struct slice *buf, struct ref_record refs[], int n) {
-+ int min = 0xffffffff;
-+ int max = 0;
-+ for (int i = 0; i < n; i++) {
-+ uint64_t ui = refs[i].update_index;
-+ if (ui > max) {
-+ max = ui;
-+ }
-+ if (ui < min) {
-+ min = ui;
-+ }
-+ }
-+
-+ struct write_options opts = {
-+ .block_size = 256,
-+ };
-+
-+ struct writer *w = new_writer(&slice_write_void, buf, &opts);
-+ writer_set_limits(w, min, max);
-+
-+ for (int i = 0; i < n; i++) {
-+ uint64_t before = refs[i].update_index;
-+ int n = writer_add_ref(w, &refs[i]);
-+ assert(n == 0);
-+ assert(before == refs[i].update_index);
-+ }
-+
-+ int err = writer_close(w);
-+ assert(err == 0);
-+
-+ writer_free(w);
-+ w = NULL;
++void test_pq(void)
++{
++ char *names[54] = {};
++ int N = ARRAYSIZE(names) - 1;
++
++ int i = 0;
++ for (i = 0; i < N; i++) {
++ char name[100];
++ snprintf(name, sizeof(name), "%02d", i);
++ names[i] = strdup(name);
++ }
++
++ struct merged_iter_pqueue pq = {};
++
++ i = 1;
++ do {
++ struct record rec = new_record(BLOCK_TYPE_REF);
++ record_as_ref(rec)->ref_name = names[i];
++
++ struct pq_entry e = {
++ .rec = rec,
++ };
++ merged_iter_pqueue_add(&pq, e);
++ merged_iter_pqueue_check(pq);
++ i = (i * 7) % N;
++ } while (i != 1);
++
++ const char *last = NULL;
++ while (!merged_iter_pqueue_is_empty(pq)) {
++ struct pq_entry e = merged_iter_pqueue_remove(&pq);
++ merged_iter_pqueue_check(pq);
++ struct ref_record *ref = record_as_ref(e.rec);
++
++ if (last != NULL) {
++ assert(strcmp(last, ref->ref_name) < 0);
++ }
++ last = ref->ref_name;
++ ref->ref_name = NULL;
++ free(ref);
++ }
++
++ for (i = 0; i < N; i++) {
++ free(names[i]);
++ }
++
++ merged_iter_pqueue_clear(&pq);
++}
++
++void write_test_table(struct slice *buf, struct ref_record refs[], int n)
++{
++ int min = 0xffffffff;
++ int max = 0;
++ int i = 0;
++ for (i = 0; i < n; i++) {
++ uint64_t ui = refs[i].update_index;
++ if (ui > max) {
++ max = ui;
++ }
++ if (ui < min) {
++ min = ui;
++ }
++ }
++
++ struct write_options opts = {
++ .block_size = 256,
++ };
++
++ struct writer *w = new_writer(&slice_write_void, buf, &opts);
++ writer_set_limits(w, min, max);
++
++ for (i = 0; i < n; i++) {
++ uint64_t before = refs[i].update_index;
++ int n = writer_add_ref(w, &refs[i]);
++ assert(n == 0);
++ assert(before == refs[i].update_index);
++ }
++
++ int err = writer_close(w);
++ assert(err == 0);
++
++ writer_free(w);
++ w = NULL;
+}
+
+static struct merged_table *merged_table_from_records(struct ref_record **refs,
-+ int *sizes,
-+ struct slice *buf,
-+ int n) {
-+ struct block_source *source = calloc(n, sizeof(*source));
-+ struct reader **rd = calloc(n, sizeof(*rd));
-+ for (int i = 0; i < n; i++) {
-+ write_test_table(&buf[i], refs[i], sizes[i]);
-+ block_source_from_slice(&source[i], &buf[i]);
-+
-+ int err = new_reader(&rd[i], source[i], "name");
-+ assert(err == 0);
-+ }
-+
-+ struct merged_table *mt = NULL;
-+ int err = new_merged_table(&mt, rd, n);
-+ assert(err == 0);
-+ return mt;
-+}
-+
-+void test_merged_between(void) {
-+ byte hash1[SHA1_SIZE];
-+ byte hash2[SHA1_SIZE];
-+
-+ set_test_hash(hash1, 1);
-+ set_test_hash(hash2, 2);
-+ struct ref_record r1[] = {{
-+ .ref_name = "b",
-+ .update_index = 1,
-+ .value = hash1,
-+ }};
-+ struct ref_record r2[] = {{
-+ .ref_name = "a",
-+ .update_index = 2,
-+ }};
-+
-+ struct ref_record *refs[] = {r1, r2};
-+ int sizes[] = {1, 1};
-+ struct slice bufs[2] = {};
-+ struct merged_table *mt = merged_table_from_records(refs, sizes, bufs, 2);
-+
-+ struct iterator it = {};
-+ int err = merged_table_seek_ref(mt, &it, "a");
-+ assert(err == 0);
-+
-+ struct ref_record ref = {};
-+ err = iterator_next_ref(it, &ref);
-+ assert_err(err);
-+ assert(ref.update_index == 2);
-+}
-+
-+void test_merged(void) {
-+ byte hash1[SHA1_SIZE];
-+ byte hash2[SHA1_SIZE];
-+
-+ set_test_hash(hash1, 1);
-+ set_test_hash(hash2, 2);
-+ struct ref_record r1[] = {{
-+ .ref_name = "a",
-+ .update_index = 1,
-+ .value = hash1,
-+ },
-+ {
-+ .ref_name = "b",
-+ .update_index = 1,
-+ .value = hash1,
-+ },
-+ {
-+ .ref_name = "c",
-+ .update_index = 1,
-+ .value = hash1,
-+ }};
-+ struct ref_record r2[] = {{
-+ .ref_name = "a",
-+ .update_index = 2,
-+ }};
-+ struct ref_record r3[] = {
-+ {
-+ .ref_name = "c",
-+ .update_index = 3,
-+ .value = hash2,
-+ },
-+ {
-+ .ref_name = "d",
-+ .update_index = 3,
-+ .value = hash1,
-+ },
-+ };
-+
-+ struct ref_record *refs[] = {r1, r2, r3};
-+ int sizes[3] = {3, 1, 2};
-+ struct slice bufs[3] = {};
-+
-+ struct merged_table *mt = merged_table_from_records(refs, sizes, bufs, 3);
-+
-+ struct iterator it = {};
-+ int err = merged_table_seek_ref(mt, &it, "a");
-+ assert(err == 0);
-+
-+ struct ref_record *out = NULL;
-+ int len = 0;
-+ int cap = 0;
-+ while (len < 100) { // cap loops/recursion.
-+ struct ref_record ref = {};
-+ int err = iterator_next_ref(it, &ref);
-+ if (err > 0) {
-+ break;
-+ }
-+ if (len == cap) {
-+ cap = 2 * cap + 1;
-+ out = realloc(out, sizeof(struct ref_record) * cap);
-+ }
-+ out[len++] = ref;
-+ }
-+ iterator_destroy(&it);
-+
-+ struct ref_record want[] = {
-+ r2[0],
-+ r1[1],
-+ r3[0],
-+ r3[1],
-+ };
-+ assert(ARRAYSIZE(want) == len);
-+ for (int i = 0; i < len; i++) {
-+ assert(ref_record_equal(&want[i], &out[i], SHA1_SIZE));
-+ }
-+ for (int i = 0; i < len; i++) {
-+ ref_record_clear(&out[i]);
-+ }
-+ free(out);
-+
-+ for (int i = 0; i < 3; i++) {
-+ free(slice_yield(&bufs[i]));
-+ }
-+ merged_table_close(mt);
-+ merged_table_free(mt);
-+}
-+
-+// XXX test refs_for(oid)
-+
-+int main() {
-+ add_test_case("test_merged_between", &test_merged_between);
-+ add_test_case("test_pq", &test_pq);
-+ add_test_case("test_merged", &test_merged);
-+ test_main();
++ int *sizes,
++ struct slice *buf, int n)
++{
++ struct block_source *source = calloc(n, sizeof(*source));
++ struct reader **rd = calloc(n, sizeof(*rd));
++ int i = 0;
++ for (i = 0; i < n; i++) {
++ write_test_table(&buf[i], refs[i], sizes[i]);
++ block_source_from_slice(&source[i], &buf[i]);
++
++ int err = new_reader(&rd[i], source[i], "name");
++ assert(err == 0);
++ }
++
++ struct merged_table *mt = NULL;
++ int err = new_merged_table(&mt, rd, n);
++ assert(err == 0);
++ return mt;
++}
++
++void test_merged_between(void)
++{
++ byte hash1[SHA1_SIZE];
++ byte hash2[SHA1_SIZE];
++
++ set_test_hash(hash1, 1);
++ set_test_hash(hash2, 2);
++ struct ref_record r1[] = { {
++ .ref_name = "b",
++ .update_index = 1,
++ .value = hash1,
++ } };
++ struct ref_record r2[] = { {
++ .ref_name = "a",
++ .update_index = 2,
++ } };
++
++ struct ref_record *refs[] = { r1, r2 };
++ int sizes[] = { 1, 1 };
++ struct slice bufs[2] = {};
++ struct merged_table *mt =
++ merged_table_from_records(refs, sizes, bufs, 2);
++
++ struct iterator it = {};
++ int err = merged_table_seek_ref(mt, &it, "a");
++ assert(err == 0);
++
++ struct ref_record ref = {};
++ err = iterator_next_ref(it, &ref);
++ assert_err(err);
++ assert(ref.update_index == 2);
++}
++
++void test_merged(void)
++{
++ byte hash1[SHA1_SIZE];
++ byte hash2[SHA1_SIZE];
++
++ set_test_hash(hash1, 1);
++ set_test_hash(hash2, 2);
++ struct ref_record r1[] = { {
++ .ref_name = "a",
++ .update_index = 1,
++ .value = hash1,
++ },
++ {
++ .ref_name = "b",
++ .update_index = 1,
++ .value = hash1,
++ },
++ {
++ .ref_name = "c",
++ .update_index = 1,
++ .value = hash1,
++ } };
++ struct ref_record r2[] = { {
++ .ref_name = "a",
++ .update_index = 2,
++ } };
++ struct ref_record r3[] = {
++ {
++ .ref_name = "c",
++ .update_index = 3,
++ .value = hash2,
++ },
++ {
++ .ref_name = "d",
++ .update_index = 3,
++ .value = hash1,
++ },
++ };
++
++ struct ref_record *refs[] = { r1, r2, r3 };
++ int sizes[3] = { 3, 1, 2 };
++ struct slice bufs[3] = {};
++
++ struct merged_table *mt =
++ merged_table_from_records(refs, sizes, bufs, 3);
++
++ struct iterator it = {};
++ int err = merged_table_seek_ref(mt, &it, "a");
++ assert(err == 0);
++
++ struct ref_record *out = NULL;
++ int len = 0;
++ int cap = 0;
++ while (len < 100) { /* cap loops/recursion. */
++ struct ref_record ref = {};
++ int err = iterator_next_ref(it, &ref);
++ if (err > 0) {
++ break;
++ }
++ if (len == cap) {
++ cap = 2 * cap + 1;
++ out = realloc(out, sizeof(struct ref_record) * cap);
++ }
++ out[len++] = ref;
++ }
++ iterator_destroy(&it);
++
++ struct ref_record want[] = {
++ r2[0],
++ r1[1],
++ r3[0],
++ r3[1],
++ };
++ assert(ARRAYSIZE(want) == len);
++ int i = 0;
++ for (i = 0; i < len; i++) {
++ assert(ref_record_equal(&want[i], &out[i], SHA1_SIZE));
++ }
++ for (i = 0; i < len; i++) {
++ ref_record_clear(&out[i]);
++ }
++ free(out);
++
++ for (i = 0; i < 3; i++) {
++ free(slice_yield(&bufs[i]));
++ }
++ merged_table_close(mt);
++ merged_table_free(mt);
++}
++
++/* XXX test refs_for(oid) */
++
++int main()
++{
++ add_test_case("test_merged_between", &test_merged_between);
++ add_test_case("test_pq", &test_pq);
++ add_test_case("test_merged", &test_merged);
++ test_main();
+}
diff --git a/reftable/pq.c b/reftable/pq.c
@@ -2297,111 +2176,119 @@
+
+#include "pq.h"
+
-+#include <assert.h>
-+#include <stdlib.h>
++#include "system.h"
+
-+int pq_less(struct pq_entry a, struct pq_entry b) {
-+ struct slice ak = {};
-+ struct slice bk = {};
-+ int cmp = 0;
-+ record_key(a.rec, &ak);
-+ record_key(b.rec, &bk);
++int pq_less(struct pq_entry a, struct pq_entry b)
++{
++ struct slice ak = {};
++ struct slice bk = {};
++ int cmp = 0;
++ record_key(a.rec, &ak);
++ record_key(b.rec, &bk);
+
-+ cmp = slice_compare(ak, bk);
++ cmp = slice_compare(ak, bk);
++
++ free(slice_yield(&ak));
++ free(slice_yield(&bk));
+
-+ free(slice_yield(&ak));
-+ free(slice_yield(&bk));
++ if (cmp == 0) {
++ return a.index > b.index;
++ }
+
-+ if (cmp == 0) {
-+ return a.index > b.index;
-+ }
-+
-+ return cmp < 0;
++ return cmp < 0;
+}
+
-+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq) {
-+ return pq.heap[0];
++struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)
++{
++ return pq.heap[0];
+}
+
-+bool merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq) {
-+ return pq.len == 0;
++bool merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)
++{
++ return pq.len == 0;
+}
+
-+void merged_iter_pqueue_check(struct merged_iter_pqueue pq) {
-+ for (int i = 1; i < pq.len; i++) {
-+ int parent = (i - 1) / 2;
++void merged_iter_pqueue_check(struct merged_iter_pqueue pq)
++{
++ int i = 0;
++ for (i = 1; i < pq.len; i++) {
++ int parent = (i - 1) / 2;
+
-+ assert(pq_less(pq.heap[parent], pq.heap[i]));
-+ }
++ assert(pq_less(pq.heap[parent], pq.heap[i]));
++ }
+}
+
-+struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq) {
-+ int i = 0;
-+ struct pq_entry e = pq->heap[0];
-+ pq->heap[0] = pq->heap[pq->len - 1];
-+ pq->len--;
++struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
++{
++ int i = 0;
++ struct pq_entry e = pq->heap[0];
++ pq->heap[0] = pq->heap[pq->len - 1];
++ pq->len--;
+
-+ i = 0;
-+ while (i < pq->len) {
-+ int min = i;
-+ int j = 2 * i + 1;
-+ int k = 2 * i + 2;
-+ if (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {
-+ min = j;
-+ }
-+ if (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {
-+ min = k;
-+ }
++ i = 0;
++ while (i < pq->len) {
++ int min = i;
++ int j = 2 * i + 1;
++ int k = 2 * i + 2;
++ if (j < pq->len && pq_less(pq->heap[j], pq->heap[i])) {
++ min = j;
++ }
++ if (k < pq->len && pq_less(pq->heap[k], pq->heap[min])) {
++ min = k;
++ }
+
-+ if (min == i) {
-+ break;
-+ }
++ if (min == i) {
++ break;
++ }
+
-+ {
-+ struct pq_entry tmp = pq->heap[min];
-+ pq->heap[min] = pq->heap[i];
-+ pq->heap[i] = tmp;
-+ }
++ {
++ struct pq_entry tmp = pq->heap[min];
++ pq->heap[min] = pq->heap[i];
++ pq->heap[i] = tmp;
++ }
+
-+ i = min;
-+ }
++ i = min;
++ }
+
-+ return e;
++ return e;
+}
+
-+void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e) {
-+ int i = 0;
-+ if (pq->len == pq->cap) {
-+ pq->cap = 2 * pq->cap + 1;
-+ pq->heap = realloc(pq->heap, pq->cap * sizeof(struct pq_entry));
-+ }
-+
-+ pq->heap[pq->len++] = e;
-+ i = pq->len - 1;
-+ while (i > 0) {
-+ int j = (i - 1) / 2;
-+ if (pq_less(pq->heap[j], pq->heap[i])) {
-+ break;
-+ }
++void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
++{
++ int i = 0;
++ if (pq->len == pq->cap) {
++ pq->cap = 2 * pq->cap + 1;
++ pq->heap = realloc(pq->heap, pq->cap * sizeof(struct pq_entry));
++ }
+
-+ {
-+ struct pq_entry tmp = pq->heap[j];
-+ pq->heap[j] = pq->heap[i];
-+ pq->heap[i] = tmp;
-+ }
++ pq->heap[pq->len++] = e;
++ i = pq->len - 1;
++ while (i > 0) {
++ int j = (i - 1) / 2;
++ if (pq_less(pq->heap[j], pq->heap[i])) {
++ break;
++ }
+
-+ i = j;
-+ }
-+}
++ {
++ struct pq_entry tmp = pq->heap[j];
++ pq->heap[j] = pq->heap[i];
++ pq->heap[i] = tmp;
++ }
+
-+void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq) {
-+ for (int i = 0; i < pq->len; i++) {
-+ record_clear(pq->heap[i].rec);
-+ free(record_yield(&pq->heap[i].rec));
-+ }
-+ free(pq->heap);
-+ pq->heap = NULL;
-+ pq->len = pq->cap = 0;
++ i = j;
++ }
++}
++
++void merged_iter_pqueue_clear(struct merged_iter_pqueue *pq)
++{
++ int i = 0;
++ for (i = 0; i < pq->len; i++) {
++ record_clear(pq->heap[i].rec);
++ free(record_yield(&pq->heap[i].rec));
++ }
++ free(pq->heap);
++ pq->heap = NULL;
++ pq->len = pq->cap = 0;
+}
diff --git a/reftable/pq.h b/reftable/pq.h
@@ -2423,16 +2310,16 @@
+#include "record.h"
+
+struct pq_entry {
-+ struct record rec;
-+ int index;
++ struct record rec;
++ int index;
+};
+
+int pq_less(struct pq_entry a, struct pq_entry b);
+
+struct merged_iter_pqueue {
-+ struct pq_entry *heap;
-+ int len;
-+ int cap;
++ struct pq_entry *heap;
++ int len;
++ int cap;
+};
+
+struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);
@@ -2459,10 +2346,7 @@
+
+#include "reader.h"
+
-+#include <stdio.h>
-+#include <stdlib.h>
-+#include <string.h>
-+#include <zlib.h>
++#include "system.h"
+
+#include "block.h"
+#include "constants.h"
@@ -2471,655 +2355,696 @@
+#include "reftable.h"
+#include "tree.h"
+
-+uint64_t block_source_size(struct block_source source) {
-+ return source.ops->size(source.arg);
++uint64_t block_source_size(struct block_source source)
++{
++ return source.ops->size(source.arg);
+}
+
+int block_source_read_block(struct block_source source, struct block *dest,
-+ uint64_t off, uint32_t size) {
-+ int result = source.ops->read_block(source.arg, dest, off, size);
-+ dest->source = source;
-+ return result;
++ uint64_t off, uint32_t size)
++{
++ int result = source.ops->read_block(source.arg, dest, off, size);
++ dest->source = source;
++ return result;
+}
+
-+void block_source_return_block(struct block_source source,
-+ struct block *blockp) {
-+ source.ops->return_block(source.arg, blockp);
-+ blockp->data = NULL;
-+ blockp->len = 0;
-+ blockp->source.ops = NULL;
-+ blockp->source.arg = NULL;
++void block_source_return_block(struct block_source source, struct block *blockp)
++{
++ source.ops->return_block(source.arg, blockp);
++ blockp->data = NULL;
++ blockp->len = 0;
++ blockp->source.ops = NULL;
++ blockp->source.arg = NULL;
+}
+
-+void block_source_close(struct block_source *source) {
-+ if (source->ops == NULL) {
-+ return;
-+ }
++void block_source_close(struct block_source *source)
++{
++ if (source->ops == NULL) {
++ return;
++ }
+
-+ source->ops->close(source->arg);
-+ source->ops = NULL;
++ source->ops->close(source->arg);
++ source->ops = NULL;
+}
+
-+static struct reader_offsets *reader_offsets_for(struct reader *r, byte typ) {
-+ switch (typ) {
-+ case BLOCK_TYPE_REF:
-+ return &r->ref_offsets;
-+ case BLOCK_TYPE_LOG:
-+ return &r->log_offsets;
-+ case BLOCK_TYPE_OBJ:
-+ return &r->obj_offsets;
-+ }
-+ abort();
++static struct reader_offsets *reader_offsets_for(struct reader *r, byte typ)
++{
++ switch (typ) {
++ case BLOCK_TYPE_REF:
++ return &r->ref_offsets;
++ case BLOCK_TYPE_LOG:
++ return &r->log_offsets;
++ case BLOCK_TYPE_OBJ:
++ return &r->obj_offsets;
++ }
++ abort();
+}
+
+static int reader_get_block(struct reader *r, struct block *dest, uint64_t off,
-+ uint32_t sz) {
-+ if (off >= r->size) {
-+ return 0;
-+ }
-+
-+ if (off + sz > r->size) {
-+ sz = r->size - off;
-+ }
-+
-+ return block_source_read_block(r->source, dest, off, sz);
-+}
-+
-+void reader_return_block(struct reader *r, struct block *p) {
-+ block_source_return_block(r->source, p);
-+}
-+
-+const char *reader_name(struct reader *r) { return r->name; }
-+
-+static int parse_footer(struct reader *r, byte *footer, byte *header) {
-+ byte *f = footer;
-+ int err = 0;
-+ if (memcmp(f, "REFT", 4)) {
-+ err = FORMAT_ERROR;
-+ goto exit;
-+ }
-+ f += 4;
-+
-+ if (0 != memcmp(footer, header, HEADER_SIZE)) {
-+ err = FORMAT_ERROR;
-+ goto exit;
-+ }
-+
-+ {
-+ byte version = *f++;
-+ if (version != 1) {
-+ err = FORMAT_ERROR;
-+ goto exit;
-+ }
-+ }
-+
-+ r->block_size = get_u24(f);
-+
-+ f += 3;
-+ r->min_update_index = get_u64(f);
-+ f += 8;
-+ r->max_update_index = get_u64(f);
-+ f += 8;
-+
-+ r->ref_offsets.index_offset = get_u64(f);
-+ f += 8;
-+
-+ r->obj_offsets.offset = get_u64(f);
-+ f += 8;
-+
-+ r->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);
-+ r->obj_offsets.offset >>= 5;
-+
-+ r->obj_offsets.index_offset = get_u64(f);
-+ f += 8;
-+ r->log_offsets.offset = get_u64(f);
-+ f += 8;
-+ r->log_offsets.index_offset = get_u64(f);
-+ f += 8;
-+
-+ {
-+ uint32_t computed_crc = crc32(0, footer, f - footer);
-+ uint32_t file_crc = get_u32(f);
-+ f += 4;
-+ if (computed_crc != file_crc) {
-+ err = FORMAT_ERROR;
-+ goto exit;
-+ }
-+ }
-+
-+ {
-+ byte first_block_typ = header[HEADER_SIZE];
-+ r->ref_offsets.present = (first_block_typ == BLOCK_TYPE_REF);
-+ r->ref_offsets.offset = 0;
-+ r->log_offsets.present =
-+ (first_block_typ == BLOCK_TYPE_LOG || r->log_offsets.offset > 0);
-+ r->obj_offsets.present = r->obj_offsets.offset > 0;
-+ }
-+ err = 0;
++ uint32_t sz)
++{
++ if (off >= r->size) {
++ return 0;
++ }
++
++ if (off + sz > r->size) {
++ sz = r->size - off;
++ }
++
++ return block_source_read_block(r->source, dest, off, sz);
++}
++
++void reader_return_block(struct reader *r, struct block *p)
++{
++ block_source_return_block(r->source, p);
++}
++
++const char *reader_name(struct reader *r)
++{
++ return r->name;
++}
++
++static int parse_footer(struct reader *r, byte *footer, byte *header)
++{
++ byte *f = footer;
++ int err = 0;
++ if (memcmp(f, "REFT", 4)) {
++ err = FORMAT_ERROR;
++ goto exit;
++ }
++ f += 4;
++
++ if (memcmp(footer, header, HEADER_SIZE)) {
++ err = FORMAT_ERROR;
++ goto exit;
++ }
++
++ {
++ byte version = *f++;
++ if (version != 1) {
++ err = FORMAT_ERROR;
++ goto exit;
++ }
++ }
++
++ r->block_size = get_u24(f);
++
++ f += 3;
++ r->min_update_index = get_u64(f);
++ f += 8;
++ r->max_update_index = get_u64(f);
++ f += 8;
++
++ r->ref_offsets.index_offset = get_u64(f);
++ f += 8;
++
++ r->obj_offsets.offset = get_u64(f);
++ f += 8;
++
++ r->object_id_len = r->obj_offsets.offset & ((1 << 5) - 1);
++ r->obj_offsets.offset >>= 5;
++
++ r->obj_offsets.index_offset = get_u64(f);
++ f += 8;
++ r->log_offsets.offset = get_u64(f);
++ f += 8;
++ r->log_offsets.index_offset = get_u64(f);
++ f += 8;
++
++ {
++ uint32_t computed_crc = crc32(0, footer, f - footer);
++ uint32_t file_crc = get_u32(f);
++ f += 4;
++ if (computed_crc != file_crc) {
++ err = FORMAT_ERROR;
++ goto exit;
++ }
++ }
++
++ {
++ byte first_block_typ = header[HEADER_SIZE];
++ r->ref_offsets.present = (first_block_typ == BLOCK_TYPE_REF);
++ r->ref_offsets.offset = 0;
++ r->log_offsets.present = (first_block_typ == BLOCK_TYPE_LOG ||
++ r->log_offsets.offset > 0);
++ r->obj_offsets.present = r->obj_offsets.offset > 0;
++ }
++ err = 0;
+exit:
-+ return err;
-+}
-+
-+int init_reader(struct reader *r, struct block_source source,
-+ const char *name) {
-+ struct block footer = {};
-+ struct block header = {};
-+ int err = 0;
-+
-+ memset(r, 0, sizeof(struct reader));
-+ r->size = block_source_size(source) - FOOTER_SIZE;
-+ r->source = source;
-+ r->name = strdup(name);
-+ r->hash_size = SHA1_SIZE;
-+
-+ err = block_source_read_block(source, &footer, r->size, FOOTER_SIZE);
-+ if (err != FOOTER_SIZE) {
-+ err = IO_ERROR;
-+ goto exit;
-+ }
-+
-+ // Need +1 to read type of first block.
-+ err = reader_get_block(r, &header, 0, HEADER_SIZE + 1);
-+ if (err != HEADER_SIZE + 1) {
-+ err = IO_ERROR;
-+ goto exit;
-+ }
-+
-+ err = parse_footer(r, footer.data, header.data);
++ return err;
++}
++
++int init_reader(struct reader *r, struct block_source source, const char *name)
++{
++ struct block footer = {};
++ struct block header = {};
++ int err = 0;
++
++ memset(r, 0, sizeof(struct reader));
++ r->size = block_source_size(source) - FOOTER_SIZE;
++ r->source = source;
++ r->name = strdup(name);
++ r->hash_size = SHA1_SIZE;
++
++ err = block_source_read_block(source, &footer, r->size, FOOTER_SIZE);
++ if (err != FOOTER_SIZE) {
++ err = IO_ERROR;
++ goto exit;
++ }
++
++ /* Need +1 to read type of first block. */
++ err = reader_get_block(r, &header, 0, HEADER_SIZE + 1);
++ if (err != HEADER_SIZE + 1) {
++ err = IO_ERROR;
++ goto exit;
++ }
++
++ err = parse_footer(r, footer.data, header.data);
+exit:
-+ block_source_return_block(r->source, &footer);
-+ block_source_return_block(r->source, &header);
-+ return err;
++ block_source_return_block(r->source, &footer);
++ block_source_return_block(r->source, &header);
++ return err;
+}
+
+struct table_iter {
-+ struct reader *r;
-+ byte typ;
-+ uint64_t block_off;
-+ struct block_iter bi;
-+ bool finished;
++ struct reader *r;
++ byte typ;
++ uint64_t block_off;
++ struct block_iter bi;
++ bool finished;
+};
+
+static void table_iter_copy_from(struct table_iter *dest,
-+ struct table_iter *src) {
-+ dest->r = src->r;
-+ dest->typ = src->typ;
-+ dest->block_off = src->block_off;
-+ dest->finished = src->finished;
-+ block_iter_copy_from(&dest->bi, &src->bi);
++ struct table_iter *src)
++{
++ dest->r = src->r;
++ dest->typ = src->typ;
++ dest->block_off = src->block_off;
++ dest->finished = src->finished;
++ block_iter_copy_from(&dest->bi, &src->bi);
+}
+
-+static int table_iter_next_in_block(struct table_iter *ti, struct record rec) {
-+ int res = block_iter_next(&ti->bi, rec);
-+ if (res == 0 && record_type(rec) == BLOCK_TYPE_REF) {
-+ ((struct ref_record *)rec.data)->update_index += ti->r->min_update_index;
-+ }
++static int table_iter_next_in_block(struct table_iter *ti, struct record rec)
++{
++ int res = block_iter_next(&ti->bi, rec);
++ if (res == 0 && record_type(rec) == BLOCK_TYPE_REF) {
++ ((struct ref_record *)rec.data)->update_index +=
++ ti->r->min_update_index;
++ }
+
-+ return res;
++ return res;
+}
+
-+static void table_iter_block_done(struct table_iter *ti) {
-+ if (ti->bi.br == NULL) {
-+ return;
-+ }
-+ reader_return_block(ti->r, &ti->bi.br->block);
-+ free(ti->bi.br);
-+ ti->bi.br = NULL;
++static void table_iter_block_done(struct table_iter *ti)
++{
++ if (ti->bi.br == NULL) {
++ return;
++ }
++ reader_return_block(ti->r, &ti->bi.br->block);
++ free(ti->bi.br);
++ ti->bi.br = NULL;
+
-+ ti->bi.last_key.len = 0;
-+ ti->bi.next_off = 0;
++ ti->bi.last_key.len = 0;
++ ti->bi.next_off = 0;
+}
+
-+static int32_t extract_block_size(byte *data, byte *typ, uint64_t off) {
-+ int32_t result = 0;
++static int32_t extract_block_size(byte *data, byte *typ, uint64_t off)
++{
++ int32_t result = 0;
+
-+ if (off == 0) {
-+ data += 24;
-+ }
++ if (off == 0) {
++ data += 24;
++ }
+
-+ *typ = data[0];
-+ if (is_block_type(*typ)) {
-+ result = get_u24(data + 1);
-+ }
-+ return result;
++ *typ = data[0];
++ if (is_block_type(*typ)) {
++ result = get_u24(data + 1);
++ }
++ return result;
+}
+
+int reader_init_block_reader(struct reader *r, struct block_reader *br,
-+ uint64_t next_off, byte want_typ) {
-+ int32_t guess_block_size = r->block_size ? r->block_size : DEFAULT_BLOCK_SIZE;
-+ struct block block = {};
-+ byte block_typ = 0;
-+ int err = 0;
-+ uint32_t header_off = next_off ? 0 : HEADER_SIZE;
-+ int32_t block_size = 0;
-+
-+ if (next_off >= r->size) {
-+ return 1;
-+ }
-+
-+ err = reader_get_block(r, &block, next_off, guess_block_size);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ block_size = extract_block_size(block.data, &block_typ, next_off);
-+ if (block_size < 0) {
-+ return block_size;
-+ }
-+
-+ if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
-+ reader_return_block(r, &block);
-+ return 1;
-+ }
-+
-+ if (block_size > guess_block_size) {
-+ reader_return_block(r, &block);
-+ err = reader_get_block(r, &block, next_off, block_size);
-+ if (err < 0) {
-+ return err;
-+ }
-+ }
-+
-+ return block_reader_init(br, &block, header_off, r->block_size, r->hash_size);
++ uint64_t next_off, byte want_typ)
++{
++ int32_t guess_block_size = r->block_size ? r->block_size :
++ DEFAULT_BLOCK_SIZE;
++ struct block block = {};
++ byte block_typ = 0;
++ int err = 0;
++ uint32_t header_off = next_off ? 0 : HEADER_SIZE;
++ int32_t block_size = 0;
++
++ if (next_off >= r->size) {
++ return 1;
++ }
++
++ err = reader_get_block(r, &block, next_off, guess_block_size);
++ if (err < 0) {
++ return err;
++ }
++
++ block_size = extract_block_size(block.data, &block_typ, next_off);
++ if (block_size < 0) {
++ return block_size;
++ }
++
++ if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
++ reader_return_block(r, &block);
++ return 1;
++ }
++
++ if (block_size > guess_block_size) {
++ reader_return_block(r, &block);
++ err = reader_get_block(r, &block, next_off, block_size);
++ if (err < 0) {
++ return err;
++ }
++ }
++
++ return block_reader_init(br, &block, header_off, r->block_size,
++ r->hash_size);
+}
+
+static int table_iter_next_block(struct table_iter *dest,
-+ struct table_iter *src) {
-+ uint64_t next_block_off = src->block_off + src->bi.br->full_block_size;
-+ struct block_reader br = {};
-+ int err = 0;
-+
-+ dest->r = src->r;
-+ dest->typ = src->typ;
-+ dest->block_off = next_block_off;
-+
-+ err = reader_init_block_reader(src->r, &br, next_block_off, src->typ);
-+ if (err > 0) {
-+ dest->finished = true;
-+ return 1;
-+ }
-+ if (err != 0) {
-+ return err;
-+ }
-+
-+ {
-+ struct block_reader *brp = malloc(sizeof(struct block_reader));
-+ *brp = br;
-+
-+ dest->finished = false;
-+ block_reader_start(brp, &dest->bi);
-+ }
-+ return 0;
-+}
-+
-+static int table_iter_next(struct table_iter *ti, struct record rec) {
-+ if (record_type(rec) != ti->typ) {
-+ return API_ERROR;
-+ }
-+
-+ while (true) {
-+ struct table_iter next = {};
-+ int err = 0;
-+ if (ti->finished) {
-+ return 1;
-+ }
-+
-+ err = table_iter_next_in_block(ti, rec);
-+ if (err <= 0) {
-+ return err;
-+ }
-+
-+ err = table_iter_next_block(&next, ti);
-+ if (err != 0) {
-+ ti->finished = true;
-+ }
-+ table_iter_block_done(ti);
-+ if (err != 0) {
-+ return err;
-+ }
-+ table_iter_copy_from(ti, &next);
-+ block_iter_close(&next.bi);
-+ }
-+}
-+
-+static int table_iter_next_void(void *ti, struct record rec) {
-+ return table_iter_next((struct table_iter *)ti, rec);
-+}
-+
-+static void table_iter_close(void *p) {
-+ struct table_iter *ti = (struct table_iter *)p;
-+ table_iter_block_done(ti);
-+ block_iter_close(&ti->bi);
++ struct table_iter *src)
++{
++ uint64_t next_block_off = src->block_off + src->bi.br->full_block_size;
++ struct block_reader br = {};
++ int err = 0;
++
++ dest->r = src->r;
++ dest->typ = src->typ;
++ dest->block_off = next_block_off;
++
++ err = reader_init_block_reader(src->r, &br, next_block_off, src->typ);
++ if (err > 0) {
++ dest->finished = true;
++ return 1;
++ }
++ if (err != 0) {
++ return err;
++ }
++
++ {
++ struct block_reader *brp = malloc(sizeof(struct block_reader));
++ *brp = br;
++
++ dest->finished = false;
++ block_reader_start(brp, &dest->bi);
++ }
++ return 0;
++}
++
++static int table_iter_next(struct table_iter *ti, struct record rec)
++{
++ if (record_type(rec) != ti->typ) {
++ return API_ERROR;
++ }
++
++ while (true) {
++ struct table_iter next = {};
++ int err = 0;
++ if (ti->finished) {
++ return 1;
++ }
++
++ err = table_iter_next_in_block(ti, rec);
++ if (err <= 0) {
++ return err;
++ }
++
++ err = table_iter_next_block(&next, ti);
++ if (err != 0) {
++ ti->finished = true;
++ }
++ table_iter_block_done(ti);
++ if (err != 0) {
++ return err;
++ }
++ table_iter_copy_from(ti, &next);
++ block_iter_close(&next.bi);
++ }
++}
++
++static int table_iter_next_void(void *ti, struct record rec)
++{
++ return table_iter_next((struct table_iter *)ti, rec);
++}
++
++static void table_iter_close(void *p)
++{
++ struct table_iter *ti = (struct table_iter *)p;
++ table_iter_block_done(ti);
++ block_iter_close(&ti->bi);
+}
+
+struct iterator_vtable table_iter_vtable = {
-+ .next = &table_iter_next_void,
-+ .close = &table_iter_close,
++ .next = &table_iter_next_void,
++ .close = &table_iter_close,
+};
+
-+static void iterator_from_table_iter(struct iterator *it,
-+ struct table_iter *ti) {
-+ it->iter_arg = ti;
-+ it->ops = &table_iter_vtable;
++static void iterator_from_table_iter(struct iterator *it, struct table_iter *ti)
++{
++ it->iter_arg = ti;
++ it->ops = &table_iter_vtable;
+}
+
+static int reader_table_iter_at(struct reader *r, struct table_iter *ti,
-+ uint64_t off, byte typ) {
-+ struct block_reader br = {};
-+ struct block_reader *brp = NULL;
++ uint64_t off, byte typ)
++{
++ struct block_reader br = {};
++ struct block_reader *brp = NULL;
+
-+ int err = reader_init_block_reader(r, &br, off, typ);
-+ if (err != 0) {
-+ return err;
-+ }
++ int err = reader_init_block_reader(r, &br, off, typ);
++ if (err != 0) {
++ return err;
++ }
+
-+ brp = malloc(sizeof(struct block_reader));
-+ *brp = br;
-+ ti->r = r;
-+ ti->typ = block_reader_type(brp);
-+ ti->block_off = off;
-+ block_reader_start(brp, &ti->bi);
-+ return 0;
++ brp = malloc(sizeof(struct block_reader));
++ *brp = br;
++ ti->r = r;
++ ti->typ = block_reader_type(brp);
++ ti->block_off = off;
++ block_reader_start(brp, &ti->bi);
++ return 0;
+}
+
+static int reader_start(struct reader *r, struct table_iter *ti, byte typ,
-+ bool index) {
-+ struct reader_offsets *offs = reader_offsets_for(r, typ);
-+ uint64_t off = offs->offset;
-+ if (index) {
-+ off = offs->index_offset;
-+ if (off == 0) {
-+ return 1;
-+ }
-+ typ = BLOCK_TYPE_INDEX;
-+ }
++ bool index)
++{
++ struct reader_offsets *offs = reader_offsets_for(r, typ);
++ uint64_t off = offs->offset;
++ if (index) {
++ off = offs->index_offset;
++ if (off == 0) {
++ return 1;
++ }
++ typ = BLOCK_TYPE_INDEX;
++ }
+
-+ return reader_table_iter_at(r, ti, off, typ);
++ return reader_table_iter_at(r, ti, off, typ);
+}
+
+static int reader_seek_linear(struct reader *r, struct table_iter *ti,
-+ struct record want) {
-+ struct record rec = new_record(record_type(want));
-+ struct slice want_key = {};
-+ struct slice got_key = {};
-+ struct table_iter next = {};
-+ int err = -1;
-+ record_key(want, &want_key);
-+
-+ while (true) {
-+ err = table_iter_next_block(&next, ti);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+
-+ if (err > 0) {
-+ break;
-+ }
-+
-+ err = block_reader_first_key(next.bi.br, &got_key);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+ {
-+ int cmp = slice_compare(got_key, want_key);
-+ if (cmp > 0) {
-+ table_iter_block_done(&next);
-+ break;
-+ }
-+ }
-+
-+ table_iter_block_done(ti);
-+ table_iter_copy_from(ti, &next);
-+ }
-+
-+ err = block_iter_seek(&ti->bi, want_key);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+ err = 0;
++ struct record want)
++{
++ struct record rec = new_record(record_type(want));
++ struct slice want_key = {};
++ struct slice got_key = {};
++ struct table_iter next = {};
++ int err = -1;
++ record_key(want, &want_key);
++
++ while (true) {
++ err = table_iter_next_block(&next, ti);
++ if (err < 0) {
++ goto exit;
++ }
++
++ if (err > 0) {
++ break;
++ }
++
++ err = block_reader_first_key(next.bi.br, &got_key);
++ if (err < 0) {
++ goto exit;
++ }
++ {
++ int cmp = slice_compare(got_key, want_key);
++ if (cmp > 0) {
++ table_iter_block_done(&next);
++ break;
++ }
++ }
++
++ table_iter_block_done(ti);
++ table_iter_copy_from(ti, &next);
++ }
++
++ err = block_iter_seek(&ti->bi, want_key);
++ if (err < 0) {
++ goto exit;
++ }
++ err = 0;
+
+exit:
-+ block_iter_close(&next.bi);
-+ record_clear(rec);
-+ free(record_yield(&rec));
-+ free(slice_yield(&want_key));
-+ free(slice_yield(&got_key));
-+ return err;
++ block_iter_close(&next.bi);
++ record_clear(rec);
++ free(record_yield(&rec));
++ free(slice_yield(&want_key));
++ free(slice_yield(&got_key));
++ return err;
+}
+
+static int reader_seek_indexed(struct reader *r, struct iterator *it,
-+ struct record rec) {
-+ struct index_record want_index = {};
-+ struct record want_index_rec = {};
-+ struct index_record index_result = {};
-+ struct record index_result_rec = {};
-+ struct table_iter index_iter = {};
-+ struct table_iter next = {};
-+ int err = 0;
-+
-+ record_key(rec, &want_index.last_key);
-+ record_from_index(&want_index_rec, &want_index);
-+ record_from_index(&index_result_rec, &index_result);
-+
-+ err = reader_start(r, &index_iter, record_type(rec), true);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+
-+ err = reader_seek_linear(r, &index_iter, want_index_rec);
-+ while (true) {
-+ err = table_iter_next(&index_iter, index_result_rec);
-+ table_iter_block_done(&index_iter);
-+ if (err != 0) {
-+ goto exit;
-+ }
-+
-+ err = reader_table_iter_at(r, &next, index_result.offset, 0);
-+ if (err != 0) {
-+ goto exit;
-+ }
-+
-+ err = block_iter_seek(&next.bi, want_index.last_key);
-+ if (err < 0) {
-+ goto exit;
-+ }
-+
-+ if (next.typ == record_type(rec)) {
-+ err = 0;
-+ break;
-+ }
-+
-+ if (next.typ != BLOCK_TYPE_INDEX) {
-+ err = FORMAT_ERROR;
-+ break;
-+ }
-+
-+ table_iter_copy_from(&index_iter, &next);
-+ }
-+
-+ if (err == 0) {
-+ struct table_iter *malloced = calloc(sizeof(struct table_iter), 1);
-+ table_iter_copy_from(malloced, &next);
-+ iterator_from_table_iter(it, malloced);
-+ }
++ struct record rec)
++{
++ struct index_record want_index = {};
++ struct record want_index_rec = {};
++ struct index_record index_result = {};
++ struct record index_result_rec = {};
++ struct table_iter index_iter = {};
++ struct table_iter next = {};
++ int err = 0;
++
++ record_key(rec, &want_index.last_key);
++ record_from_index(&want_index_rec, &want_index);
++ record_from_index(&index_result_rec, &index_result);
++
++ err = reader_start(r, &index_iter, record_type(rec), true);
++ if (err < 0) {
++ goto exit;
++ }
++
++ err = reader_seek_linear(r, &index_iter, want_index_rec);
++ while (true) {
++ err = table_iter_next(&index_iter, index_result_rec);
++ table_iter_block_done(&index_iter);
++ if (err != 0) {
++ goto exit;
++ }
++
++ err = reader_table_iter_at(r, &next, index_result.offset, 0);
++ if (err != 0) {
++ goto exit;
++ }
++
++ err = block_iter_seek(&next.bi, want_index.last_key);
++ if (err < 0) {
++ goto exit;
++ }
++
++ if (next.typ == record_type(rec)) {
++ err = 0;
++ break;
++ }
++
++ if (next.typ != BLOCK_TYPE_INDEX) {
++ err = FORMAT_ERROR;
++ break;
++ }
++
++ table_iter_copy_from(&index_iter, &next);
++ }
++
++ if (err == 0) {
++ struct table_iter *malloced =
++ calloc(sizeof(struct table_iter), 1);
++ table_iter_copy_from(malloced, &next);
++ iterator_from_table_iter(it, malloced);
++ }
+exit:
-+ block_iter_close(&next.bi);
-+ table_iter_close(&index_iter);
-+ record_clear(want_index_rec);
-+ record_clear(index_result_rec);
-+ return err;
++ block_iter_close(&next.bi);
++ table_iter_close(&index_iter);
++ record_clear(want_index_rec);
++ record_clear(index_result_rec);
++ return err;
+}
+
+static int reader_seek_internal(struct reader *r, struct iterator *it,
-+ struct record rec) {
-+ struct reader_offsets *offs = reader_offsets_for(r, record_type(rec));
-+ uint64_t idx = offs->index_offset;
-+ struct table_iter ti = {};
-+ int err = 0;
-+ if (idx > 0) {
-+ return reader_seek_indexed(r, it, rec);
-+ }
-+
-+ err = reader_start(r, &ti, record_type(rec), false);
-+ if (err < 0) {
-+ return err;
-+ }
-+ err = reader_seek_linear(r, &ti, rec);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ {
-+ struct table_iter *p = malloc(sizeof(struct table_iter));
-+ *p = ti;
-+ iterator_from_table_iter(it, p);
-+ }
-+
-+ return 0;
-+}
-+
-+int reader_seek(struct reader *r, struct iterator *it, struct record rec) {
-+ byte typ = record_type(rec);
-+
-+ struct reader_offsets *offs = reader_offsets_for(r, typ);
-+ if (!offs->present) {
-+ iterator_set_empty(it);
-+ return 0;
-+ }
-+
-+ return reader_seek_internal(r, it, rec);
-+}
-+
-+int reader_seek_ref(struct reader *r, struct iterator *it, const char *name) {
-+ struct ref_record ref = {
-+ .ref_name = (char *)name,
-+ };
-+ struct record rec = {};
-+ record_from_ref(&rec, &ref);
-+ return reader_seek(r, it, rec);
++ struct record rec)
++{
++ struct reader_offsets *offs = reader_offsets_for(r, record_type(rec));
++ uint64_t idx = offs->index_offset;
++ struct table_iter ti = {};
++ int err = 0;
++ if (idx > 0) {
++ return reader_seek_indexed(r, it, rec);
++ }
++
++ err = reader_start(r, &ti, record_type(rec), false);
++ if (err < 0) {
++ return err;
++ }
++ err = reader_seek_linear(r, &ti, rec);
++ if (err < 0) {
++ return err;
++ }
++
++ {
++ struct table_iter *p = malloc(sizeof(struct table_iter));
++ *p = ti;
++ iterator_from_table_iter(it, p);
++ }
++
++ return 0;
++}
++
++int reader_seek(struct reader *r, struct iterator *it, struct record rec)
++{
++ byte typ = record_type(rec);
++
++ struct reader_offsets *offs = reader_offsets_for(r, typ);
++ if (!offs->present) {
++ iterator_set_empty(it);
++ return 0;
++ }
++
++ return reader_seek_internal(r, it, rec);
++}
++
++int reader_seek_ref(struct reader *r, struct iterator *it, const char *name)
++{
++ struct ref_record ref = {
++ .ref_name = (char *)name,
++ };
++ struct record rec = {};
++ record_from_ref(&rec, &ref);
++ return reader_seek(r, it, rec);
+}
+
+int reader_seek_log_at(struct reader *r, struct iterator *it, const char *name,
-+ uint64_t update_index) {
-+ struct log_record log = {
-+ .ref_name = (char *)name,
-+ .update_index = update_index,
-+ };
-+ struct record rec = {};
-+ record_from_log(&rec, &log);
-+ return reader_seek(r, it, rec);
++ uint64_t update_index)
++{
++ struct log_record log = {
++ .ref_name = (char *)name,
++ .update_index = update_index,
++ };
++ struct record rec = {};
++ record_from_log(&rec, &log);
++ return reader_seek(r, it, rec);
+}
+
-+int reader_seek_log(struct reader *r, struct iterator *it, const char *name) {
-+ uint64_t max = ~((uint64_t)0);
-+ return reader_seek_log_at(r, it, name, max);
++int reader_seek_log(struct reader *r, struct iterator *it, const char *name)
++{
++ uint64_t max = ~((uint64_t)0);
++ return reader_seek_log_at(r, it, name, max);
+}
+
-+void reader_close(struct reader *r) {
-+ block_source_close(&r->source);
-+ free(r->name);
-+ r->name = NULL;
++void reader_close(struct reader *r)
++{
++ block_source_close(&r->source);
++ free(r->name);
++ r->name = NULL;
+}
+
-+int new_reader(struct reader **p, struct block_source src, char const *name) {
-+ struct reader *rd = calloc(sizeof(struct reader), 1);
-+ int err = init_reader(rd, src, name);
-+ if (err == 0) {
-+ *p = rd;
-+ } else {
-+ free(rd);
-+ }
-+ return err;
++int new_reader(struct reader **p, struct block_source src, char const *name)
++{
++ struct reader *rd = calloc(sizeof(struct reader), 1);
++ int err = init_reader(rd, src, name);
++ if (err == 0) {
++ *p = rd;
++ } else {
++ free(rd);
++ }
++ return err;
+}
+
-+void reader_free(struct reader *r) {
-+ reader_close(r);
-+ free(r);
++void reader_free(struct reader *r)
++{
++ reader_close(r);
++ free(r);
+}
+
+static int reader_refs_for_indexed(struct reader *r, struct iterator *it,
-+ byte *oid) {
-+ struct obj_record want = {
-+ .hash_prefix = oid,
-+ .hash_prefix_len = r->object_id_len,
-+ };
-+ struct record want_rec = {};
-+ struct iterator oit = {};
-+ struct obj_record got = {};
-+ struct record got_rec = {};
-+ int err = 0;
-+
-+ record_from_obj(&want_rec, &want);
-+
-+ err = reader_seek(r, &oit, want_rec);
-+ if (err != 0) {
-+ return err;
-+ }
-+
-+ record_from_obj(&got_rec, &got);
-+ err = iterator_next(oit, got_rec);
-+ iterator_destroy(&oit);
-+ if (err < 0) {
-+ return err;
-+ }
-+
-+ if (err > 0 || memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
-+ iterator_set_empty(it);
-+ return 0;
-+ }
-+
-+ {
-+ struct indexed_table_ref_iter *itr = NULL;
-+ err = new_indexed_table_ref_iter(&itr, r, oid, r->hash_size, got.offsets,
-+ got.offset_len);
-+ if (err < 0) {
-+ record_clear(got_rec);
-+ return err;
-+ }
-+ got.offsets = NULL;
-+ record_clear(got_rec);
-+
-+ iterator_from_indexed_table_ref_iter(it, itr);
-+ }
-+
-+ return 0;
++ byte *oid)
++{
++ struct obj_record want = {
++ .hash_prefix = oid,
++ .hash_prefix_len = r->object_id_len,
++ };
++ struct record want_rec = {};
++ struct iterator oit = {};
++ struct obj_record got = {};
++ struct record got_rec = {};
++ int err = 0;
++
++ record_from_obj(&want_rec, &want);
++
++ err = reader_seek(r, &oit, want_rec);
++ if (err != 0) {
++ return err;
++ }
++
++ record_from_obj(&got_rec, &got);
++ err = iterator_next(oit, got_rec);
++ iterator_destroy(&oit);
++ if (err < 0) {
++ return err;
++ }
++
++ if (err > 0 ||
++ memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
++ iterator_set_empty(it);
++ return 0;
++ }
++
++ {
++ struct indexed_table_ref_iter *itr = NULL;
++ err = new_indexed_table_ref_iter(&itr, r, oid, r->hash_size,
++ got.offsets, got.offset_len);
++ if (err < 0) {
++ record_clear(got_rec);
++ return err;
++ }
++ got.offsets = NULL;
++ record_clear(got_rec);
++
++ iterator_from_indexed_table_ref_iter(it, itr);
++ }
++
++ return 0;
+}
+
+static int reader_refs_for_unindexed(struct reader *r, struct iterator *it,
-+ byte *oid, int oid_len) {
-+ struct table_iter *ti = calloc(sizeof(struct table_iter), 1);
-+ struct filtering_ref_iterator *filter = NULL;
-+ int err = reader_start(r, ti, BLOCK_TYPE_REF, false);
-+ if (err < 0) {
-+ free(ti);
-+ return err;
-+ }
-+
-+ filter = calloc(sizeof(struct filtering_ref_iterator), 1);
-+ slice_resize(&filter->oid, oid_len);
-+ memcpy(filter->oid.buf, oid, oid_len);
-+ filter->r = r;
-+ filter->double_check = false;
-+ iterator_from_table_iter(&filter->it, ti);
-+
-+ iterator_from_filtering_ref_iterator(it, filter);
-+ return 0;
++ byte *oid, int oid_len)
++{
++ struct table_iter *ti = calloc(sizeof(struct table_iter), 1);
++ struct filtering_ref_iterator *filter = NULL;
++ int err = reader_start(r, ti, BLOCK_TYPE_REF, false);
++ if (err < 0) {
++ free(ti);
++ return err;
++ }
++
++ filter = calloc(sizeof(struct filtering_ref_iterator), 1);
++ slice_resize(&filter->oid, oid_len);
++ memcpy(filter->oid.buf, oid, oid_len);
++ filter->r = r;
++ filter->double_check = false;
++ iterator_from_table_iter(&filter->it, ti);
++
++ iterator_from_filtering_ref_iterator(it, filter);
++ return 0;
+}
+
+int reader_refs_for(struct reader *r, struct iterator *it, byte *oid,
-+ int oid_len) {
-+ if (r->obj_offsets.present) {
-+ return reader_refs_for_indexed(r, it, oid);
-+ }
-+ return reader_refs_for_unindexed(r, it, oid, oid_len);
++ int oid_len)
++{
++ if (r->obj_offsets.present) {
++ return reader_refs_for_indexed(r, it, oid);
++ }
++ return reader_refs_for_unindexed(r, it, oid, oid_len);
+}
+
-+uint64_t reader_max_update_index(struct reader *r) {
-+ return r->max_update_index;
++uint64_t reader_max_update_index(struct reader *r)
++{
++ return r->max_update_index;
+}
+
-+uint64_t reader_min_update_index(struct reader *r) {
-+ return r->min_update_index;
++uint64_t reader_min_update_index(struct reader *r)
++{
++ return r->min_update_index;
+}
diff --git a/reftable/reader.h b/reftable/reader.h
@@ -3145,29 +3070,29 @@
+uint64_t block_source_size(struct block_source source);
+
+int block_source_read_block(struct block_source source, struct block *dest,
-+ uint64_t off, uint32_t size);
++ uint64_t off, uint32_t size);
+void block_source_return_block(struct block_source source, struct block *ret);
+void block_source_close(struct block_source *source);
+
+struct reader_offsets {
-+ bool present;
-+ uint64_t offset;
-+ uint64_t index_offset;
++ bool present;
++ uint64_t offset;
++ uint64_t index_offset;
+};
+
+struct reader {
-+ struct block_source source;
-+ char *name;
-+ int hash_size;
-+ uint64_t size;
-+ uint32_t block_size;
-+ uint64_t min_update_index;
-+ uint64_t max_update_index;
-+ int object_id_len;
-+
-+ struct reader_offsets ref_offsets;
-+ struct reader_offsets obj_offsets;
-+ struct reader_offsets log_offsets;
++ struct block_source source;
++ char *name;
++ int hash_size;
++ uint64_t size;
++ uint32_t block_size;
++ uint64_t min_update_index;
++ uint64_t max_update_index;
++ int object_id_len;
++
++ struct reader_offsets ref_offsets;
++ struct reader_offsets obj_offsets;
++ struct reader_offsets log_offsets;
+};
+
+int init_reader(struct reader *r, struct block_source source, const char *name);
@@ -3176,7 +3101,7 @@
+const char *reader_name(struct reader *r);
+void reader_return_block(struct reader *r, struct block *p);
+int reader_init_block_reader(struct reader *r, struct block_reader *br,
-+ uint64_t next_off, byte want_typ);
++ uint64_t next_off, byte want_typ);
+
+#endif
@@ -3195,1025 +3120,1105 @@
+
+#include "record.h"
+
-+#include <assert.h>
-+#include <stdio.h>
-+#include <stdlib.h>
-+#include <string.h>
++#include "system.h"
+
+#include "constants.h"
+#include "reftable.h"
+
-+int is_block_type(byte typ) {
-+ switch (typ) {
-+ case BLOCK_TYPE_REF:
-+ case BLOCK_TYPE_LOG:
-+ case BLOCK_TYPE_OBJ:
-+ case BLOCK_TYPE_INDEX:
-+ return true;
-+ }
-+ return false;
-+}
-+
-+int get_var_int(uint64_t *dest, struct slice in) {
-+ int ptr = 0;
-+ uint64_t val;
-+
-+ if (in.len == 0) {
-+ return -1;
-+ }
-+ val = in.buf[ptr] & 0x7f;
-+
-+ while (in.buf[ptr] & 0x80) {
-+ ptr++;
-+ if (ptr > in.len) {
-+ return -1;
-+ }
-+ val = (val + 1) << 7 | (uint64_t)(in.buf[ptr] & 0x7f);
-+ }
-+
-+ *dest = val;
-+ return ptr + 1;
-+}
-+
-+int put_var_int(struct slice dest, uint64_t val) {
-+ byte buf[10] = {};
-+ int i = 9;
-+ buf[i] = (byte)(val & 0x7f);
-+ i--;
-+ while (true) {
-+ val >>= 7;
-+ if (!val) {
-+ break;
-+ }
-+ val--;
-+ buf[i] = 0x80 | (byte)(val & 0x7f);
-+ i--;
-+ }
-+
-+ {
-+ int n = sizeof(buf) - i - 1;
-+ if (dest.len < n) {
-+ return -1;
-+ }
-+ memcpy(dest.buf, &buf[i + 1], n);
-+ return n;
-+ }
-+}
-+
-+int common_prefix_size(struct slice a, struct slice b) {
-+ int p = 0;
-+ while (p < a.len && p < b.len) {
-+ if (a.buf[p] != b.buf[p]) {
-+ break;
-+ }
-+ p++;
-+ }
-+
-+ return p;
-+}
-+
-+static int decode_string(struct slice *dest, struct slice in) {
-+ int start_len = in.len;
-+ uint64_t tsize = 0;
-+ int n = get_var_int(&tsize, in);
-+ if (n <= 0) {
-+ return -1;
-+ }
-+ in.buf += n;
-+ in.len -= n;
-+ if (in.len < tsize) {
-+ return -1;
-+ }
-+
-+ slice_resize(dest, tsize + 1);
-+ dest->buf[tsize] = 0;
-+ memcpy(dest->buf, in.buf, tsize);
-+ in.buf += tsize;
-+ in.len -= tsize;
-+
-+ return start_len - in.len;
++int is_block_type(byte typ)
++{
++ switch (typ) {
++ case BLOCK_TYPE_REF:
++ case BLOCK_TYPE_LOG:
++ case BLOCK_TYPE_OBJ:
++ case BLOCK_TYPE_INDEX:
++ return true;
++ }
++ return false;
++}
++
++int get_var_int(uint64_t *dest, struct slice in)
++{
++ int ptr = 0;
++ uint64_t val;
++
++ if (in.len == 0) {
++ return -1;
++ }
++ val = in.buf[ptr] & 0x7f;
++
++ while (in.buf[ptr] & 0x80) {
++ ptr++;
++ if (ptr > in.len) {
++ return -1;
++ }
++ val = (val + 1) << 7 | (uint64_t)(in.buf[ptr] & 0x7f);
++ }
++
++ *dest = val;
++ return ptr + 1;
++}
++
++int put_var_int(struct slice dest, uint64_t val)
++{
++ byte buf[10] = {};
++ int i = 9;
++ buf[i] = (byte)(val & 0x7f);
++ i--;
++ while (true) {
++ val >>= 7;
++ if (!val) {
++ break;
++ }
++ val--;
++ buf[i] = 0x80 | (byte)(val & 0x7f);
++ i--;
++ }
++
++ {
++ int n = sizeof(buf) - i - 1;
++ if (dest.len < n) {
++ return -1;
++ }
++ memcpy(dest.buf, &buf[i + 1], n);
++ return n;
++ }
++}
++
++int common_prefix_size(struct slice a, struct slice b)
++{
++ int p = 0;
++ while (p < a.len && p < b.len) {
++ if (a.buf[p] != b.buf[p]) {
++ break;
++ }
++ p++;
++ }
++
++ return p;
++}
++
++static int decode_string(struct slice *dest, struct slice in)
++{
++ int start_len = in.len;
++ uint64_t tsize = 0;
++ int n = get_var_int(&tsize, in);
++ if (n <= 0) {
++ return -1;
++ }
++ in.buf += n;
++ in.len -= n;
++ if (in.len < tsize) {
++ return -1;
++ }
++
++ slice_resize(dest, tsize + 1);
++ dest->buf[tsize] = 0;
++ memcpy(dest->buf, in.buf, tsize);
++ in.buf += tsize;
++ in.len -= tsize;
++
++ return start_len - in.len;
+}
+
+int encode_key(bool *restart, struct slice dest, struct slice prev_key,
-+ struct slice key, byte extra) {
-+ struct slice start = dest;
-+ int prefix_len = common_prefix_size(prev_key, key);
-+ uint64_t suffix_len = key.len - prefix_len;
-+ int n = put_var_int(dest, (uint64_t)prefix_len);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ dest.buf += n;
-+ dest.len -= n;
-+
-+ *restart = (prefix_len == 0);
-+
-+ n = put_var_int(dest, suffix_len << 3 | (uint64_t)extra);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ dest.buf += n;
-+ dest.len -= n;
-+
-+ if (dest.len < suffix_len) {
-+ return -1;
-+ }
-+ memcpy(dest.buf, key.buf + prefix_len, suffix_len);
-+ dest.buf += suffix_len;
-+ dest.len -= suffix_len;
-+
-+ return start.len - dest.len;
-+}
-+
-+static byte ref_record_type(void) { return BLOCK_TYPE_REF; }
-+
-+static void ref_record_key(const void *r, struct slice *dest) {
-+ const struct ref_record *rec = (const struct ref_record *)r;
-+ slice_set_string(dest, rec->ref_name);
-+}
-+
-+static void ref_record_copy_from(void *rec, const void *src_rec,
-+ int hash_size) {
-+ struct ref_record *ref = (struct ref_record *)rec;
-+ struct ref_record *src = (struct ref_record *)src_rec;
-+ assert(hash_size > 0);
-+
-+ // This is simple and correct, but we could probably reuse the hash fields.
-+ ref_record_clear(ref);
-+ if (src->ref_name != NULL) {
-+ ref->ref_name = strdup(src->ref_name);
-+ }
-+
-+ if (src->target != NULL) {
-+ ref->target = strdup(src->target);
-+ }
-+
-+ if (src->target_value != NULL) {
-+ ref->target_value = malloc(hash_size);
-+ memcpy(ref->target_value, src->target_value, hash_size);
-+ }
-+
-+ if (src->value != NULL) {
-+ ref->value = malloc(hash_size);
-+ memcpy(ref->value, src->value, hash_size);
-+ }
-+ ref->update_index = src->update_index;
-+}
-+
-+static char hexdigit(int c) {
-+ if (c <= 9) {
-+ return '0' + c;
-+ }
-+ return 'a' + (c - 10);
-+}
-+
-+static void hex_format(char *dest, byte *src, int hash_size) {
-+ assert(hash_size > 0);
-+ if (src != NULL) {
-+ for (int i = 0; i < hash_size; i++) {
-+ dest[2 * i] = hexdigit(src[i] >> 4);
-+ dest[2 * i + 1] = hexdigit(src[i] & 0xf);
-+ }
-+ dest[2 * hash_size] = 0;
-+ }
-+}
-+
-+void ref_record_print(struct ref_record *ref, int hash_size) {
-+ char hex[SHA256_SIZE + 1] = {};
-+
-+ printf("ref{%s(%ld) ", ref->ref_name, ref->update_index);
-+ if (ref->value != NULL) {
-+ hex_format(hex, ref->value, hash_size);
-+ printf("%s", hex);
-+ }
-+ if (ref->target_value != NULL) {
-+ hex_format(hex, ref->target_value, hash_size);
-+ printf(" (T %s)", hex);
-+ }
-+ if (ref->target != NULL) {
-+ printf("=> %s", ref->target);
-+ }
-+ printf("}\n");
-+}
-+
-+static void ref_record_clear_void(void *rec) {
-+ ref_record_clear((struct ref_record *)rec);
-+}
-+
-+void ref_record_clear(struct ref_record *ref) {
-+ free(ref->ref_name);
-+ free(ref->target);
-+ free(ref->target_value);
-+ free(ref->value);
-+ memset(ref, 0, sizeof(struct ref_record));
-+}
-+
-+static byte ref_record_val_type(const void *rec) {
-+ const struct ref_record *r = (const struct ref_record *)rec;
-+ if (r->value != NULL) {
-+ if (r->target_value != NULL) {
-+ return 2;
-+ } else {
-+ return 1;
-+ }
-+ } else if (r->target != NULL) {
-+ return 3;
-+ }
-+ return 0;
-+}
-+
-+static int encode_string(char *str, struct slice s) {
-+ struct slice start = s;
-+ int l = strlen(str);
-+ int n = put_var_int(s, l);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ s.buf += n;
-+ s.len -= n;
-+ if (s.len < l) {
-+ return -1;
-+ }
-+ memcpy(s.buf, str, l);
-+ s.buf += l;
-+ s.len -= l;
-+
-+ return start.len - s.len;
-+}
-+
-+static int ref_record_encode(const void *rec, struct slice s, int hash_size) {
-+ const struct ref_record *r = (const struct ref_record *)rec;
-+ struct slice start = s;
-+ int n = put_var_int(s, r->update_index);
-+ assert(hash_size > 0);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ s.buf += n;
-+ s.len -= n;
-+
-+ if (r->value != NULL) {
-+ if (s.len < hash_size) {
-+ return -1;
-+ }
-+ memcpy(s.buf, r->value, hash_size);
-+ s.buf += hash_size;
-+ s.len -= hash_size;
-+ }
-+
-+ if (r->target_value != NULL) {
-+ if (s.len < hash_size) {
-+ return -1;
-+ }
-+ memcpy(s.buf, r->target_value, hash_size);
-+ s.buf += hash_size;
-+ s.len -= hash_size;
-+ }
-+
-+ if (r->target != NULL) {
-+ int n = encode_string(r->target, s);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ s.buf += n;
-+ s.len -= n;
-+ }
-+
-+ return start.len - s.len;
++ struct slice key, byte extra)
++{
++ struct slice start = dest;
++ int prefix_len = common_prefix_size(prev_key, key);
++ uint64_t suffix_len = key.len - prefix_len;
++ int n = put_var_int(dest, (uint64_t)prefix_len);
++ if (n < 0) {
++ return -1;
++ }
++ dest.buf += n;
++ dest.len -= n;
++
++ *restart = (prefix_len == 0);
++
++ n = put_var_int(dest, suffix_len << 3 | (uint64_t)extra);
++ if (n < 0) {
++ return -1;
++ }
++ dest.buf += n;
++ dest.len -= n;
++
++ if (dest.len < suffix_len) {
++ return -1;
++ }
++ memcpy(dest.buf, key.buf + prefix_len, suffix_len);
++ dest.buf += suffix_len;
++ dest.len -= suffix_len;
++
++ return start.len - dest.len;
++}
++
++static byte ref_record_type(void)
++{
++ return BLOCK_TYPE_REF;
++}
++
++static void ref_record_key(const void *r, struct slice *dest)
++{
++ const struct ref_record *rec = (const struct ref_record *)r;
++ slice_set_string(dest, rec->ref_name);
++}
++
++static void ref_record_copy_from(void *rec, const void *src_rec, int hash_size)
++{
++ struct ref_record *ref = (struct ref_record *)rec;
++ struct ref_record *src = (struct ref_record *)src_rec;
++ assert(hash_size > 0);
++
++ /* This is simple and correct, but we could probably reuse the hash
++ fields. */
++ ref_record_clear(ref);
++ if (src->ref_name != NULL) {
++ ref->ref_name = strdup(src->ref_name);
++ }
++
++ if (src->target != NULL) {
++ ref->target = strdup(src->target);
++ }
++
++ if (src->target_value != NULL) {
++ ref->target_value = malloc(hash_size);
++ memcpy(ref->target_value, src->target_value, hash_size);
++ }
++
++ if (src->value != NULL) {
++ ref->value = malloc(hash_size);
++ memcpy(ref->value, src->value, hash_size);
++ }
++ ref->update_index = src->update_index;
++}
++
++static char hexdigit(int c)
++{
++ if (c <= 9) {
++ return '0' + c;
++ }
++ return 'a' + (c - 10);
++}
++
++static void hex_format(char *dest, byte *src, int hash_size)
++{
++ assert(hash_size > 0);
++ if (src != NULL) {
++ int i = 0;
++ for (i = 0; i < hash_size; i++) {
++ dest[2 * i] = hexdigit(src[i] >> 4);
++ dest[2 * i + 1] = hexdigit(src[i] & 0xf);
++ }
++ dest[2 * hash_size] = 0;
++ }
++}
++
++void ref_record_print(struct ref_record *ref, int hash_size)
++{
++ char hex[SHA256_SIZE + 1] = {};
++
++ printf("ref{%s(%" PRIdMAX ") ", ref->ref_name, ref->update_index);
++ if (ref->value != NULL) {
++ hex_format(hex, ref->value, hash_size);
++ printf("%s", hex);
++ }
++ if (ref->target_value != NULL) {
++ hex_format(hex, ref->target_value, hash_size);
++ printf(" (T %s)", hex);
++ }
++ if (ref->target != NULL) {
++ printf("=> %s", ref->target);
++ }
++ printf("}\n");
++}
++
++static void ref_record_clear_void(void *rec)
++{
++ ref_record_clear((struct ref_record *)rec);
++}
++
++void ref_record_clear(struct ref_record *ref)
++{
++ free(ref->ref_name);
++ free(ref->target);
++ free(ref->target_value);
++ free(ref->value);
++ memset(ref, 0, sizeof(struct ref_record));
++}
++
++static byte ref_record_val_type(const void *rec)
++{
++ const struct ref_record *r = (const struct ref_record *)rec;
++ if (r->value != NULL) {
++ if (r->target_value != NULL) {
++ return 2;
++ } else {
++ return 1;
++ }
++ } else if (r->target != NULL) {
++ return 3;
++ }
++ return 0;
++}
++
++static int encode_string(char *str, struct slice s)
++{
++ struct slice start = s;
++ int l = strlen(str);
++ int n = put_var_int(s, l);
++ if (n < 0) {
++ return -1;
++ }
++ s.buf += n;
++ s.len -= n;
++ if (s.len < l) {
++ return -1;
++ }
++ memcpy(s.buf, str, l);
++ s.buf += l;
++ s.len -= l;
++
++ return start.len - s.len;
++}
++
++static int ref_record_encode(const void *rec, struct slice s, int hash_size)
++{
++ const struct ref_record *r = (const struct ref_record *)rec;
++ struct slice start = s;
++ int n = put_var_int(s, r->update_index);
++ assert(hash_size > 0);
++ if (n < 0) {
++ return -1;
++ }
++ s.buf += n;
++ s.len -= n;
++
++ if (r->value != NULL) {
++ if (s.len < hash_size) {
++ return -1;
++ }
++ memcpy(s.buf, r->value, hash_size);
++ s.buf += hash_size;
++ s.len -= hash_size;
++ }
++
++ if (r->target_value != NULL) {
++ if (s.len < hash_size) {
++ return -1;
++ }
++ memcpy(s.buf, r->target_value, hash_size);
++ s.buf += hash_size;
++ s.len -= hash_size;
++ }
++
++ if (r->target != NULL) {
++ int n = encode_string(r->target, s);
++ if (n < 0) {
++ return -1;
++ }
++ s.buf += n;
++ s.len -= n;
++ }
++
++ return start.len - s.len;
+}
+
+static int ref_record_decode(void *rec, struct slice key, byte val_type,
-+ struct slice in, int hash_size) {
-+ struct ref_record *r = (struct ref_record *)rec;
-+ struct slice start = in;
-+ bool seen_value = false;
-+ bool seen_target_value = false;
-+ bool seen_target = false;
-+
-+ int n = get_var_int(&r->update_index, in);
-+ if (n < 0) {
-+ return n;
-+ }
-+ assert(hash_size > 0);
-+
-+ in.buf += n;
-+ in.len -= n;
-+
-+ r->ref_name = realloc(r->ref_name, key.len + 1);
-+ memcpy(r->ref_name, key.buf, key.len);
-+ r->ref_name[key.len] = 0;
-+
-+ switch (val_type) {
-+ case 1:
-+ case 2:
-+ if (in.len < hash_size) {
-+ return -1;
-+ }
-+
-+ if (r->value == NULL) {
-+ r->value = malloc(hash_size);
-+ }
-+ seen_value = true;
-+ memcpy(r->value, in.buf, hash_size);
-+ in.buf += hash_size;
-+ in.len -= hash_size;
-+ if (val_type == 1) {
-+ break;
-+ }
-+ if (r->target_value == NULL) {
-+ r->target_value = malloc(hash_size);
-+ }
-+ seen_target_value = true;
-+ memcpy(r->target_value, in.buf, hash_size);
-+ in.buf += hash_size;
-+ in.len -= hash_size;
-+ break;
-+ case 3: {
-+ struct slice dest = {};
-+ int n = decode_string(&dest, in);
-+ if (n < 0) {
-+ return -1;
-+ }
-+ in.buf += n;
-+ in.len -= n;
-+ seen_target = true;
-+ r->target = (char *)slice_as_string(&dest);
-+ } break;
-+
-+ case 0:
-+ break;
-+ default:
-+ abort();
-+ break;
-+ }
-+
-+ if (!seen_target && r->target != NULL) {
-+ free(r->target);
-+ r->target = NULL;
-+ }
-+ if (!seen_target_value && r->target_value != NULL) {
-+ free(r->target_value);
-+ r->target_value = NULL;
-+ }
-+ if (!seen_value && r->value != NULL) {
-+ free(r->value);
-+ r->value = NULL;8,927 lines not shown · raw message
1 line not shown · raw message
Part of this message is not shown here. View the raw message.
PATCH v2 1/5 setup.c: enable repo detection for reftable
From: Han-Wen Nienhuys <[email protected]>
* only check R_OK for .git/refs.
In the reftable format, tables are stored under $GITDIR/reftable/ and the list of tables is managed in a file $GITDIR/refs. Generally, this file does not have the +x bit set
* allow missing HEAD if there is a reftable/
Change-Id I5d22317a15a84c8529aa503ae357a4afba247fe9
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- setup.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
setup.c +14 -6
@@ -304,22 +304,23 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)* - either an objects/ directory _or_ the proper * GIT_OBJECT_DIRECTORY environment variable * - a refs/ directory -* - either a HEAD symlink or a HEAD file that is formatted as -* a proper "ref:", or a regular file HEAD that has a properly -* formatted sha1 object name. +* - a reftable/ director or, either a HEAD ref. +* The HEAD ref should be a HEAD symlink or a HEAD file that is formatted as +* a proper "ref:", or a regular file HEAD that has a properly formatted sha1 +* object name. */ int is_git_directory (const char *suspect) {struct strbuf path =STRBUF_INIT ;int ret =0 ; +int have_head =0 ;size_t len;/* Check worktree-related signatures */ strbuf_addstr (&path, suspect);strbuf_complete (&path,'/' );strbuf_addstr (&path,"HEAD" ); -if (validate_headref (path.buf)) - goto done; + have_head = !validate_headref (path.buf);strbuf_reset (&path);get_common_dir (&path, suspect);
@@ -339,9 +340,16 @@ int is_git_directory(const char *suspect)strbuf_setlen (&path, len);strbuf_addstr (&path,"/refs" ); -if (access (path.buf,X_OK )) +if (access (path.buf,R_OK )) goto done; +if (!have_head) { +strbuf_setlen (&path, len); +strbuf_addstr (&path,"/reftable" ); +if (access (path.buf,X_OK )) + goto done; + } + ret =1 ; done:strbuf_release (&path);
Signature · 1 line
gitgitgadget
PATCH v2 2/5 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which creates a file in that place.
Signed-off-by Han-Wen Nienhuys <[email protected]>
Change-Id I2fc47c89f5ec605734007ceff90321c02474aa92
--- builtin/init-db.c | 2 -- refs/files-backend.c | 4 ++++ 2 files changed, 4 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +4 -0
@@ -3158,6 +3158,10 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +// XXX adjust_shared_perm ? +/* * Create .git/refs/{heads,tags} */
Signature · 1 line
gitgitgadget
Re: [PATCH v2 2/5] create .git/refs in files-backend.c
Quoted 5 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
Subject: Re: [PATCH v2 2/5] create .git/refs infiles-backend.cThis prepares for supporting the reftable format, which creates a file in that place.
The idea is sound, I think. We want to let each backend to be responsible for creating and maintaining what is at .git/refs on the filesystem.
Quoted 4 lines
Signed-off-by Han-Wen Nienhuys <[email protected]>Change-Id: I2fc47c89f5ec605734007ceff90321c02474aa92
Do we need to keep this, which is pretty much private name for the patch that is not valid for most of the people on the list?
Quoted 23 lines
--- builtin/init-db.c | 2 -- refs/files-backend.c | 4 ++++ 2 files changed, 4 insertions(+), 2 deletions(-)builtin/init-db.c +0
-2@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);refs/files-backend.c +0
-0
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3163
Quoted 6 lines
refs/files-backend.c +3
-0@@ -3158,6 +3158,10 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +// XXX adjust_shared_perm ?
I am not sure what's there to wonder about with the question mark.
If this step is meant to be a preparation before we actually allow a different backend to be used, shouldn't the updated and prepared code behave identically in externally visible ways?
Thanks.
Re: [PATCH v2 2/5] create .git/refs in files-backend.c
Quoted 7 lines · Junio C Hamano
Quoted 4 lines
Signed-off-by Han-Wen Nienhuys <[email protected]>Change-Id: I2fc47c89f5ec605734007ceff90321c02474aa92
Do we need to keep this, which is pretty much private name for the patch that is not valid for most of the people on the list?
No, I can make do with like gitgitgadget.
Quoted 13 lines
Quoted 13 lines
builtin/init-db.c +0
-2-safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);refs/files-backend.c +0
-0@@ -3158,6 +3158,10 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ;
Quoted 5 lines
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3163
Quoted 3 lines
refs/files-backend.c +3
-0+files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +// XXX adjust_shared_perm ? I am not sure what's there to wonder about with the question mark.
I forgot why I put the XXX, but note that safe_create_dirs runs adjust_shared_perms implicitly.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 2/5] create .git/refs in files-backend.c
Quoted 6 lines · Han-Wen Nienhuys
Quoted 3 lines
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3163
Quoted 3 lines
refs/files-backend.c +3
-0+files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +// XXX adjust_shared_perm ? I forgot why I put the XXX, but note that
safe_create_dirsrunsadjust_shared_permsimplicitly.
That piece of information would make an excellent part of the log message, which is to describe why "questionable" changes are made.
"git init" is designed to be runnable in an existing repository, but safe_create_dir() would not run adjust_shared_perm() when mkdir() says EEXIST, and I think that is why the original makes a call to adjust_shared_perm() after safe_create_dir() returns.
Thanks.
PATCH v2 3/5 Document how ref iterators and symrefs interact
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Change-Id Ie3ee63c52254c000ef712986246ca28f312b4301
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 3 +++ 1 file changed, 3 insertions(+)
refs/refs-internal.h +3 -0
@@ -269,6 +269,9 @@ int refs_rename_ref_available(struct ref_store *refs,* to the next entry, ref_iterator_advance() aborts the iteration, * frees the ref_iterator, and returns ITER_ERROR. * +* Ref iterators cannot return symref targets, so symbolic refs must be +* dereferenced during the iteration. +* * The reference currently being looked at can be peeled by calling * ref_iterator_peel(). This function is often faster than peel_ref(), * so it should be preferred when iterating over references.
Signature · 1 line
gitgitgadget
Re: [PATCH v2 3/5] Document how ref iterators and symrefs interact
Quoted 2 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
Subject: Re: [PATCH v2 3/5] Document how ref iterators and symrefs interact
"Subject: refs: document how ...", perhaps?
Also isn't it more like iterators and symrefs do not interact in any unexpected way, is it? iterators while enumerating refs when they see a symref, they do not dereference and give the underlying ref the symref is pointing at---the underlying ref will be listed when it comes its turn to be shown as an ordinary ref. I am not sure what is there to single out and document...
Quoted 11 lines
Change-Id Ie3ee63c52254c000ef712986246ca28f312b4301Signed-off-by Han-Wen Nienhuys <[email protected]>--- refs/refs-internal.h | 3 +++ 1 file changed, 3 insertions(+)refs/refs-internal.h +0
-0
refs/refs-internal.h int refs_rename_ref_available(struct ref_store *refs, · line 273
Quoted 5 lines
refs/refs-internal.h +2
-0@@ -269,6 +269,9 @@ int refs_rename_ref_available(struct ref_store *refs,* to the next entry, ref_iterator_advance() aborts the iteration, * frees the ref_iterator, and returns ITER_ERROR. * +* Ref iterators cannot return symref targets, so symbolic refs must be +* dereferenced during the iteration.
What this says is not techincally incorrect. Iterators do not give each_ref_fn what underlying ref a symref is pointing at. But it also is misleading. If your callback wants to know what object each ref is pointing at do not need to do anything extra when it sees a symref, as name of the object pointed at by the underlying ref is given to it. Only callbacks that wants to know the other ref, not the value of the other ref, needs to dereference when called with a symref.
But I wonder if we need to even say this. Isn't it obvious from the each_ref_fn API that nothing other than the refname, object id, and what kind of ref it is, will be given to the user of the API, so it would be natural for a caller that wants to do extra things it needs to do for symrefs must act when it learns a ref is a symref? After all, that is why the flags word is one of the parameters given to an each_ref_fn.
Re: [PATCH v2 3/5] Document how ref iterators and symrefs interact
Quoted 6 lines · Junio C Hamano
Quoted 2 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
Subject: Re: [PATCH v2 3/5] Document how ref iterators and symrefs interact
Quoted 8 lines
"Subject: refs: document how ...", perhaps?
Also isn't it more like iterators and symrefs do not interact in any unexpected way, is it? iterators while enumerating refs when they see a symref, they do not dereference and give the underlying ref the symref is pointing at---the underlying ref will be listed when it comes its turn to be shown as an ordinary ref. I am not sure what is there to single out and document...
I mean, you can't return a zero OID and set REF_ISSYMLINK. Instead, the ref backend has to follow the symlink recursively until it finds a ref pointing to a OID.
Quoted 26 lines
Quoted 11 lines
Change-Id Ie3ee63c52254c000ef712986246ca28f312b4301Signed-off-by Han-Wen Nienhuys <[email protected]>--- refs/refs-internal.h | 3 +++ 1 file changed, 3 insertions(+)refs/refs-internal.h +0
-0refs/refs-internal.h int refs_rename_ref_available(struct ref_store *refs, · line 273
Quoted 5 lines
refs/refs-internal.h +2
-0@@ -269,6 +269,9 @@ int refs_rename_ref_available(struct ref_store *refs,* to the next entry, ref_iterator_advance() aborts the iteration, * frees the ref_iterator, and returns ITER_ERROR. * +* Ref iterators cannot return symref targets, so symbolic refs must be +* dereferenced during the iteration. What this says is not techincally incorrect. Iterators do not give
each_ref_fnwhat underlying ref a symref is pointing at. But it also is misleading. If your callback wants to know what object each ref is pointing at do not need to do anything extra when it sees a symref, as name of the object pointed at by the underlying ref is given to it. Only callbacks that wants to know the other ref, not the value of the other ref, needs to dereference when called with a symref.
Quoted 7 lines
But I wonder if we need to even say this. Isn't it obvious from the
each_ref_fnAPI that nothing other than the refname, object id, and what kind of ref it is, will be given to the user of the API, so it would be natural for a caller that wants to do extra things it needs to do for symrefs must act when it learns a ref is a symref? After all, that is why the flags word is one of the parameters given to aneach_ref_fn.
Maybe it is obvious to you, but it was not obvious to me, coming from JGit working on the RefDatabase. One could imagine that the caller needs to do something special to handle a symref, and returning a zero object ID is OK, and this is a natural assumption if you implement a ref backend, as dereferencing the symref to find out the final OID may do more work than the caller needs.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 3/5] Document how ref iterators and symrefs interact
Quoted 10 lines · Han-Wen Nienhuys
Quoted 7 lines
But I wonder if we need to even say this. Isn't it obvious from the
each_ref_fnAPI that nothing other than the refname, object id, and what kind of ref it is, will be given to the user of the API, so it would be natural for a caller that wants to do extra things it needs to do for symrefs must act when it learns a ref is a symref? After all, that is why the flags word is one of the parameters given to aneach_ref_fn.Maybe it is obvious to you, but it was not obvious to me, coming from JGit working on the RefDatabase.
If you found each_ref_fn API docs lacking, that probably is where the information should go (i.e. regardless of type of refs, the object name is filled correctly). We need to make it obvious to the developers who needs to use and work on the code with the API defined and described in refs.h
Thanks.
PATCH v2 4/5 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Reftable is a new format for storing the ref database. It provides the following benefits:
* Simple and fast atomic ref transactions, including multiple refs and reflogs.
* Compact storage of ref data.
* Fast look ups of ref data.
* Case-sensitive ref names on Windows/OSX, regardless of file system
* Eliminates file/directory conflicts in ref names
Further context and motivation can be found in background reading:
* Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
* Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
* First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
* Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
* First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
* libgit2 support issue: https://github.com/libgit2/libgit2/issues
* GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by Han-Wen Nienhuys <[email protected]>
Change-Id Id396ff42be8b42b9e11f194a32e2f95b8250c109
--- reftable/LICENSE | 31 ++ reftable/README.md | 17 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 38 ++ reftable/block.c | 401 ++++++++++++++ reftable/block.h | 71 +++ reftable/block_test.c | 151 +++++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 230 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 288 ++++++++++ reftable/merged.h | 34 ++ reftable/merged_test.c | 258 +++++++++ reftable/pq.c | 124 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 710 ++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1110 +++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/record_test.c | 332 +++++++++++ reftable/reftable.h | 399 +++++++++++++ reftable/reftable_test.c | 481 ++++++++++++++++ reftable/slice.c | 199 +++++++ reftable/slice.h | 39 ++ reftable/slice_test.c | 38 ++ reftable/stack.c | 985 ++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/stack_test.c | 281 ++++++++++ reftable/system.h | 36 ++ reftable/test_framework.c | 67 +++ reftable/test_framework.h | 64 +++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/tree_test.c | 61 ++ reftable/writer.c | 624 +++++++++++++++++++++ reftable/writer.h | 46 ++ 42 files changed, 7909 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/block_test.c create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/record_test.c create mode 100644 reftable/reftable.h create mode 100644 reftable/reftable_test.c create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/slice_test.c create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c create mode 100644 reftable/system.h create mode 100644 reftable/test_framework.c create mode 100644 reftable/test_framework.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/tree_test.c create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h
reftable/LICENSE new file +31 -0
@@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
reftable/README.md new file +17 -0
@@ -0,0 +1,17 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + echo '/* empty */' > reftable/config.h + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +5 -0
@@ -0,0 +1,5 @@ +commit c616d53b88657c3a5fe4d2e7243a48effc34c626 +Author: Han-Wen Nienhuys <[email protected]> +Date: Mon Jan 27 15:05:43 2020 +0100 + + C: ban // comments
reftable/basics.c new file +196 -0
196 lines not on this page · raw message
reftable/basics.h new file +38 -0
@@ -0,0 +1,38 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 +#define ARRAYSIZE (a) sizeof(a) / sizeof(a[0]) + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +401 -0
401 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/block_test.c new file +151 -0
151 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/config.h new file +1 -0
@@ -0,0 +1 @@ +/* empty */
reftable/constants.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +97 -0
97 lines not on this page · raw message
reftable/file.c new file +97 -0
97 lines not on this page · raw message
reftable/iter.c new file +230 -0
230 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +288 -0
288 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/merged_test.c new file +258 -0
258 lines not on this page · raw message
reftable/pq.c new file +124 -0
124 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +710 -0
710 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1110 -0
62 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/record_test.c new file +332 -0
332 lines not on this page · raw message
reftable/reftable.h new file +399 -0
399 lines not on this page · raw message
reftable/reftable_test.c new file +481 -0
481 lines not on this page · raw message
reftable/slice.c new file +199 -0
199 lines not on this page · raw message
reftable/stack.c new file +985 -0
985 lines not on this page · raw message
reftable/stack_test.c new file +281 -0
281 lines not on this page · raw message
reftable/test_framework.c new file +67 -0
67 lines not on this page · raw message
reftable/test_framework.h new file +64 -0
64 lines not on this page · raw message
reftable/tree.c new file +66 -0
66 lines not on this page · raw message
reftable/tree_test.c new file +61 -0
61 lines not on this page · raw message
reftable/writer.c new file +624 -0
624 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
PATCH v2 5/5 Reftable support for git-core
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
For background, see the previous commit introducing the library.
TODO:
* Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
Example use:
$ ~/vc/git/git init
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
$ ~/vc/git/git add a
$ ~/vc/git/git commit -mx
fatal: not a git repository (or any of the parent directories): .git
[master (root-commit) 373d969] x
1 file changed, 1 insertion(+)
create mode 100644 a
$ ~/vc/git/git show-ref
373d96972fca9b63595740bba3898a762778ba20 HEAD
373d96972fca9b63595740bba3898a762778ba20 refs/heads/master
$ ls -l .git/reftable/
total 12
-rw------- 1 hanwen primarygroup 126 Jan 23 20:08 000000000001-000000000001.ref
-rw------- 1 hanwen primarygroup 4277 Jan 23 20:08 000000000002-000000000002.ref
$ go run ~/vc/reftable/cmd/dump.go -table /tmp/qz/.git/reftable/000000000002-000000000002.ref
** DEBUG **
name /tmp/qz/.git/reftable/000000000002-000000000002.ref, sz 4209: 'r' reftable.readerOffsets{Present:true, Offset:0x0, IndexOffset:0x0}, 'o' reftable.readerOffsets{Present:false, Offset:0x0, IndexOffset:0x0} 'g' reftable.readerOffsets{Present:true, Offset:0x1000, IndexOffset:0x0}
** REFS **
reftable.RefRecord{RefName:"refs/heads/master", UpdateIndex:0x2, Value:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, TargetValue:[]uint8(nil), Target:""}
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}Change-Id I225ee6317b7911edf9aa95f43299f6c7c4511914
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- Makefile | 23 +- refs.c | 18 +- refs.h | 2 + refs/refs-internal.h | 1 + refs/reftable-backend.c | 812 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 852 insertions(+), 4 deletions(-) create mode 100644 refs/reftable-backend.c
Makefile +22 -1
@@ -813,6 +813,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += command-list.h
@@ -958,6 +959,7 @@ LIB_OBJS += rebase-interactive.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
@@ -1162,7 +1164,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION)
@@ -2343,11 +2345,27 @@ VCSSVN_OBJS += vcs-svn/fast_export.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL
@@ -2484,6 +2502,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
refs.c +15 -3
@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1839,10 +1839,22 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,static struct ref_store *ref_store_init (const char *gitdir,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct strbuf refs_path =STRBUF_INIT ; + +/* XXX this should probably come from a git config setting and not +default to reftable. */ +const char *be_name ="reftable" ; +struct ref_storage_be *be;struct ref_store *refs; +strbuf_addstr (&refs_path, gitdir); +strbuf_addstr (&refs_path,"/refs" ); + +if (!is_directory (refs_path.buf)) { + be_name ="reftable" ; + } +strbuf_release (&refs_path); + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
refs.h +2 -0
@@ -456,6 +456,8 @@ int refs_for_each_reflog_ent_reverse(struct ref_store *refs,const char *refname, each_reflog_ent_fn fn, void *cb_data); + +/* XXX which ordering are these? Newest or oldest first? */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
refs/refs-internal.h +1 -0
@@ -659,6 +659,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +812 -0
812 lines not on this page · raw message
Signature · 1 line
gitgitgadget
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 1 line · Han-Wen Nienhuys via GitGitGadget
[...]
I'll try to add my 2 cents to all of the XXX spots you flagged.
refs.c static struct ref_store *lookup_ref_store_map(struct hashmap *map, · line 1846
Quoted 11 lines
refs.c +5
-2@@ -1839,10 +1839,22 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,static struct ref_store *ref_store_init (const char *gitdir,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct strbuf refs_path =STRBUF_INIT ; + +/* XXX this should probably come from a git config setting and not +default to reftable. */ +const char *be_name ="reftable" ;
I think the scheme here needs to be something like:
- "struct repository" gets a new "ref_format" field
- setup.c:check_repo_format() learns about an extensions.refFormat
config key, which we use to set repo->ref_format
- init/clone should take a command-line option for the ref format of the new repository. Anybody choosing reftables would want to set core.repositoryformatversion to "1" and set the extensions.refFormat key.
- likewise, there should be a config setting to choose the default ref format. It would obviously stay at "files" for now, but we'd eventually perhaps flip the default to "reftable".
Some thoughts on compatibility:
The idea of the config changes is that older versions of Git will either complain about repositoryformatversion (if very old), or will complain that they don't know about extensions.refFormat. But the changes you made in patch 1 indicate that existing versions of Git won't consider it to be a Git repository at all!
I think this is slightly non-ideal, in that we'd continue walking up the tree looking for a Git repo. And either:
- we'd find one, which would be confusing and possibly wrong
- we wouldn't, in which case the error would be something like "no git repository", and not "your git repository is too new"
So it would be really nice if we continued to have a HEAD file (just with some sentinel value in it) and a refs/ directory, so that existing versions of Git realize "there's a repository here, but it's too new for me".
There's a slight downside in that tools which _aren't_ careful about repositoryformatversion might try to operate on the repository, writing into refs/ or whatever. But such tools are broken, and I'm not sure we should be catering to them (besides which, the packed-refs ship sailed long ago, so they're already potentially dangerous).
refs.h int refs_for_each_reflog_ent_reverse(struct ref_store *refs, · line 462
Quoted 3 lines
refs.h +1
-0+/* XXX which ordering are these? Newest or oldest first? */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
The top one is chronological order (i.e., reading the file from start to finish), and the latter is reverse-chronological (seeking backwards in the file).
Quoted 7 lines
refs/reftable-backend.c new file +7
-0+static struct ref_iterator * +reftable_ref_iterator_begin (struct ref_store *ref_store,const char *prefix, +unsigned int flags) +{ +struct reftable_ref_store *refs = + (struct reftable_ref_store *)ref_store; +struct reftable_iterator *ri =xcalloc (1 ,sizeof (*ri));
refs/reftable-backend.c line 187
Quoted 10 lines
refs/reftable-backend.c new file +10
-0+struct merged_table *mt =NULL ; +int err =0 ; +if (refs->err) { +/* how to propagate errors? */ +return NULL ; + } + + mt =stack_merged_table (refs->stack); + +/* XXX something with flags? */
I think the flags here are DO_FOR_EACH_*. You might need to consider INCLUDE_BROKEN here (or it might be dealt with at a layer above).
There's also PER_WORKTREE_ONLY; I think you'd need to check ref_type() there, like the files backend does.
refs/reftable-backend.c line 189
Quoted 2 lines
refs/reftable-backend.c new file +2
-0+ err =merged_table_seek_ref (mt, &ri->iter, prefix); +/* XXX what to do with err? */
Hmm, I don't think you can return an error from iterator_begin(). It would probably be OK to record the error in your local struct, and then later return ITER_ERROR from iterator_advance().
I notice that the more-recent dir_iterator_begin() returns NULL in case of an error, but looking at the callers of the ref iterators, they don't seem to be ready to handle NULL.
refs/reftable-backend.c line 247
Quoted 10 lines
refs/reftable-backend.c new file +10
-0+static int write_transaction_table (struct writer *writer,void *arg) +{ +struct ref_transaction *transaction = (struct ref_transaction *)arg; +struct reftable_ref_store *refs = + (struct reftable_ref_store *)transaction->ref_store; +uint64_t ts =stack_next_update_index (refs->stack); +int err =0 ; +/* XXX - are we allowed to mutate the input data? */ +qsort (transaction->updates, transaction->nr, +sizeof (struct ref_update *), ref_update_cmp);
I don't offhand know of anything that would break, but it's probably a bad idea to do so. If you need a sorted view, can you make an auxiliary array-of-pointers? Also, the QSORT() macro is a little shorter and has some extra checks.
refs/reftable-backend.c line 269
Quoted 9 lines
refs/reftable-backend.c new file +9
-0+for (int i =0 ; i < transaction->nr; i++) { +struct ref_update *u = transaction->updates[i]; +if (u->flags &REF_HAVE_NEW ) { +struct object_id out_oid = {}; +int out_flags =0 ; +/* XXX who owns the memory here? */ +const char *resolved =refs_resolve_ref_unsafe ( + transaction->ref_store, u->refname,0 , &out_oid, + &out_flags);
In the "unsafe" version, the memory belongs to a static buffer inside the function. You shouldn't need to free it.
refs/reftable-backend.c line 563
Quoted 5 lines
refs/reftable-backend.c new file +4
-0+static int +reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator) + /* XXX const? */ + memcpy(&ri->oid.hash, ri->log.new_hash, GIT_SHA1_RAWSZ);
You'd want probably want to use hashcpy() here (or better yet, use "struct object_id" in ri->log, too, and then use oidcpy()).
But that raises a question: how ready are reftables to handle non-sha1 object ids? I see a lot of GIT_SHA1_RAWSZ, and I think the on-disk format actually has binary sha1s, right? In theory if those all become the_hash_algo->rawsz, then it might "Just Work" to read and write slightly larger entries.
refs/reftable-backend.c line 644
Quoted 10 lines
refs/reftable-backend.c new file +9
-0+reftable_for_each_reflog_ent_newest_first(struct ref_store *ref_store, + const char *refname, + each_reflog_ent_fn fn, void *cb_data) + /* XXX committer = email? name? */ + if (fn(&old_oid, &new_oid, log.name, log.time, + log.tz_offset, log.message, cb_data)) { + err =-1 ; +break ; + }
The committer entry we pass back to each_reflog_ent_fn should be the full "Human Name <email@host>".
Quoted 12 lines
refs/reftable-backend.c new file +12
-0+static int reftable_reflog_expire (struct ref_store *ref_store, +const char *refname, +const struct object_id *oid, +unsigned int flags, +reflog_expiry_prepare_fn prepare_fn, +reflog_expiry_should_prune_fn should_prune_fn, +reflog_expiry_cleanup_fn cleanup_fn, +void *policy_cb_data) +{ +/* +XXX +
refs/reftable-backend.c line 760
Quoted 10 lines
refs/reftable-backend.c new file +10
-0+This doesn't fit with the reftable API. If we are expiring for space +reasons, the expiry should be combined with a compaction, and there +should be a policy that can be called for all refnames, not for a +single ref name. + +If this is for cleaning up individual entries, we'll have to write +extra data to create tombstones. +*/ +return 0 ; +}
I agree that we'd generally want to expire and compact all entries at once. Unfortunately I think this per-ref interface is exposed by git-reflog. I.e., you can do "git reflog expire refs/heads/foo".
Could we add an extra API call for "expire everything", and then:
- have refs/files-backend.c implement that by just iterating over all of the refs and pruning them one by one
- have builtin/reflog.c trigger the "expire everything" API when it
sees "--all"
- teach refs/reftables-backend.c a crappy version of the per-ref expiration, where it just inserts tombstones but doesn't do any compaction. It doesn't have to be fast or produce a good output; it's just for compatibility.
Alternatively, I'd be OK with die("sorry, reftables doesn't support per-ref reflog expiration") as a first step.
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 21 lines · MST Jeff King
Some thoughts on compatibility:
The idea of the config changes is that older versions of Git will either complain about repositoryformatversion (if very old), or will complain that they don't know about extensions.refFormat. But the changes you made in patch 1 indicate that existing versions of Git won't consider it to be a Git repository at all!
I think this is slightly non-ideal, in that we'd continue walking up the tree looking for a Git repo. And either:
- we'd find one, which would be confusing and possibly wrong
- we wouldn't, in which case the error would be something like "no git repository", and not "your git repository is too new"
So it would be really nice if we continued to have a HEAD file (just with some sentinel value in it) and a refs/ directory, so that existing versions of Git realize "there's a repository here, but it's too new for me".
Quoted 5 lines
There's a slight downside in that tools which _aren't_ careful about repositoryformatversion might try to operate on the repository, writing into refs/ or whatever. But such tools are broken, and I'm not sure we should be catering to them (besides which, the packed-refs ship sailed long ago, so they're already potentially dangerous).
Could you elaborate on this a bit because it seems on the surface that these tools aren't very dangerous, and therefore likely many of them exist?
What are the dangers today of tools that understand/operate on loose and packed refs without reading repositoryformatversion?
-Martin
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 11 lines · Martin Fick
Quoted 5 lines
There's a slight downside in that tools which _aren't_ careful about repositoryformatversion might try to operate on the repository, writing into refs/ or whatever. But such tools are broken, and I'm not sure we should be catering to them (besides which, the packed-refs ship sailed long ago, so they're already potentially dangerous).
Could you elaborate on this a bit because it seems on the surface that these tools aren't very dangerous, and therefore likely many of them exist?
What are the dangers today of tools that understand/operate on loose and packed refs without reading repositoryformatversion?
I was mostly thinking of hacky scripts that tried to touch .git/refs directly. And there are a few levels of dangerous there:
- if you're doing "echo $sha1 >.git/refs/heads/master", then you're not locking properly. But it would probably work most of the time.
- if you're properly taking a lock on ".git/refs/heads/master.lock" and renaming into place but not looking at packed-refs, then you might overwrite somebody else's update which is in the packed file
- if you're trying to read refs and not reading packed-refs, obviously you might miss some values
If you're actually doing the correct locking and packed-refs read (which "real" implementations like libgit2 do) then no, I don't think that's dangerous. And I think libgit2 properly complains when it sees a repositoryformatversion above 0. I don't know offhand about JGit, or any of the lesser-used ones like dulwich or go-git.
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 24 lines · MST Jeff King
Quoted 6 lines · Martin Fick
Quoted 5 lines
There's a slight downside in that tools which _aren't_ careful about repositoryformatversion might try to operate on the repository, writing into refs/ or whatever. But such tools are broken, and I'm not sure we should be catering to them (besides which, the packed-refs ship sailed long ago, so they're already potentially dangerous).
Quoted 6 lines
Could you elaborate on this a bit because it seems on the surface that these tools aren't very dangerous, and therefore likely many of them exist?
What are the dangers today of tools that understand/operate on loose and packed refs without reading repositoryformatversion?
I was mostly thinking of hacky scripts that tried to touch .git/refs directly. And there are a few levels of dangerous there:
- if you're doing "echo $sha1 >.git/refs/heads/master", then you're not locking properly. But it would probably work most of the time.
- if you're properly taking a lock on ".git/refs/heads/master.lock" and renaming into place but not looking at packed-refs, then you might overwrite somebody else's update which is in the packed file
Quoted 8 lines
- if you're trying to read refs and not reading packed-refs, obviously you might miss some values
If you're actually doing the correct locking and packed-refs read (which "real" implementations like libgit2 do) then no, I don't think that's dangerous. And I think libgit2 properly complains when it sees a repositoryformatversion above 0. I don't know offhand about JGit, or any of the lesser-used ones like dulwich or go-git.
Today, some of these sound like shortcuts that are very likely taken quite a bit by cleanup and other maintenance scripts (not necessarily formal git tools), and the impact of these shortcuts is likely low with the current model. However, I suspect these tools/scripts could be seriously disruptive if we leave the refs dir around when using reftable,
-Martin
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 11 lines · Martin Fick
Quoted 5 lines
If you're actually doing the correct locking and packed-refs read (which "real" implementations like libgit2 do) then no, I don't think that's dangerous. And I think libgit2 properly complains when it sees a repositoryformatversion above 0. I don't know offhand about JGit, or any of the lesser-used ones like dulwich or go-git.
Today, some of these sound like shortcuts that are very likely taken quite a bit by cleanup and other maintenance scripts (not necessarily formal git tools), and the impact of these shortcuts is likely low with the current model. However, I suspect these
tools/scriptscould be seriously disruptive if we leave the refs dir around when using reftable,
Maybe we can leave the refs dir, but have no heads/ directory inside, and make the whole thing read-only?
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 7 lines · MST Han-Wen Nienhuys
Quoted 6 lines · Martin Fick
Quoted 5 lines
If you're actually doing the correct locking and packed-refs read (which "real" implementations like libgit2 do) then no, I don't think that's dangerous. And I think libgit2 properly complains when it sees a repositoryformatversion above 0. I don't know offhand about JGit, or any of the lesser-used ones like dulwich or go-git.
Quoted 8 lines
Quoted 5 lines
Today, some of these sound like shortcuts that are very likely taken quite a bit by cleanup and other maintenance scripts (not necessarily formal git tools), and the impact of these shortcuts is likely low with the current model. However, I suspect these
tools/scriptscould be seriously disruptive if we leave the refs dir around when using reftable,Maybe we can leave the refs dir, but have no heads/ directory inside, and make the whole thing read-only?
That might be a good enough safety. I guess the next question would be, would it be OK for reftable to ignore and entries under the refs/ dir if they happen to appear there somehow?
-Martin
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 8 lines · Martin Fick
Quoted 6 lines · MST Han-Wen Nienhuys
Quoted 5 lines
Today, some of these sound like shortcuts that are very likely taken quite a bit by cleanup and other maintenance scripts (not necessarily formal git tools), and the impact of these shortcuts is likely low with the current model. However, I suspect these
tools/scriptscould be seriously disruptive if we leave the refs dir around when using reftable,
Quoted 6 lines
Quoted 2 lines
Maybe we can leave the refs dir, but have no heads/ directory inside, and make the whole thing read-only?
That might be a good enough safety. I guess the next question would be, would it be OK for reftable to ignore and entries under the refs/ dir if they happen to appear there somehow?
I propose to ignore refs/ if it is read-only, and fail if it is r/w. We're not going to look over the files under refs/ . If people actively try to shoot themselves in the foot, why would we stop them?
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 7 lines · Han-Wen Nienhuys
Quoted 3 lines
That might be a good enough safety. I guess the next question would be, would it be OK for reftable to ignore and entries under the refs/ dir if they happen to appear there somehow?
I propose to ignore refs/ if it is read-only, and fail if it is r/w. We're not going to look over the files under refs/ . If people actively try to shoot themselves in the foot, why would we stop them?
I'm worried that playing games with permissions is going to lead to confusing outcomes. There are reasons one might want a r/o refs/ directory with the current system (e.g., you could have a repository on a read-only mount). Or you might have a system which doesn't implement the full POSIX permissions, and everything appears to be r/w by the user.
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 11 lines · Jeff King
Quoted 7 lines · Han-Wen Nienhuys
Quoted 3 lines
That might be a good enough safety. I guess the next question would be, would it be OK for reftable to ignore and entries under the refs/ dir if they happen to appear there somehow?
I propose to ignore refs/ if it is read-only, and fail if it is r/w. We're not going to look over the files under refs/ . If people actively try to shoot themselves in the foot, why would we stop them?
Quoted 6 lines
I'm worried that playing games with permissions is going to lead to confusing outcomes. There are reasons one might want a r/o refs/ directory with the current system (e.g., you could have a repository on a read-only mount). Or you might have a system which doesn't implement the full POSIX permissions, and everything appears to be r/w by the user.
OK, so permissions are out. How about:
HEAD - convincing enough for old versions to accept refs/ - a standard rwx directory
reftable/ - a normal directory
reftable-list - the list of tables
reads/heads - a file containing "this repo uses reftables. Upgrade
to git vXYZ"
this would prevent people from erroneously creating normal branches.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 10 lines · Han-Wen Nienhuys
OK, so permissions are out. How about:
HEAD - convincing enough for old versions to accept refs/ - a standard rwx directory
reftable/ - a normal directory
reftable-list - the list of tables
reads/heads - a file containing "this repo uses reftables. Upgrade
to git vXYZ"this would prevent people from erroneously creating normal branches.
That seems reasonable. They could still write "refs/tags/foo", etc, but presumably branches would be more common.
Trying to write to any ref there (including HEAD, assuming it has a dummy value in refs/heads/) would yield something like:
fatal: cannot lock ref 'HEAD': unable to create lock file .git/refs/heads/master.lock; non-directory in the way
Unfortunately reading with an old version would quietly return "no such ref", at least with old versions of Git (but those maybe aren't as interesting as alternate implementations, since old versions of Git would also complain about the extensions marked in the config).
Iterating with "for-each-ref" would say something like:
$ git for-each-ref warning: ignoring broken ref refs/heads $ echo $? 0
which is so-so. Replacing "refs/" with a file with an executable bit (which is still playing with permissions, but I think in a little bit less exotic way) behaves similarly, except that for-each-ref does not even produce a warning. :-/
I dunno. I wish the failure mode were a little more complete. OTOH, I'm really not that worried about it. The config repo-version stuff is _supposed_ to be the definitive word on whether Git is willing to open up a repository. So this is really belt-and-suspenders on top of that.
-Peff
PS I don't know if it's set in stone yet, but if we're changing things
around, is it an option to put reftable-list inside the reftable directory, just to keep the top-level uncluttered?
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 3 lines · Jeff King
PS I don't know if it's set in stone yet, but if we're changing things
around, is it an option to put reftable-list inside the reftable directory, just to keep the top-level uncluttered?
If we're moving things around, we can add this little difference too. The original idea was to create a conflict with the old-style storage, but putting the list in reftable/ actually makes files -> restable conversion slightly more atomic.
--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays. --
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Jeff KingQuoted 9 lines · Jeff King
Quoted 8 lines
HEAD - convincing enough for old versions to accept refs/ - a standard rwx directory
reftable/ - a normal directory
reftable-list - the list of tables
reads/heads - a file containing "this repo uses reftables. Upgrade
to git vXYZ"this would prevent people from erroneously creating normal branches.
Quoted 7 lines
That seems reasonable. They could still write "refs/tags/foo", etc, but presumably branches would be more common.
..PS I don't know if it's set in stone yet, but if we're changing things
around, is it an option to put reftable-list inside the reftable directory, just to keep the top-level uncluttered?
I put up https://git.eclipse.org/r/c/157167/ to update the spec inside JGit. Please review.
Signature · 7 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 6 lines · Han-Wen Nienhuys
Quoted 3 lines
PS I don't know if it's set in stone yet, but if we're changing things
around, is it an option to put reftable-list inside the reftable directory, just to keep the top-level uncluttered?I put up https://git.eclipse.org/r/c/157167/ to update the spec inside JGit. Please review.
That looks quite reasonable. The one change I'd make is to put "refs/heads/.invalid" into HEAD (instead of "refs/.invalid"). In my experiments, existing Git is happy to write into an invalid refname via "git update-ref HEAD". But by putting it past the non-directory "heads", such a write would always fail.
It's also supposed to be a requirement that HEAD only puts to refs/heads/, though our validate_headref() does not enforce that. I don't know if any other implementations might, though.
I did compare earlier your "make refs/heads a file" versus "make refs/ a file and just give it an executable bit". And I think the "refs/heads" solution does behave slightly better in a few cases. But if I understand correctly, just giving "refs/" an executable bit would mean we retain compatibility with the existing JGit implementation.
It does end up playing games with permissions, which I warned against, but I think it might be tolerable:
1. Unlike read vs write, there's not generally a reason users would need to separate read and execute.
2. Portability-wise, POSIX permissions give us what we need, and non-POSIX systems tend to just say everything is executable.
So I could also live with that direction, if switching the JGit behavior at this point is too much of a pain.
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 7 lines · Jeff King
Quoted 4 lines · Han-Wen Nienhuys
Quoted 3 lines
PS I don't know if it's set in stone yet, but if we're changing things
around, is it an option to put reftable-list inside the reftable directory, just to keep the top-level uncluttered?
Quoted 8 lines
Quoted 2 lines
I put up https://git.eclipse.org/r/c/157167/ to update the spec inside JGit. Please review.
That looks quite reasonable. The one change I'd make is to put "refs/heads/.invalid" into HEAD (instead of "refs/.invalid"). In my experiments, existing Git is happy to write into an invalid refname via "git update-ref HEAD". But by putting it past the non-directory "heads", such a write would always fail.
thanks, incorporated.
Quoted 9 lines
It's also supposed to be a requirement that HEAD only puts to refs/heads/, though our
validate_headref()does not enforce that. I don't know if any other implementations might, though.I did compare earlier your "make refs/heads a file" versus "make refs/ a file and just give it an executable bit". And I think the "refs/heads" solution does behave slightly better in a few cases. But if I understand correctly, just giving "refs/" an executable bit would mean we retain compatibility with the existing JGit implementation.
I hadn't looked at the JGit discovery in detail, but it looks like the currently proposed approach works with JGit. (I haven't adapted JGit for the new path setup yet.)
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Jeff KingQuoted 6 lines · Jeff King
I was mostly thinking of hacky scripts that tried to touch .git/refs directly. And there are a few levels of dangerous there:
- if you're doing "echo $sha1 >.git/refs/heads/master", then you're not locking properly. But it would probably work most of the time.
Quoted 6 lines
- if you're properly taking a lock on ".git/refs/heads/master.lock" and renaming into place but not looking at packed-refs, then you might overwrite somebody else's update which is in the packed file
- if you're trying to read refs and not reading packed-refs, obviously you might miss some values
... but the values you read are the right ones as loose refs overlay and trump what is in packed-refs.
- if you're trying to delete a branch by removing a file under refs/, that would uncover the corresponding entry in packed-refs, if exists---in other words, you may resurrect an older value of a branch instead of deleting it.
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Jeff KingQuoted 7 lines · Jeff King
Quoted 1 line · Han-Wen Nienhuys via GitGitGadget
[...]
I'll try to add my 2 cents to all of the XXX spots you flagged.
thanks! that was really helpful.
Quoted 14 lines
Some thoughts on compatibility:
The idea of the config changes is that older versions of Git will either complain about repositoryformatversion (if very old), or will complain that they don't know about extensions.refFormat. But the changes you made in patch 1 indicate that existing versions of Git won't consider it to be a Git repository at all!
I think this is slightly non-ideal, in that we'd continue walking up the tree looking for a Git repo. And either:
- we'd find one, which would be confusing and possibly wrong
Quoted 7 lines
- we wouldn't, in which case the error would be something like "no git repository", and not "your git repository is too new"
So it would be really nice if we continued to have a HEAD file (just with some sentinel value in it) and a refs/ directory, so that existing versions of Git realize "there's a repository here, but it's too new for me".
You have good points.
JGit currently implements what we have here, as this is what's spelled out in the spec that Shawn posted back in the day. It's probably acceptable to this, though, as the reftable support has only landed in JGit very recently and will probably appear very experimental to folks.
How would the layout be then? We'll have
HEAD - dummy file
reftable/ - the tables
refs/ - dummy dir
where shall we store the reftable list? maybe in a file called
reftable-list
If we have both HEAD/refs + (refable/reftable-list), what should we put there to ensure that no git version actually manages to use the repository? (what happens if someone deletes the version setting from the .git/config file)
Quoted 7 lines
Quoted 3 lines
+/* XXX which ordering are these? Newest or oldest first? */
intfor_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data); intfor_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data);The top one is chronological order (i.e., reading the file from start to finish), and the latter is reverse-chronological (seeking backwards in the file).
thanks, I put in a commit to clarify this.
Quoted 6 lines
refs/reftable-backend.c line 189
Quoted 2 lines
refs/reftable-backend.c new file +2
-0+ err =merged_table_seek_ref (mt, &ri->iter, prefix); +/* XXX what to do with err? */ Hmm, I don't think you can return an error from
iterator_begin(). It would probably be OK to record the error in your local struct, and then later returnITER_ERRORfromiterator_advance().
good point. I did that.
Quoted 8 lines
refs/reftable-backend.c line 247
Quoted 3 lines
refs/reftable-backend.c new file +3
-0+/* XXX - are we allowed to mutate the input data? */ +qsort (transaction->updates, transaction->nr, +sizeof (struct ref_update *), ref_update_cmp);I don't offhand know of anything that would break, but it's probably a bad idea to do so. If you need a sorted view, can you make an auxiliary array-of-pointers? Also, the
QSORT()macro is a little shorter and has some extra checks.
Done.
Quoted 2 lines
In the "unsafe" version, the memory belongs to a static buffer inside the function. You shouldn't need to free it.
OK.
Quoted 8 lines
refs/reftable-backend.c line 563
Quoted 5 lines
refs/reftable-backend.c new file +4
-0+static int +reftable_reflog_ref_iterator_advance(struct ref_iterator *ref_iterator) + /* XXX const? */ + memcpy(&ri->oid.hash, ri->log.new_hash, GIT_SHA1_RAWSZ);You'd want probably want to use
hashcpy()here (or better yet, use "structobject_id" inri->log, too, and then useoidcpy()).
It's more practical in the library to use pointers rather than fixed size arrays, so it'll be hashcpy.
Quoted 5 lines
But that raises a question: how ready are reftables to handle non-sha1 object ids? I see a lot of
GIT_SHA1_RAWSZ, and I think the on-disk format actually has binary sha1s, right? In theory if those all becomethe_hash_algo->rawsz, then it might "Just Work" to read and write slightly larger entries.
The format fixes the reftable at 20 bytes, and there is not enough framing information to just write more data. We'll have to encode the hash size in the version number somehow, eg. we could use the higher order bit of the version byte to encode it, for example.
But it needs a new version of the spec. I think it's premature to do this while v1 of reftable isn't in git-core yet.
Quoted 2 lines
The committer entry we pass back to
each_reflog_ent_fnshould be the full "Human Name <email@host>".
thx, done.
Quoted 5 lines
I agree that we'd generally want to expire and compact all entries at once. Unfortunately I think this per-ref interface is exposed by git-reflog. I.e., you can do "git reflog expire refs/heads/foo".
Could we add an extra API call for "expire everything", and then:
I took note of your remarks, and added a BUG() for now.
I'll focus on getting the CI on gitgitgadget to build this code first.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 12 lines · Han-Wen Nienhuys
JGit currently implements what we have here, as this is what's spelled out in the spec that Shawn posted back in the day. It's probably acceptable to this, though, as the reftable support has only landed in JGit very recently and will probably appear very experimental to folks.
How would the layout be then? We'll have
HEAD - dummy file
reftable/ - the tables
refs/ - dummy dir
Quoted 8 lines
where shall we store the reftable list? maybe in a file called
reftable-list
If we have both HEAD/refs + (refable/reftable-list), what should we put there to ensure that no git version actually manages to use the repository? (what happens if someone deletes the version setting from the .git/config file)
Yeah, it would be nice to have something that an older version of Git would totally choke on, but I'm not sure we have a lot of leeway. What we put in HEAD has to be syntactically legitimate enough to appease validate_headref(), so our options are either "ref:
refs/something/bogus" or an object hash that we don't have (e.g., 0{40}). The former would be preferable because it would (in theory) prevent us from writing to HEAD, as well.
I wondered what would happen if you put in a syntactically invalid ref, like "ref: refs/.not/.valid" (leading dots are not allowed in path components of refnames). It does cause _some_ parts of Git to choke, but sadly "git update-ref HEAD $sha1" actually writes to .git/refs/.not/.valid.
Even "refs/../../dangerous" doesn't give it pause. Yikes. It seems we're pretty willing to accept symref destinations without further checking.
Making "refs" a file instead of a directory does work nicely, as any attempts to read or write would get ENOTDIR. And we can fool is_git_directory() as long as it's marked executable. That's OK on POSIX systems, but I'm not sure how it would work on Windows (or maybe it would work just fine, since we presumably just say "yep, everything is executable").
So perhaps that's enough, and what we put in HEAD won't matter (since nobody will be able to write into refs/ anyway).
Quoted 6 lines
Quoted 5 lines
But that raises a question: how ready are reftables to handle non-sha1 object ids? I see a lot of
GIT_SHA1_RAWSZ, and I think the on-disk format actually has binary sha1s, right? In theory if those all becomethe_hash_algo->rawsz, then it might "Just Work" to read and write slightly larger entries.
Quoted 7 lines
The format fixes the reftable at 20 bytes, and there is not enough framing information to just write more data. We'll have to encode the hash size in the version number somehow, eg. we could use the higher order bit of the version byte to encode it, for example.
But it needs a new version of the spec. I think it's premature to do this while v1 of reftable isn't in git-core yet.
I don't know that we technically need the reftables file to say how long the hashes are. The git config will tell us which hash we're using, and everything else is supposed to follow. So I think it would work OK as long as you're able to be told by the rest of Git that hashes are N bytes, and just use that to compute the fixed-size records.
That said, it might make for easier debugging if the reftables file declares the size it assumes.
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 9 lines · Jeff King
Making "refs" a file instead of a directory does work nicely, as any attempts to read or write would get ENOTDIR. And we can fool
is_git_directory()as long as it's marked executable. That's OK on POSIX systems, but I'm not sure how it would work on Windows (or maybe it would work just fine, since we presumably just say "yep, everything is executable").So perhaps that's enough, and what we put in HEAD won't matter (since nobody will be able to write into refs/ anyway).
I wonder if it would help to take the "looser repository detection" code alone and have it in a release, way before the rest of the reftable topic is ready. Then by the time a repository created by a reftable-enabled Git appears on people's disks, all the older versions of Git that are still in people's hands would at least know that it is a repository supported by future Git that they themselves do not know how to handle, stop repository discovery correctly and refrain from damaging the repository with an extension unknown to them?
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 13 lines · Junio C Hamano
Quoted 9 lines · Jeff King
Making "refs" a file instead of a directory does work nicely, as any attempts to read or write would get ENOTDIR. And we can fool
is_git_directory()as long as it's marked executable. That's OK on POSIX systems, but I'm not sure how it would work on Windows (or maybe it would work just fine, since we presumably just say "yep, everything is executable").So perhaps that's enough, and what we put in HEAD won't matter (since nobody will be able to write into refs/ anyway).
Quoted 9 lines
I wonder if it would help to take the "looser repository detection" code alone and have it in a release, way before the rest of the reftable topic is ready. Then by the time a repository created by a reftable-enabled Git appears on people's disks, all the older versions of Git that are still in people's hands would at least know that it is a repository supported by future Git that they themselves do not know how to handle, stop repository discovery correctly and refrain from damaging the repository with an extension unknown to them?
Sure. I suspect that the reftable topic will take some time to get fully ready.
However, it would be good to have a definite plan for the layout now, because JGit (if you tell it to) is already creating repositories with the layout we're moving away from.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Junio C HamanoQuoted 12 lines · Junio C Hamano
Quoted 9 lines · Jeff King
Making "refs" a file instead of a directory does work nicely, as any attempts to read or write would get ENOTDIR. And we can fool
is_git_directory()as long as it's marked executable. That's OK on POSIX systems, but I'm not sure how it would work on Windows (or maybe it would work just fine, since we presumably just say "yep, everything is executable").So perhaps that's enough, and what we put in HEAD won't matter (since nobody will be able to write into refs/ anyway).
Quoted 9 lines
I wonder if it would help to take the "looser repository detection" code alone and have it in a release, way before the rest of the reftable topic is ready. Then by the time a repository created by a reftable-enabled Git appears on people's disks, all the older versions of Git that are still in people's hands would at least know that it is a repository supported by future Git that they themselves do not know how to handle, stop repository discovery correctly and refrain from damaging the repository with an extension unknown to them?
That helps, but it just slightly expands the window where we do the right thing. I.e., in a rollout like this:
1. Status quo: we only consider it a repo if "refs/" is executable
2. We introduce a version of Git that can read and write reftables.
3. We flip reftables support on by default.
Then any version of git after step 2 is fine. It's the ones from step 1 and older we care about. And there's probably some time between steps 2 and 3 to let the new versions percolate, shake out bugs, etc.
Adding a new step 1.5, "looser repository detection" expands the window of "good" versions by however long it takes to go from step 1.5 to step
2. But I expect that to be much smaller than the window between 2 and 3, so I'm not sure it makes a meaningful impact.
(Of course people who voluntarily turn on the feature as soon as they get hold of a step 2 version have no window at all, but presumably early adopters are OK with the risk).
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Jeff KingQuoted 2 lines · Jeff King
That said, it might make for easier debugging if the reftables file declares the size it assumes.
If there is a mismatch in size, the reftable will look completely corrupted data. I think this will be a bad experience.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 7 lines · Han-Wen Nienhuys
Quoted 2 lines · Jeff King
That said, it might make for easier debugging if the reftables file declares the size it assumes.
If there is a mismatch in size, the reftable will look completely corrupted data. I think this will be a bad experience.
Yes, but I think it would take a bunch of other failures to get there. And it's true of all of the other parts of Git, too; the master switch is a config setting that says "I'm a sha-256 repository".
That said, I'm not at all opposed to a header in the reftables file saying "I'm a sha-256 reftable". We shouldn't ever see a mismatch there, but that's what I meant by "easier debugging".
-Peff
Re: [PATCH v2 5/5] Reftable support for git-core
In reply to Jeff KingQuoted 14 lines · Jeff King
refs.c static struct ref_store *lookup_ref_store_map(struct hashmap *map, · line 1846
Quoted 9 lines
{
refs.c +5
-2-const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct strbuf refs_path =STRBUF_INIT ; + +/* XXX this should probably come from a git config setting and not +default to reftable. */ +const char *be_name ="reftable" ;I think the scheme here needs to be something like:
- "struct repository" gets a new "
ref_format" field
Quoted 7 lines
- setup.c:check_repo_format() learns about an extensions.refFormat
config key, which we use to set repo->ref_format- init/clone should take a command-line option for the ref format of the new repository. Anybody choosing reftables would want to set core.repositoryformatversion to "1" and set the extensions.refFormat key.
I did this, but are you sure this works? Where would the repo->ref_storage_format get set? I tried adding it to repo_init(), but this doesn't get called in a normal startup sequence.
Breakpoint 3, ref_store_init (gitdir=0x555555884b70 ".git", be_name=0x5555557942ca "reftable", flags=15) at refs.c:1841
warning: Source file is more recent than executable. 1841 {
(gdb) up
#1 0x00005555556de2c8 in get_main_ref_store (r=0x555555871d40 <the_repo>) at refs.c:1862
(gdb) p r->ref_storage_format
$1 = 0x0
}
I'm sending the revised patch series now.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v2 5/5] Reftable support for git-core
Quoted 9 lines · Han-Wen Nienhuys
Quoted 4 lines
- init/clone should take a command-line option for the ref format of the new repository. Anybody choosing reftables would want to set core.repositoryformatversion to "1" and set the extensions.refFormat key.
I did this, but are you sure this works? Where would the
repo->ref_storage_formatget set? I tried adding it torepo_init(), but this doesn't get called in a normal startup sequence.
Quoted 10 lines
Breakpoint 3,
ref_store_init(gitdir=0x555555884b70 ".git", be_name=0x5555557942ca "reftable", flags=15) at refs.c:1841
warning: Source file is more recent than executable. 1841 {
(gdb) up
#10x00005555556de2c8inget_main_ref_store(r=0x555555871d40 <the_repo>) at refs.c:1862
(gdb) pr->ref_storage_format
$1 =0x0
}
Hmm. Looks like repo_init() is only used for submodules. Which is pretty horrid, because that setup sequence is basically duplicated between there and setup_git_directory(), which is the traditional mechanism. That's something that ought to be cleaned up, but it's definitely outside the scope of your series.
The patch below was enough to make something like this work:
git init --reftable repo cd repo git commit --allow-empty -m foo git for-each-ref
I had to make some tweaks to the init code, too:
- setting the_repository->ref_storage_format based on the --reftable flag
- we can still recognize a reinit by the presence of a HEAD file (whether it's a real one or a dummy). And we should call create_symref() in either case, either to create HEAD or to create the HEAD symlink inside reftables
I also fixed a bug in your first patch, where the creation of refs/ moves into files_init_db(). The strbuf needs reset there, or we just append to it, and end up with a doubled path (you could see it by running the same commands above without --reftable, but of course not with your patch series as-is because it will _always_ choose reftable).
builtin/init-db.c +16 -9
@@ -213,6 +213,8 @@ static int create_default_files(const char *template_path, is_bare_repository_cfg = init_is_bare_repository;if (init_shared_repository !=-1 )set_shared_repository (init_shared_repository); +if (flags &INIT_DB_REFTABLE ) + the_repository->ref_storage_format =xstrdup ("reftable" );/* * We would have created the above under user's umask -- under
@@ -222,6 +224,15 @@ static int create_default_files(const char *template_path,adjust_shared_perm (get_git_dir ()); } +/* +* Check to see if .git/HEAD exists; this must happen before +* initializing the ref db, because we want to see if there is an +* existing HEAD. +*/ + path =git_path_buf (&buf,"HEAD" ); + reinit = (!access (path,R_OK ) || +readlink (path, junk,sizeof (junk) -1 ) !=-1 ); +/* * We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository.
@@ -234,18 +245,14 @@ static int create_default_files(const char *template_path,* Create the default symlink from ".git/HEAD" to the "master" * branch, if it does not exist yet. */ -if (flags &INIT_DB_REFTABLE ) { - reinit =0 ;/* XXX - how do we recognize a reinit, -* and what should we do? */ - }else { - path =git_path_buf (&buf,"HEAD" ); - reinit = (!access (path,R_OK ) || -readlink (path, junk,sizeof (junk) -1 ) !=-1 ); - } -if (!reinit) {if (create_symref ("HEAD" ,"refs/heads/master" ,NULL ) <0 )exit (1 ); + }else { +/* +* XXX should check whether our ref backend matches the +* original one; if not, either die() or convert +*/ }/* This forces creation of new config file */
refs.c +3 -4
@@ -1859,10 +1859,9 @@ struct ref_store *get_main_ref_store(struct repository *r)BUG ("attempting to get main_ref_store outside of repository" ); r->refs =ref_store_init (r->gitdir, -/* XXX r->ref_storage_format == NULL. Where -* should the config file be parsed out? */ - r->ref_storage_format ? r->ref_storage_format : -"reftable" , + r->ref_storage_format ? + r->ref_storage_format : +"files" ,REF_STORE_ALL_CAPS );return r->refs; }
refs/files-backend.c +1 -0
@@ -3166,6 +3166,7 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)/* * Create .git/refs/{heads,tags} */ +strbuf_reset (&sb);files_ref_path (refs, &sb,"refs/heads" );safe_create_dir (sb.buf,1 );
repository.c +1 -3
@@ -174,9 +174,7 @@ int repo_init(struct repository *repo,if (worktree)repo_set_worktree (repo, worktree); - repo->ref_storage_format = format.ref_storage !=NULL ? -xstrdup (format.ref_storage) : -"files" ;/* XXX */ + repo->ref_storage_format =xstrdup_or_null (format.ref_storage);clear_repository_format (&format);return 0 ;
setup.c +5 -2
@@ -450,7 +450,7 @@ static int check_repo_format(const char *var, const char *value, void *vdata) data->partial_clone =xstrdup (value); }else if (!strcmp (ext,"worktreeconfig" )) { data->worktree_config =git_config_bool (var, value); - }else if (!strcmp (ext,"refStorage" )) { + }else if (!strcmp (ext,"refstorage" )) { data->ref_storage =xstrdup (value); }else string_list_append (&data->unknown_extensions, ext);
@@ -1184,8 +1184,11 @@ const char *setup_git_directory_gently(int *nongit_ok) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ;setup_git_env (gitdir); } -if (startup_info->have_repository) +if (startup_info->have_repository) {repo_set_hash_algo (the_repository, repo_fmt.hash_algo); + the_repository->ref_storage_format = +xstrdup_or_null (repo_fmt.ref_storage); + } }strbuf_release (&dir);
v3 cover Reftable support git-core
In reply to Han-Wen Nienhuys via GitGitGadgetThis adds the reftable library, and hooks it up as a ref backend.
At this point, I am mainly interested in feedback on the spots marked with XXX in the Git source code, in particular, how to handle reflog expiry in this backend.
v2
* address Jun's nits.
* address Dscho's portability comments
* more background in commit messages.
Han-Wen Nienhuys (6):
refs.h: clarify reflog iteration order
setup.c: enable repo detection for reftable
create .git/refs in files-backend.c
refs: document how ref_iterator_advance_fn should handle symrefs Add reftable library
Reftable support for git-core
Makefile | 24 +- builtin/init-db.c | 42 +- cache.h | 2 + refs.c | 22 +- refs.h | 5 +- refs/files-backend.c | 5 + refs/refs-internal.h | 6 + refs/reftable-backend.c | 880 +++++++++++++++++++++++++++++++ reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 401 ++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 286 ++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 ++++ reftable/pq.h | 34 ++ reftable/reader.c | 708 +++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 +++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 399 ++++++++++++++ reftable/slice.c | 199 +++++++ reftable/slice.h | 39 ++ reftable/stack.c | 983 ++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 57 ++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 622 ++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ repository.c | 4 + repository.h | 3 + setup.c | 27 +- 45 files changed, 7255 insertions(+), 33 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
base-commit: 5b0ca878e008e82f91300091e793427205ce3544
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Range-diff vs v2:
-: ---------- > 1: c00403c94d refs.h: clarify reflog iteration order 1: 174b98f6db ! 2: 57c7342319 setup.c: enable repo detection for reftable
@@ -10,7 +10,6 @@
* allow missing HEAD if there is a reftable/
- Change-Id: I5d22317a15a84c8529aa503ae357a4afba247fe9
Signed-off-by Han-Wen Nienhuys <[email protected]>
diff --git a/setup.c b/setup.c
2: d7d642dcf6 ! 3: 5b7060cb2f create .git/refs in files-backend.c
@@ -2,11 +2,10 @@
create .git/refs in files-backend.c
- This prepares for supporting the reftable format, which creates a file
- in that place.
+ This prepares for supporting the reftable format, which will want
+ create its own file system layout in .git
Signed-off-by: Han-Wen Nienhuys <[email protected]>
- Change-Id: I2fc47c89f5ec605734007ceff90321c02474aa92
diff --git a/builtin/init-db.c b/builtin/init-db.c
--- a/builtin/init-db.c
@@ -30,7 +29,8 @@
+ files_ref_path(refs, &sb, "refs");
+ safe_create_dir(sb.buf, 1);
-+ // XXX adjust_shared_perm ?
++ /* adjust permissions even if directory already exists. */
++ adjust_shared_perm(sb.buf);
+
/*
* Create .git/refs/{heads,tags} 3: 9cf185b51f ! 4: 1b01c735a9 Document how ref iterators and symrefs interact
@@ -1,20 +1,21 @@
Author: Han-Wen Nienhuys <[email protected]>
- Document how ref iterators and symrefs interact
+ refs: document how ref_iterator_advance_fn should handle symrefs
- Change-Id: Ie3ee63c52254c000ef712986246ca28f312b4301
Signed-off-by Han-Wen Nienhuys <[email protected]>
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@
- * to the next entry, ref_iterator_advance() aborts the iteration,
- * frees the ref_iterator, and returns ITER_ERROR.
- *
-+ * Ref iterators cannot return symref targets, so symbolic refs must be -+ * dereferenced during the iteration.
-+ *
- * The reference currently being looked at can be peeled by calling
- * ref_iterator_peel(). This function is often faster than peel_ref(),
- * so it should be preferred when iterating over references.
+
+ /* Virtual function declarations for ref_iterators: */
+
++/*
++ * backend-specific implementation of ref_iterator_advance.
++ * For symrefs, the function should set REF_ISSYMREF, and it should also dereference ++ * the symref to provide the OID referent.
++ */
+ typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator); +
+ typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,
4: 2106ff286b ! 5: eb0df10068 Add reftable library
@@ -30,7 +30,6 @@
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by: Han-Wen Nienhuys <[email protected]>
- Change-Id: Id396ff42be8b42b9e11f194a32e2f95b8250c109
diff --git a/reftable/LICENSE b/reftable/LICENSE
new file mode 100644
@@ -88,6 +87,8 @@
+ git --git-dir reftable-repo/.git show --no-patch origin/master \
+ > reftable/VERSION && \
+ echo '/* empty */' > reftable/config.h
++ rm reftable/*_test.c reftable/test_framework.*
++ git add reftable/*.[ch]
+
+Bugfixes should be accompanied by a test and applied to upstream project at
+https://github.com/google/reftable.
@@ -97,11 +98,11 @@
--- /dev/null
+++ b/reftable/VERSION
@@
-+commit c616d53b88657c3a5fe4d2e7243a48effc34c626
++commit e54326f73d95bfe8b17f264c400f4c365dbd5e5e
+Author: Han-Wen Nienhuys <[email protected]>
-+Date: Mon Jan 27 15:05:43 2020 +0100
++Date: Tue Feb 4 19:48:17 2020 +0100
+
-+ C: ban // comments
++ C: PRI?MAX use; clang-format.
diff --git a/reftable/basics.c b/reftable/basics.c
new file mode 100644
@@ -327,7 +328,6 @@
+
+#define true 1
+#define false 0
-+#define ARRAYSIZE(a) sizeof(a) / sizeof(a[0])
+
+void put_u24(byte *out, uint32_t i);
+uint32_t get_u24(byte *in);
@@ -528,9 +528,10 @@
+ slice_resize(&uncompressed, sz);
+ memcpy(uncompressed.buf, block->data, block_header_skip);
+
-+ if (Z_OK !=
-+ uncompress2(uncompressed.buf + block_header_skip, &dst_len,
-+ block->data + block_header_skip, &src_len)) {
++ if (Z_OK != uncompress_return_consumed(
++ uncompressed.buf + block_header_skip,
++ &dst_len, block->data + block_header_skip,
++ &src_len)) {
+ free(slice_yield(&uncompressed));
+ return ZLIB_ERROR;
+ }
@@ -545,8 +546,8 @@
+ } else if (sz < full_block_size && sz < block->len &&
+ block->data[sz] != 0) {
+ /* If the block is smaller than the full block size, it is
-+ padded (data followed by '\0') or the next block is
-+ unaligned. */
++ padded (data followed by '\0') or the next block is
++ unaligned. */
+ full_block_size = sz;
+ }
+
@@ -750,8 +751,7 @@
+
+void block_writer_clear(struct block_writer *bw)
+{
-+ free(bw->restarts);
-+ bw->restarts = NULL;
++ FREE_AND_NULL(bw->restarts);
+ free(slice_yield(&bw->last_key));
+ /* the block is not owned. */
+}
@@ -833,163 +833,6 @@
+
+#endif
- diff --git a/reftable/block_test.c b/reftable/block_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/block_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "block.h"
-+
-+#include "system.h"
-+
-+#include "basics.h"
-+#include "constants.h"
-+#include "record.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+struct binsearch_args {
-+ int key;
-+ int *arr;
-+};
-+
-+static int binsearch_func(int i, void *void_args)
-+{
-+ struct binsearch_args *args = (struct binsearch_args *)void_args;
-+
-+ return args->key < args->arr[i];
-+}
-+
-+void test_binsearch()
-+{
-+ int arr[] = { 2, 4, 6, 8, 10 };
-+ int sz = ARRAYSIZE(arr);
-+ struct binsearch_args args = {
-+ .arr = arr,
-+ };
-+
-+ int i = 0;
-+ for (i = 1; i < 11; i++) {
-+ args.key = i;
-+ int res = binsearch(sz, &binsearch_func, &args);
-+
-+ if (res < sz) {
-+ assert(args.key < arr[res]);
-+ if (res > 0) {
-+ assert(args.key >= arr[res - 1]);
-+ }
-+ } else {
-+ assert(args.key == 10 || args.key == 11);
-+ }
-+ }
-+}
-+
-+void test_block_read_write()
-+{
-+ const int header_off = 21; /* random */
-+ const int N = 30;
-+ char *names[N];
-+ const int block_size = 1024;
-+ struct block block = {};
-+ block.data = calloc(block_size, 1);
-+ block.len = block_size;
-+
-+ struct block_writer bw = {};
-+ block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
-+ header_off, SHA1_SIZE);
-+ struct ref_record ref = {};
-+ struct record rec = {};
-+ record_from_ref(&rec, &ref);
-+
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ char name[100];
-+ snprintf(name, sizeof(name), "branch%02d", i);
-+
-+ byte hash[SHA1_SIZE];
-+ memset(hash, i, sizeof(hash));
-+
-+ ref.ref_name = name;
-+ ref.value = hash;
-+ names[i] = strdup(name);
-+ int n = block_writer_add(&bw, rec);
-+ ref.ref_name = NULL;
-+ ref.value = NULL;
-+ assert(n == 0);
-+ }
-+
-+ int n = block_writer_finish(&bw);
-+ assert(n > 0);
-+
-+ block_writer_clear(&bw);
-+
-+ struct block_reader br = {};
-+ block_reader_init(&br, &block, header_off, block_size, SHA1_SIZE);
-+
-+ struct block_iter it = {};
-+ block_reader_start(&br, &it);
-+
-+ int j = 0;
-+ while (true) {
-+ int r = block_iter_next(&it, rec);
-+ assert(r >= 0);
-+ if (r > 0) {
-+ break;
-+ }
-+ assert_streq(names[j], ref.ref_name);
-+ j++;
-+ }
-+
-+ record_clear(rec);
-+ block_iter_close(&it);
-+
-+ struct slice want = {};
-+ for (i = 0; i < N; i++) {
-+ slice_set_string(&want, names[i]);
-+
-+ struct block_iter it = {};
-+ int n = block_reader_seek(&br, &it, want);
-+ assert(n == 0);
-+
-+ n = block_iter_next(&it, rec);
-+ assert(n == 0);
-+
-+ assert_streq(names[i], ref.ref_name);
-+
-+ want.len--;
-+ n = block_reader_seek(&br, &it, want);
-+ assert(n == 0);
-+
-+ n = block_iter_next(&it, rec);
-+ assert(n == 0);
-+ assert_streq(names[10 * (i / 10)], ref.ref_name);
-+
-+ block_iter_close(&it);
-+ }
-+
-+ record_clear(rec);
-+ free(block.data);
-+ free(slice_yield(&want));
-+ for (i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+}
-+
-+int main()
-+{
-+ add_test_case("binsearch", &test_binsearch);
-+ add_test_case("block_read_write", &test_block_read_write);
-+ test_main();
-+}
-
diff --git a/reftable/blocksource.h b/reftable/blocksource.h
new file mode 100644
--- /dev/null
@@ -1324,8 +1167,7 @@
+ }
+ it->ops->close(it->iter_arg);
+ it->ops = NULL;
-+ free(it->iter_arg);
-+ it->iter_arg = NULL;
++ FREE_AND_NULL(it->iter_arg);
+}
+
+int iterator_next_ref(struct iterator it, struct ref_record *ref)
@@ -1751,16 +1593,14 @@
+ for (i = 0; i < mt->stack_len; i++) {
+ reader_free(mt->stack[i]);
+ }
-+ free(mt->stack);
-+ mt->stack = NULL;
++ FREE_AND_NULL(mt->stack);
+ mt->stack_len = 0;
+}
+
+/* clears the list of subtable, without affecting the readers themselves. */
+void merged_table_clear(struct merged_table *mt)
+{
-+ free(mt->stack);
-+ mt->stack = NULL;
++ FREE_AND_NULL(mt->stack);
+ mt->stack_len = 0;
+}
+
@@ -1897,270 +1737,6 @@
+
+#endif
- diff --git a/reftable/merged_test.c b/reftable/merged_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/merged_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "merged.h"
-+
-+#include "system.h"
-+
-+#include "basics.h"
-+#include "block.h"
-+#include "constants.h"
-+#include "pq.h"
-+#include "reader.h"
-+#include "record.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+void test_pq(void)
-+{
-+ char *names[54] = {};
-+ int N = ARRAYSIZE(names) - 1;
-+
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ char name[100];
-+ snprintf(name, sizeof(name), "%02d", i);
-+ names[i] = strdup(name);
-+ }
-+
-+ struct merged_iter_pqueue pq = {};
-+
-+ i = 1;
-+ do {
-+ struct record rec = new_record(BLOCK_TYPE_REF);
-+ record_as_ref(rec)->ref_name = names[i];
-+
-+ struct pq_entry e = {
-+ .rec = rec,
-+ };
-+ merged_iter_pqueue_add(&pq, e);
-+ merged_iter_pqueue_check(pq);
-+ i = (i * 7) % N;
-+ } while (i != 1);
-+
-+ const char *last = NULL;
-+ while (!merged_iter_pqueue_is_empty(pq)) {
-+ struct pq_entry e = merged_iter_pqueue_remove(&pq);
-+ merged_iter_pqueue_check(pq);
-+ struct ref_record *ref = record_as_ref(e.rec);
-+
-+ if (last != NULL) {
-+ assert(strcmp(last, ref->ref_name) < 0);
-+ }
-+ last = ref->ref_name;
-+ ref->ref_name = NULL;
-+ free(ref);
-+ }
-+
-+ for (i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+
-+ merged_iter_pqueue_clear(&pq);
-+}
-+
-+void write_test_table(struct slice *buf, struct ref_record refs[], int n)
-+{
-+ int min = 0xffffffff;
-+ int max = 0;
-+ int i = 0;
-+ for (i = 0; i < n; i++) {
-+ uint64_t ui = refs[i].update_index;
-+ if (ui > max) {
-+ max = ui;
-+ }
-+ if (ui < min) {
-+ min = ui;
-+ }
-+ }
-+
-+ struct write_options opts = {
-+ .block_size = 256,
-+ };
-+
-+ struct writer *w = new_writer(&slice_write_void, buf, &opts);
-+ writer_set_limits(w, min, max);
-+
-+ for (i = 0; i < n; i++) {
-+ uint64_t before = refs[i].update_index;
-+ int n = writer_add_ref(w, &refs[i]);
-+ assert(n == 0);
-+ assert(before == refs[i].update_index);
-+ }
-+
-+ int err = writer_close(w);
-+ assert(err == 0);
-+
-+ writer_free(w);
-+ w = NULL;
-+}
-+
-+static struct merged_table *merged_table_from_records(struct ref_record **refs,
-+ int *sizes,
-+ struct slice *buf, int n)
-+{
-+ struct block_source *source = calloc(n, sizeof(*source));
-+ struct reader **rd = calloc(n, sizeof(*rd));
-+ int i = 0;
-+ for (i = 0; i < n; i++) {
-+ write_test_table(&buf[i], refs[i], sizes[i]);
-+ block_source_from_slice(&source[i], &buf[i]);
-+
-+ int err = new_reader(&rd[i], source[i], "name");
-+ assert(err == 0);
-+ }
-+
-+ struct merged_table *mt = NULL;
-+ int err = new_merged_table(&mt, rd, n);
-+ assert(err == 0);
-+ return mt;
-+}
-+
-+void test_merged_between(void)
-+{
-+ byte hash1[SHA1_SIZE];
-+ byte hash2[SHA1_SIZE];
-+
-+ set_test_hash(hash1, 1);
-+ set_test_hash(hash2, 2);
-+ struct ref_record r1[] = { {
-+ .ref_name = "b",
-+ .update_index = 1,
-+ .value = hash1,
-+ } };
-+ struct ref_record r2[] = { {
-+ .ref_name = "a",
-+ .update_index = 2,
-+ } };
-+
-+ struct ref_record *refs[] = { r1, r2 };
-+ int sizes[] = { 1, 1 };
-+ struct slice bufs[2] = {};
-+ struct merged_table *mt =
-+ merged_table_from_records(refs, sizes, bufs, 2);
-+
-+ struct iterator it = {};
-+ int err = merged_table_seek_ref(mt, &it, "a");
-+ assert(err == 0);
-+
-+ struct ref_record ref = {};
-+ err = iterator_next_ref(it, &ref);
-+ assert_err(err);
-+ assert(ref.update_index == 2);
-+}
-+
-+void test_merged(void)
-+{
-+ byte hash1[SHA1_SIZE];
-+ byte hash2[SHA1_SIZE];
-+
-+ set_test_hash(hash1, 1);
-+ set_test_hash(hash2, 2);
-+ struct ref_record r1[] = { {
-+ .ref_name = "a",
-+ .update_index = 1,
-+ .value = hash1,
-+ },
-+ {
-+ .ref_name = "b",
-+ .update_index = 1,
-+ .value = hash1,
-+ },
-+ {
-+ .ref_name = "c",
-+ .update_index = 1,
-+ .value = hash1,
-+ } };
-+ struct ref_record r2[] = { {
-+ .ref_name = "a",
-+ .update_index = 2,
-+ } };
-+ struct ref_record r3[] = {
-+ {
-+ .ref_name = "c",
-+ .update_index = 3,
-+ .value = hash2,
-+ },
-+ {
-+ .ref_name = "d",
-+ .update_index = 3,
-+ .value = hash1,
-+ },
-+ };
-+
-+ struct ref_record *refs[] = { r1, r2, r3 };
-+ int sizes[3] = { 3, 1, 2 };
-+ struct slice bufs[3] = {};
-+
-+ struct merged_table *mt =
-+ merged_table_from_records(refs, sizes, bufs, 3);
-+
-+ struct iterator it = {};
-+ int err = merged_table_seek_ref(mt, &it, "a");
-+ assert(err == 0);
-+
-+ struct ref_record *out = NULL;
-+ int len = 0;
-+ int cap = 0;
-+ while (len < 100) { /* cap loops/recursion. */
-+ struct ref_record ref = {};
-+ int err = iterator_next_ref(it, &ref);
-+ if (err > 0) {
-+ break;
-+ }
-+ if (len == cap) {
-+ cap = 2 * cap + 1;
-+ out = realloc(out, sizeof(struct ref_record) * cap);
-+ }
-+ out[len++] = ref;
-+ }
-+ iterator_destroy(&it);
-+
-+ struct ref_record want[] = {
-+ r2[0],
-+ r1[1],
-+ r3[0],
-+ r3[1],
-+ };
-+ assert(ARRAYSIZE(want) == len);
-+ int i = 0;
-+ for (i = 0; i < len; i++) {
-+ assert(ref_record_equal(&want[i], &out[i], SHA1_SIZE));
-+ }
-+ for (i = 0; i < len; i++) {
-+ ref_record_clear(&out[i]);
-+ }
-+ free(out);
-+
-+ for (i = 0; i < 3; i++) {
-+ free(slice_yield(&bufs[i]));
-+ }
-+ merged_table_close(mt);
-+ merged_table_free(mt);
-+}
-+
-+/* XXX test refs_for(oid) */
-+
-+int main()
-+{
-+ add_test_case("test_merged_between", &test_merged_between);
-+ add_test_case("test_pq", &test_pq);
-+ add_test_case("test_merged", &test_merged);
-+ test_main();
-+}
-
diff --git a/reftable/pq.c b/reftable/pq.c
new file mode 100644
--- /dev/null
@@ -2241,12 +1817,7 @@
+ break;
+ }
+
-+ {
-+ struct pq_entry tmp = pq->heap[min];
-+ pq->heap[min] = pq->heap[i];
-+ pq->heap[i] = tmp;
-+ }
-+
++ SWAP(pq->heap[i], pq->heap[min]);
+ i = min;
+ }
+
@@ -2269,11 +1840,7 @@
+ break;
+ }
+
-+ {
-+ struct pq_entry tmp = pq->heap[j];
-+ pq->heap[j] = pq->heap[i];
-+ pq->heap[i] = tmp;
-+ }
++ SWAP(pq->heap[j], pq->heap[i]);
+
+ i = j;
+ }
@@ -2286,8 +1853,7 @@
+ record_clear(pq->heap[i].rec);
+ free(record_yield(&pq->heap[i].rec));
+ }
-+ free(pq->heap);
-+ pq->heap = NULL;
++ FREE_AND_NULL(pq->heap);
+ pq->len = pq->cap = 0;
+}
@@ -2561,8 +2127,7 @@
+ return;
+ }
+ reader_return_block(ti->r, &ti->bi.br->block);
-+ free(ti->bi.br);
-+ ti->bi.br = NULL;
++ FREE_AND_NULL(ti->bi.br);
+
+ ti->bi.last_key.len = 0;
+ ti->bi.next_off = 0;
@@ -2934,8 +2499,7 @@
+void reader_close(struct reader *r)
+{
+ block_source_close(&r->source);
-+ free(r->name);
-+ r->name = NULL;
++ FREE_AND_NULL(r->name);
+}
+
+int new_reader(struct reader **p, struct block_source src, char const *name)
@@ -3271,7 +2835,7 @@
+ assert(hash_size > 0);
+
+ /* This is simple and correct, but we could probably reuse the hash
-+ fields. */
++ fields. */
+ ref_record_clear(ref);
+ if (src->ref_name != NULL) {
+ ref->ref_name = strdup(src->ref_name);
@@ -3491,16 +3055,13 @@
+ }
+
+ if (!seen_target && r->target != NULL) {
-+ free(r->target);
-+ r->target = NULL;
++ FREE_AND_NULL(r->target);
+ }
+ if (!seen_target_value && r->target_value != NULL) {
-+ free(r->target_value);
-+ r->target_value = NULL;
++ FREE_AND_NULL(r->target_value);
+ }
+ if (!seen_value && r->value != NULL) {
-+ free(r->value);
-+ r->value = NULL;
++ FREE_AND_NULL(r->value);
+ }
+
+ return start.len - in.len;
@@ -3588,8 +3149,8 @@
+static void obj_record_clear(void *rec)
+{
+ struct obj_record *ref = (struct obj_record *)rec;
-+ free(ref->hash_prefix);
-+ free(ref->offsets);
++ FREE_AND_NULL(ref->hash_prefix);
++ FREE_AND_NULL(ref->offsets);
+ memset(ref, 0, sizeof(struct obj_record));
+}
+
@@ -3713,9 +3274,9 @@
+{
+ char hex[SHA256_SIZE + 1] = {};
+
-+ printf("log{%s(%" PRIdMAX ") %s <%s> %lu %04d\n", log->ref_name,
-+ log->update_index, log->name, log->email, log->time,
-+ log->tz_offset);
++ printf("log{%s(%" PRIdMAX ") %s <%s> %" PRIuMAX " %04d\n",
++ log->ref_name, log->update_index, log->name, log->email,
++ log->time, log->tz_offset);
+ hex_format(hex, log->old_hash, hash_size);
+ printf("%s => ", hex);
+ hex_format(hex, log->new_hash, hash_size);
@@ -4306,10 +3867,10 @@
+
+#endif
- diff --git a/reftable/record_test.c b/reftable/record_test.c
+ diff --git a/reftable/reftable.h b/reftable/reftable.h
new file mode 100644
--- /dev/null
- +++ b/reftable/record_test.c
+ +++ b/reftable/reftable.h
@@
+/*
+Copyright 2020 Google LLC
@@ -4319,402 +3880,64 @@
+https://developers.google.com/open-source/licenses/bsd
+*/
+
-+#include "record.h"
++#ifndef REFTABLE_H
++#define REFTABLE_H
+
+#include "system.h"
+
-+#include "basics.h"
-+#include "constants.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+void varint_roundtrip()
-+{
-+ uint64_t inputs[] = { 0,
-+ 1,
-+ 27,
-+ 127,
-+ 128,
-+ 257,
-+ 4096,
-+ ((uint64_t)1 << 63),
-+ ((uint64_t)1 << 63) + ((uint64_t)1 << 63) - 1 };
-+ int i = 0;
-+ for (i = 0; i < ARRAYSIZE(inputs); i++) {
-+ byte dest[10];
++typedef uint8_t byte;
++typedef byte bool;
+
-+ struct slice out = { .buf = dest, .len = 10, .cap = 10 };
++/* block_source is a generic wrapper for a seekable readable file.
++ It is generally passed around by value.
++ */
++struct block_source {
++ struct block_source_vtable *ops;
++ void *arg;
++};
+
-+ uint64_t in = inputs[i];
-+ int n = put_var_int(out, in);
-+ assert(n > 0);
-+ out.len = n;
++/* a contiguous segment of bytes. It keeps track of its generating block_source
++ so it can return itself into the pool.
++*/
++struct block {
++ byte *data;
++ int len;
++ struct block_source source;
++};
+
-+ uint64_t got = 0;
-+ n = get_var_int(&got, out);
-+ assert(n > 0);
++/* block_source_vtable are the operations that make up block_source */
++struct block_source_vtable {
++ /* returns the size of a block source */
++ uint64_t (*size)(void *source);
+
-+ assert(got == in);
-+ }
-+}
++ /* reads a segment from the block source. It is an error to read
++ beyond the end of the block */
++ int (*read_block)(void *source, struct block *dest, uint64_t off,
++ uint32_t size);
++ /* mark the block as read; may return the data back to malloc */
++ void (*return_block)(void *source, struct block *blockp);
+
-+void test_common_prefix()
-+{
-+ struct {
-+ const char *a, *b;
-+ int want;
-+ } cases[] = {
-+ { "abc", "ab", 2 },
-+ { "", "abc", 0 },
-+ { "abc", "abd", 2 },
-+ { "abc", "pqr", 0 },
-+ };
++ /* release all resources associated with the block source */
++ void (*close)(void *source);
++};
+
-+ int i = 0;
-+ for (i = 0; i < ARRAYSIZE(cases); i++) {
-+ struct slice a = {};
-+ struct slice b = {};
-+ slice_set_string(&a, cases[i].a);
-+ slice_set_string(&b, cases[i].b);
++/* opens a file on the file system as a block_source */
++int block_source_from_file(struct block_source *block_src, const char *name);
+
-+ int got = common_prefix_size(a, b);
-+ assert(got == cases[i].want);
++/* write_options sets options for writing a single reftable. */
++struct write_options {
++ /* do not pad out blocks to block size. */
++ bool unpadded;
+
-+ free(slice_yield(&a));
-+ free(slice_yield(&b));
-+ }
-+}
++ /* the blocksize. Should be less than 2^24. */
++ uint32_t block_size;
+
-+void set_hash(byte *h, int j)
-+{
-+ int i = 0;
-+ for (i = 0; i < SHA1_SIZE; i++) {
-+ h[i] = (j >> i) & 0xff;
-+ }
-+}
++ /* do not generate a SHA1 => ref index. */
++ bool skip_index_objects;
+
-+void test_ref_record_roundtrip()
-+{
-+ int i = 0;
-+ for (i = 0; i <= 3; i++) {
-+ printf("subtest %d\n", i);
-+ struct ref_record in = {};
-+ switch (i) {
-+ case 0:
-+ break;
-+ case 1:
-+ in.value = malloc(SHA1_SIZE);
-+ set_hash(in.value, 1);
-+ break;
-+ case 2:
-+ in.value = malloc(SHA1_SIZE);
-+ set_hash(in.value, 1);
-+ in.target_value = malloc(SHA1_SIZE);
-+ set_hash(in.target_value, 2);
-+ break;
-+ case 3:
-+ in.target = strdup("target");
-+ break;
-+ }
-+ in.ref_name = strdup("refs/heads/master");
-+
-+ struct record rec = {};
-+ record_from_ref(&rec, &in);
-+ assert(record_val_type(rec) == i);
-+ byte buf[1024];
-+ struct slice key = {};
-+ record_key(rec, &key);
-+ struct slice dest = {
-+ .buf = buf,
-+ .len = sizeof(buf),
-+ };
-+ int n = record_encode(rec, dest, SHA1_SIZE);
-+ assert(n > 0);
-+
-+ struct ref_record out = {};
-+ struct record rec_out = {};
-+ record_from_ref(&rec_out, &out);
-+ int m = record_decode(rec_out, key, i, dest, SHA1_SIZE);
-+ assert(n == m);
-+
-+ assert((out.value != NULL) == (in.value != NULL));
-+ assert((out.target_value != NULL) == (in.target_value != NULL));
-+ assert((out.target != NULL) == (in.target != NULL));
-+ free(slice_yield(&key));
-+ record_clear(rec_out);
-+ ref_record_clear(&in);
-+ }
-+}
-+
-+void test_log_record_roundtrip()
-+{
-+ struct log_record in = {
-+ .ref_name = strdup("refs/heads/master"),
-+ .old_hash = malloc(SHA1_SIZE),
-+ .new_hash = malloc(SHA1_SIZE),
-+ .name = strdup("han-wen"),
-+ .email = strdup("[email protected]"),
-+ .message = strdup("test"),
-+ .update_index = 42,
-+ .time = 1577123507,
-+ .tz_offset = 100,
-+ };
-+
-+ struct record rec = {};
-+ record_from_log(&rec, &in);
-+
-+ struct slice key = {};
-+ record_key(rec, &key);
-+
-+ byte buf[1024];
-+ struct slice dest = {
-+ .buf = buf,
-+ .len = sizeof(buf),
-+ };
-+
-+ int n = record_encode(rec, dest, SHA1_SIZE);
-+ assert(n > 0);
-+
-+ struct log_record out = {};
-+ struct record rec_out = {};
-+ record_from_log(&rec_out, &out);
-+ int valtype = record_val_type(rec);
-+ int m = record_decode(rec_out, key, valtype, dest, SHA1_SIZE);
-+ assert(n == m);
-+
-+ assert(log_record_equal(&in, &out, SHA1_SIZE));
-+ log_record_clear(&in);
-+ free(slice_yield(&key));
-+ record_clear(rec_out);
-+}
-+
-+void test_u24_roundtrip()
-+{
-+ uint32_t in = 0x112233;
-+ byte dest[3];
-+
-+ put_u24(dest, in);
-+ uint32_t out = get_u24(dest);
-+ assert(in == out);
-+}
-+
-+void test_key_roundtrip()
-+{
-+ struct slice dest = {}, last_key = {}, key = {}, roundtrip = {};
-+
-+ slice_resize(&dest, 1024);
-+ slice_set_string(&last_key, "refs/heads/master");
-+ slice_set_string(&key, "refs/tags/bla");
-+
-+ bool restart;
-+ byte extra = 6;
-+ int n = encode_key(&restart, dest, last_key, key, extra);
-+ assert(!restart);
-+ assert(n > 0);
-+
-+ byte rt_extra;
-+ int m = decode_key(&roundtrip, &rt_extra, last_key, dest);
-+ assert(n == m);
-+ assert(slice_equal(key, roundtrip));
-+ assert(rt_extra == extra);
-+
-+ free(slice_yield(&last_key));
-+ free(slice_yield(&key));
-+ free(slice_yield(&dest));
-+ free(slice_yield(&roundtrip));
-+}
-+
-+void print_bytes(byte *p, int l)
-+{
-+ int i = 0;
-+ for (i = 0; i < l; i++) {
-+ byte c = *p;
-+ if (c < 32) {
-+ c = '.';
-+ }
-+ printf("%02x[%c] ", p[i], c);
-+ }
-+ printf("(%d)\n", l);
-+}
-+
-+void test_obj_record_roundtrip()
-+{
-+ byte testHash1[SHA1_SIZE] = {};
-+ set_hash(testHash1, 1);
-+ uint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 };
-+
-+ struct obj_record recs[3] = { {
-+ .hash_prefix = testHash1,
-+ .hash_prefix_len = 5,
-+ .offsets = till9,
-+ .offset_len = 3,
-+ },
-+ {
-+ .hash_prefix = testHash1,
-+ .hash_prefix_len = 5,
-+ .offsets = till9,
-+ .offset_len = 9,
-+ },
-+ {
-+ .hash_prefix = testHash1,
-+ .hash_prefix_len = 5,
-+ }
-+
-+ };
-+ int i = 0;
-+ for (i = 0; i < ARRAYSIZE(recs); i++) {
-+ printf("subtest %d\n", i);
-+ struct obj_record in = recs[i];
-+ byte buf[1024];
-+ struct record rec = {};
-+ record_from_obj(&rec, &in);
-+ struct slice key = {};
-+ record_key(rec, &key);
-+ struct slice dest = {
-+ .buf = buf,
-+ .len = sizeof(buf),
-+ };
-+ int n = record_encode(rec, dest, SHA1_SIZE);
-+ assert(n > 0);
-+ byte extra = record_val_type(rec);
-+ struct obj_record out = {};
-+ struct record rec_out = {};
-+ record_from_obj(&rec_out, &out);
-+ int m = record_decode(rec_out, key, extra, dest, SHA1_SIZE);
-+ assert(n == m);
-+
-+ assert(in.hash_prefix_len == out.hash_prefix_len);
-+ assert(in.offset_len == out.offset_len);
-+
-+ assert(!memcmp(in.hash_prefix, out.hash_prefix,
-+ in.hash_prefix_len));
-+ assert(0 == memcmp(in.offsets, out.offsets,
-+ sizeof(uint64_t) * in.offset_len));
-+ free(slice_yield(&key));
-+ record_clear(rec_out);
-+ }
-+}
-+
-+void test_index_record_roundtrip()
-+{
-+ struct index_record in = { .offset = 42 };
-+
-+ slice_set_string(&in.last_key, "refs/heads/master");
-+
-+ struct slice key = {};
-+ struct record rec = {};
-+ record_from_index(&rec, &in);
-+ record_key(rec, &key);
-+
-+ assert(0 == slice_compare(key, in.last_key));
-+
-+ byte buf[1024];
-+ struct slice dest = {
-+ .buf = buf,
-+ .len = sizeof(buf),
-+ };
-+ int n = record_encode(rec, dest, SHA1_SIZE);
-+ assert(n > 0);
-+
-+ byte extra = record_val_type(rec);
-+ struct index_record out = {};
-+ struct record out_rec;
-+ record_from_index(&out_rec, &out);
-+ int m = record_decode(out_rec, key, extra, dest, SHA1_SIZE);
-+ assert(m == n);
-+
-+ assert(in.offset == out.offset);
-+
-+ record_clear(out_rec);
-+ free(slice_yield(&key));
-+ free(slice_yield(&in.last_key));
-+}
-+
-+int main()
-+{
-+ add_test_case("test_log_record_roundtrip", &test_log_record_roundtrip);
-+ add_test_case("test_ref_record_roundtrip", &test_ref_record_roundtrip);
-+ add_test_case("varint_roundtrip", &varint_roundtrip);
-+ add_test_case("test_key_roundtrip", &test_key_roundtrip);
-+ add_test_case("test_common_prefix", &test_common_prefix);
-+ add_test_case("test_obj_record_roundtrip", &test_obj_record_roundtrip);
-+ add_test_case("test_index_record_roundtrip",
-+ &test_index_record_roundtrip);
-+ add_test_case("test_u24_roundtrip", &test_u24_roundtrip);
-+ test_main();
-+}
-
- diff --git a/reftable/reftable.h b/reftable/reftable.h
- new file mode 100644
- --- /dev/null
- +++ b/reftable/reftable.h
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#ifndef REFTABLE_H
-+#define REFTABLE_H
-+
-+#include "system.h"
-+
-+typedef uint8_t byte;
-+typedef byte bool;
-+
-+/* block_source is a generic wrapper for a seekable readable file.
-+ It is generally passed around by value.
-+ */
-+struct block_source {
-+ struct block_source_vtable *ops;
-+ void *arg;
-+};
-+
-+/* a contiguous segment of bytes. It keeps track of its generating block_source
-+ so it can return itself into the pool.
-+*/
-+struct block {
-+ byte *data;
-+ int len;
-+ struct block_source source;
-+};
-+
-+/* block_source_vtable are the operations that make up block_source */
-+struct block_source_vtable {
-+ /* returns the size of a block source */
-+ uint64_t (*size)(void *source);
-+
-+ /* reads a segment from the block source. It is an error to read
-+ beyond the end of the block */
-+ int (*read_block)(void *source, struct block *dest, uint64_t off,
-+ uint32_t size);
-+ /* mark the block as read; may return the data back to malloc */
-+ void (*return_block)(void *source, struct block *blockp);
-+
-+ /* release all resources associated with the block source */
-+ void (*close)(void *source);
-+};
-+
-+/* opens a file on the file system as a block_source */
-+int block_source_from_file(struct block_source *block_src, const char *name);
-+
-+/* write_options sets options for writing a single reftable. */
-+struct write_options {
-+ /* do not pad out blocks to block size. */
-+ bool unpadded;
-+
-+ /* the blocksize. Should be less than 2^24. */
-+ uint32_t block_size;
-+
-+ /* do not generate a SHA1 => ref index. */
-+ bool skip_index_objects;
-+
-+ /* how often to write complete keys in each block. */
-+ int restart_interval;
-+};
++ /* how often to write complete keys in each block. */
++ int restart_interval;
++};
+
+/* ref_record holds a ref database entry target_value */
+struct ref_record {
@@ -5049,493 +4272,6 @@
+
+#endif
- diff --git a/reftable/reftable_test.c b/reftable/reftable_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/reftable_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "reftable.h"
-+
-+#include "system.h"
-+
-+#include "basics.h"
-+#include "block.h"
-+#include "constants.h"
-+#include "reader.h"
-+#include "record.h"
-+#include "test_framework.h"
-+
-+static const int update_index = 5;
-+
-+void test_buffer(void)
-+{
-+ struct slice buf = {};
-+
-+ byte in[] = "hello";
-+ slice_write(&buf, in, sizeof(in));
-+ struct block_source source;
-+ block_source_from_slice(&source, &buf);
-+ assert(block_source_size(source) == 6);
-+ struct block out = {};
-+ int n = block_source_read_block(source, &out, 0, sizeof(in));
-+ assert(n == sizeof(in));
-+ assert(!memcmp(in, out.data, n));
-+ block_source_return_block(source, &out);
-+
-+ n = block_source_read_block(source, &out, 1, 2);
-+ assert(n == 2);
-+ assert(!memcmp(out.data, "el", 2));
-+
-+ block_source_return_block(source, &out);
-+ block_source_close(&source);
-+ free(slice_yield(&buf));
-+}
-+
-+void write_table(char ***names, struct slice *buf, int N, int block_size)
-+{
-+ *names = calloc(sizeof(char *), N + 1);
-+
-+ struct write_options opts = {
-+ .block_size = block_size,
-+ };
-+
-+ struct writer *w = new_writer(&slice_write_void, buf, &opts);
-+
-+ writer_set_limits(w, update_index, update_index);
-+ {
-+ struct ref_record ref = {};
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ byte hash[SHA1_SIZE];
-+ set_test_hash(hash, i);
-+
-+ char name[100];
-+ snprintf(name, sizeof(name), "refs/heads/branch%02d",
-+ i);
-+
-+ ref.ref_name = name;
-+ ref.value = hash;
-+ ref.update_index = update_index;
-+ (*names)[i] = strdup(name);
-+
-+ int n = writer_add_ref(w, &ref);
-+ assert(n == 0);
-+ }
-+ }
-+ {
-+ struct log_record log = {};
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ byte hash[SHA1_SIZE];
-+ set_test_hash(hash, i);
-+
-+ char name[100];
-+ snprintf(name, sizeof(name), "refs/heads/branch%02d",
-+ i);
-+
-+ log.ref_name = name;
-+ log.new_hash = hash;
-+ log.update_index = update_index;
-+ log.message = "message";
-+
-+ int n = writer_add_log(w, &log);
-+ assert(n == 0);
-+ }
-+ }
-+
-+ int n = writer_close(w);
-+ assert(n == 0);
-+
-+ struct stats *stats = writer_stats(w);
-+ int i = 0;
-+ for (i = 0; i < stats->ref_stats.blocks; i++) {
-+ int off = i * opts.block_size;
-+ if (off == 0) {
-+ off = HEADER_SIZE;
-+ }
-+ assert(buf->buf[off] == 'r');
-+ }
-+
-+ writer_free(w);
-+ w = NULL;
-+}
-+
-+void test_log_write_read(void)
-+{
-+ int N = 2;
-+ char **names = calloc(sizeof(char *), N + 1);
-+
-+ struct write_options opts = {
-+ .block_size = 256,
-+ };
-+
-+ struct slice buf = {};
-+ struct writer *w = new_writer(&slice_write_void, &buf, &opts);
-+
-+ writer_set_limits(w, 0, N);
-+ {
-+ struct ref_record ref = {};
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ char name[256];
-+ snprintf(name, sizeof(name), "b%02d%0*d", i, 130, 7);
-+ names[i] = strdup(name);
-+ puts(name);
-+ ref.ref_name = name;
-+ ref.update_index = i;
-+
-+ int err = writer_add_ref(w, &ref);
-+ assert_err(err);
-+ }
-+ }
-+
-+ {
-+ struct log_record log = {};
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ byte hash1[SHA1_SIZE], hash2[SHA1_SIZE];
-+ set_test_hash(hash1, i);
-+ set_test_hash(hash2, i + 1);
-+
-+ log.ref_name = names[i];
-+ log.update_index = i;
-+ log.old_hash = hash1;
-+ log.new_hash = hash2;
-+
-+ int err = writer_add_log(w, &log);
-+ assert_err(err);
-+ }
-+ }
-+
-+ int n = writer_close(w);
-+ assert(n == 0);
-+
-+ struct stats *stats = writer_stats(w);
-+ assert(stats->log_stats.blocks > 0);
-+ writer_free(w);
-+ w = NULL;
-+
-+ struct block_source source = {};
-+ block_source_from_slice(&source, &buf);
-+
-+ struct reader rd = {};
-+ int err = init_reader(&rd, source, "file.log");
-+ assert(err == 0);
-+
-+ {
-+ struct iterator it = {};
-+ err = reader_seek_ref(&rd, &it, names[N - 1]);
-+ assert(err == 0);
-+
-+ struct ref_record ref = {};
-+ err = iterator_next_ref(it, &ref);
-+ assert_err(err);
-+
-+ /* end of iteration. */
-+ err = iterator_next_ref(it, &ref);
-+ assert(0 < err);
-+
-+ iterator_destroy(&it);
-+ ref_record_clear(&ref);
-+ }
-+
-+ {
-+ struct iterator it = {};
-+ err = reader_seek_log(&rd, &it, "");
-+ assert(err == 0);
-+
-+ struct log_record log = {};
-+ int i = 0;
-+ while (true) {
-+ int err = iterator_next_log(it, &log);
-+ if (err > 0) {
-+ break;
-+ }
-+
-+ assert_err(err);
-+ assert_streq(names[i], log.ref_name);
-+ assert(i == log.update_index);
-+ i++;
-+ }
-+
-+ assert(i == N);
-+ iterator_destroy(&it);
-+ }
-+
-+ /* cleanup. */
-+ free(slice_yield(&buf));
-+ free_names(names);
-+ reader_close(&rd);
-+}
-+
-+void test_table_read_write_sequential(void)
-+{
-+ char **names;
-+ struct slice buf = {};
-+ int N = 50;
-+ write_table(&names, &buf, N, 256);
-+
-+ struct block_source source = {};
-+ block_source_from_slice(&source, &buf);
-+
-+ struct reader rd = {};
-+ int err = init_reader(&rd, source, "file.ref");
-+ assert(err == 0);
-+
-+ struct iterator it = {};
-+ err = reader_seek_ref(&rd, &it, "");
-+ assert(err == 0);
-+
-+ int j = 0;
-+ while (true) {
-+ struct ref_record ref = {};
-+ int r = iterator_next_ref(it, &ref);
-+ assert(r >= 0);
-+ if (r > 0) {
-+ break;
-+ }
-+ assert(0 == strcmp(names[j], ref.ref_name));
-+ assert(update_index == ref.update_index);
-+
-+ j++;
-+ ref_record_clear(&ref);
-+ }
-+ assert(j == N);
-+ iterator_destroy(&it);
-+ free(slice_yield(&buf));
-+ free_names(names);
-+
-+ reader_close(&rd);
-+}
-+
-+void test_table_write_small_table(void)
-+{
-+ char **names;
-+ struct slice buf = {};
-+ int N = 1;
-+ write_table(&names, &buf, N, 4096);
-+ assert(buf.len < 200);
-+ free(slice_yield(&buf));
-+ free_names(names);
-+}
-+
-+void test_table_read_api(void)
-+{
-+ char **names;
-+ struct slice buf = {};
-+ int N = 50;
-+ write_table(&names, &buf, N, 256);
-+
-+ struct reader rd = {};
-+ struct block_source source = {};
-+ block_source_from_slice(&source, &buf);
-+
-+ int err = init_reader(&rd, source, "file.ref");
-+ assert(err == 0);
-+
-+ struct iterator it = {};
-+ err = reader_seek_ref(&rd, &it, names[0]);
-+ assert(err == 0);
-+
-+ struct log_record log = {};
-+ err = iterator_next_log(it, &log);
-+ assert(err == API_ERROR);
-+
-+ free(slice_yield(&buf));
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+ free(names);
-+ reader_close(&rd);
-+}
-+
-+void test_table_read_write_seek(bool index)
-+{
-+ char **names;
-+ struct slice buf = {};
-+ int N = 50;
-+ write_table(&names, &buf, N, 256);
-+
-+ struct reader rd = {};
-+ struct block_source source = {};
-+ block_source_from_slice(&source, &buf);
-+
-+ int err = init_reader(&rd, source, "file.ref");
-+ assert(err == 0);
-+
-+ if (!index) {
-+ rd.ref_offsets.index_offset = 0;
-+ }
-+
-+ int i = 0;
-+ for (i = 1; i < N; i++) {
-+ struct iterator it = {};
-+ int err = reader_seek_ref(&rd, &it, names[i]);
-+ assert(err == 0);
-+ struct ref_record ref = {};
-+ err = iterator_next_ref(it, &ref);
-+ assert(err == 0);
-+ assert(0 == strcmp(names[i], ref.ref_name));
-+ assert(i == ref.value[0]);
-+
-+ ref_record_clear(&ref);
-+ iterator_destroy(&it);
-+ }
-+
-+ free(slice_yield(&buf));
-+ for (i = 0; i < N; i++) {
-+ free(names[i]);
-+ }
-+ free(names);
-+ reader_close(&rd);
-+}
-+
-+void test_table_read_write_seek_linear(void)
-+{
-+ test_table_read_write_seek(false);
-+}
-+
-+void test_table_read_write_seek_index(void)
-+{
-+ test_table_read_write_seek(true);
-+}
-+
-+void test_table_refs_for(bool indexed)
-+{
-+ int N = 50;
-+
-+ char **want_names = calloc(sizeof(char *), N + 1);
-+
-+ int want_names_len = 0;
-+ byte want_hash[SHA1_SIZE];
-+ set_test_hash(want_hash, 4);
-+
-+ struct write_options opts = {
-+ .block_size = 256,
-+ };
-+
-+ struct slice buf = {};
-+ struct writer *w = new_writer(&slice_write_void, &buf, &opts);
-+ {
-+ struct ref_record ref = {};
-+ int i = 0;
-+ for (i = 0; i < N; i++) {
-+ byte hash[SHA1_SIZE];
-+ memset(hash, i, sizeof(hash));
-+ char fill[51] = {};
-+ memset(fill, 'x', 50);
-+ char name[100];
-+ /* Put the variable part in the start */
-+ snprintf(name, sizeof(name), "br%02d%s", i, fill);
-+ name[40] = 0;
-+ ref.ref_name = name;
-+
-+ byte hash1[SHA1_SIZE];
-+ byte hash2[SHA1_SIZE];
-+
-+ set_test_hash(hash1, i / 4);
-+ set_test_hash(hash2, 3 + i / 4);
-+ ref.value = hash1;
-+ ref.target_value = hash2;
-+
-+ /* 80 bytes / entry, so 3 entries per block. Yields 17 */
-+ /* blocks. */
-+ int n = writer_add_ref(w, &ref);
-+ assert(n == 0);
-+
-+ if (!memcmp(hash1, want_hash, SHA1_SIZE) ||
-+ !memcmp(hash2, want_hash, SHA1_SIZE)) {
-+ want_names[want_names_len++] = strdup(name);
-+ }
-+ }
-+ }
-+
-+ int n = writer_close(w);
-+ assert(n == 0);
-+
-+ writer_free(w);
-+ w = NULL;
-+
-+ struct reader rd;
-+ struct block_source source = {};
-+ block_source_from_slice(&source, &buf);
-+
-+ int err = init_reader(&rd, source, "file.ref");
-+ assert(err == 0);
-+ if (!indexed) {
-+ rd.obj_offsets.present = 0;
-+ }
-+
-+ struct iterator it = {};
-+ err = reader_seek_ref(&rd, &it, "");
-+ assert(err == 0);
-+ iterator_destroy(&it);
-+
-+ err = reader_refs_for(&rd, &it, want_hash, SHA1_SIZE);
-+ assert(err == 0);
-+
-+ struct ref_record ref = {};
-+
-+ int j = 0;
-+ while (true) {
-+ int err = iterator_next_ref(it, &ref);
-+ assert(err >= 0);
-+ if (err > 0) {
-+ break;
-+ }
-+
-+ assert(j < want_names_len);
-+ assert(0 == strcmp(ref.ref_name, want_names[j]));
-+ j++;
-+ ref_record_clear(&ref);
-+ }
-+ assert(j == want_names_len);
-+
-+ free(slice_yield(&buf));
-+ free_names(want_names);
-+ iterator_destroy(&it);
-+ reader_close(&rd);
-+}
-+
-+void test_table_refs_for_no_index(void)
-+{
-+ test_table_refs_for(false);
-+}
-+
-+void test_table_refs_for_obj_index(void)
-+{
-+ test_table_refs_for(true);
-+}
-+
-+int main()
-+{
-+ add_test_case("test_log_write_read", test_log_write_read);
-+ add_test_case("test_table_write_small_table",
-+ &test_table_write_small_table);
-+ add_test_case("test_buffer", &test_buffer);
-+ add_test_case("test_table_read_api", &test_table_read_api);
-+ add_test_case("test_table_read_write_sequential",
-+ &test_table_read_write_sequential);
-+ add_test_case("test_table_read_write_seek_linear",
-+ &test_table_read_write_seek_linear);
-+ add_test_case("test_table_read_write_seek_index",
-+ &test_table_read_write_seek_index);
-+ add_test_case("test_table_read_write_refs_for_no_index",
-+ &test_table_refs_for_no_index);
-+ add_test_case("test_table_read_write_refs_for_obj_index",
-+ &test_table_refs_for_obj_index);
-+ test_main();
-+}
-
diff --git a/reftable/slice.c b/reftable/slice.c
new file mode 100644
--- /dev/null
@@ -5774,61 +4510,17 @@
+byte *slice_yield(struct slice *s);
+void slice_copy(struct slice *dest, struct slice src);
+void slice_resize(struct slice *s, int l);
-+int slice_compare(struct slice a, struct slice b);
-+int slice_write(struct slice *b, byte *data, int sz);
-+int slice_write_void(void *b, byte *data, int sz);
-+void slice_append(struct slice *dest, struct slice add);
-+
-+struct block_source;
-+void block_source_from_slice(struct block_source *bs, struct slice *buf);
-+
-+struct block_source malloc_block_source(void);
-+
-+#endif
-
- diff --git a/reftable/slice_test.c b/reftable/slice_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/slice_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "slice.h"
-+
-+#include "system.h"
-+
-+#include "basics.h"
-+#include "record.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+void test_slice(void)
-+{
-+ struct slice s = {};
-+ slice_set_string(&s, "abc");
-+ assert(0 == strcmp("abc", slice_as_string(&s)));
-+
-+ struct slice t = {};
-+ slice_set_string(&t, "pqr");
++int slice_compare(struct slice a, struct slice b);
++int slice_write(struct slice *b, byte *data, int sz);
++int slice_write_void(void *b, byte *data, int sz);
++void slice_append(struct slice *dest, struct slice add);
+
-+ slice_append(&s, t);
-+ assert(0 == strcmp("abcpqr", slice_as_string(&s)));
++struct block_source;
++void block_source_from_slice(struct block_source *bs, struct slice *buf);
+
-+ free(slice_yield(&s));
-+ free(slice_yield(&t));
-+}
++struct block_source malloc_block_source(void);
+
-+int main()
-+{
-+ add_test_case("test_slice", &test_slice);
-+ test_main();
-+}
++#endif
diff --git a/reftable/stack.c b/reftable/stack.c
new file mode 100644
@@ -5936,10 +4628,8 @@
+ merged_table_free(st->merged);
+ st->merged = NULL;
+
-+ free(st->list_file);
-+ st->list_file = NULL;
-+ free(st->reftable_dir);
-+ st->reftable_dir = NULL;
++ FREE_AND_NULL(st->list_file);
++ FREE_AND_NULL(st->reftable_dir);
+ free(st);
+}
+
@@ -6170,7 +4860,7 @@
+static void format_name(struct slice *dest, uint64_t min, uint64_t max)
+{
+ char buf[100];
-+ snprintf(buf, sizeof(buf), "%012lx-%012lx", min, max);
++ snprintf(buf, sizeof(buf), "%012" PRIxMAX "-%012" PRIxMAX, min, max);
+ slice_set_string(dest, buf);
+}
+
@@ -6867,293 +5557,6 @@
+
+#endif
- diff --git a/reftable/stack_test.c b/reftable/stack_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/stack_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "stack.h"
-+
-+#include "system.h"
-+
-+#include "basics.h"
-+#include "constants.h"
-+#include "record.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+void test_read_file(void)
-+{
-+ char fn[256] = "/tmp/stack.test_read_file.XXXXXX";
-+ int fd = mkstemp(fn);
-+ assert(fd > 0);
-+
-+ char out[1024] = "line1\n\nline2\nline3";
-+
-+ int n = write(fd, out, strlen(out));
-+ assert(n == strlen(out));
-+ int err = close(fd);
-+ assert(err >= 0);
-+
-+ char **names = NULL;
-+ err = read_lines(fn, &names);
-+ assert_err(err);
-+
-+ char *want[] = { "line1", "line2", "line3" };
-+ int i = 0;
-+ for (i = 0; names[i] != NULL; i++) {
-+ assert(0 == strcmp(want[i], names[i]));
-+ }
-+ free_names(names);
-+ remove(fn);
-+}
-+
-+void test_parse_names(void)
-+{
-+ char buf[] = "line\n";
-+ char **names = NULL;
-+ parse_names(buf, strlen(buf), &names);
-+
-+ assert(NULL != names[0]);
-+ assert(0 == strcmp(names[0], "line"));
-+ assert(NULL == names[1]);
-+ free_names(names);
-+}
-+
-+void test_names_equal(void)
-+{
-+ char *a[] = { "a", "b", "c", NULL };
-+ char *b[] = { "a", "b", "d", NULL };
-+ char *c[] = { "a", "b", NULL };
-+
-+ assert(names_equal(a, a));
-+ assert(!names_equal(a, b));
-+ assert(!names_equal(a, c));
-+}
-+
-+int write_test_ref(struct writer *wr, void *arg)
-+{
-+ struct ref_record *ref = arg;
-+
-+ writer_set_limits(wr, ref->update_index, ref->update_index);
-+ int err = writer_add_ref(wr, ref);
-+
-+ return err;
-+}
-+
-+int write_test_log(struct writer *wr, void *arg)
-+{
-+ struct log_record *log = arg;
-+
-+ writer_set_limits(wr, log->update_index, log->update_index);
-+ int err = writer_add_log(wr, log);
-+
-+ return err;
-+}
-+
-+void test_stack_add(void)
-+{
-+ int i = 0;
-+ char dir[256] = "/tmp/stack.test_stack_add.XXXXXX";
-+ assert(mkdtemp(dir));
-+ printf("%s\n", dir);
-+ char fn[256] = "";
-+ strcat(fn, dir);
-+ strcat(fn, "/refs");
-+
-+ struct write_options cfg = {};
-+ struct stack *st = NULL;
-+ int err = new_stack(&st, dir, fn, cfg);
-+ assert_err(err);
-+
-+ struct ref_record refs[2] = {};
-+ struct log_record logs[2] = {};
-+ int N = ARRAYSIZE(refs);
-+ for (i = 0; i < N; i++) {
-+ char buf[256];
-+ snprintf(buf, sizeof(buf), "branch%02d", i);
-+ refs[i].ref_name = strdup(buf);
-+ refs[i].value = malloc(SHA1_SIZE);
-+ refs[i].update_index = i + 1;
-+ set_test_hash(refs[i].value, i);
-+
-+ logs[i].ref_name = strdup(buf);
-+ logs[i].update_index = N + i + 1;
-+ logs[i].new_hash = malloc(SHA1_SIZE);
-+ logs[i].email = strdup("identity@invalid");
-+ set_test_hash(logs[i].new_hash, i);
-+ }
-+
-+ for (i = 0; i < N; i++) {
-+ int err = stack_add(st, &write_test_ref, &refs[i]);
-+ assert_err(err);
-+ }
-+
-+ for (i = 0; i < N; i++) {
-+ int err = stack_add(st, &write_test_log, &logs[i]);
-+ assert_err(err);
-+ }
-+
-+ err = stack_compact_all(st, NULL);
-+ assert_err(err);
-+
-+ for (i = 0; i < N; i++) {
-+ struct ref_record dest = {};
-+ int err = stack_read_ref(st, refs[i].ref_name, &dest);
-+ assert_err(err);
-+ assert(ref_record_equal(&dest, refs + i, SHA1_SIZE));
-+ ref_record_clear(&dest);
-+ }
-+
-+ for (i = 0; i < N; i++) {
-+ struct log_record dest = {};
-+ int err = stack_read_log(st, refs[i].ref_name, &dest);
-+ assert_err(err);
-+ assert(log_record_equal(&dest, logs + i, SHA1_SIZE));
-+ log_record_clear(&dest);
-+ }
-+
-+ /* cleanup */
-+ stack_destroy(st);
-+ for (i = 0; i < N; i++) {
-+ ref_record_clear(&refs[i]);
-+ log_record_clear(&logs[i]);
-+ }
-+}
-+
-+void test_log2(void)
-+{
-+ assert(1 == fastlog2(3));
-+ assert(2 == fastlog2(4));
-+ assert(2 == fastlog2(5));
-+}
-+
-+void test_sizes_to_segments(void)
-+{
-+ uint64_t sizes[] = { 2, 3, 4, 5, 7, 9 };
-+ /* .................0 1 2 3 4 5 */
-+
-+ int seglen = 0;
-+ struct segment *segs =
-+ sizes_to_segments(&seglen, sizes, ARRAYSIZE(sizes));
-+ assert(segs[2].log == 3);
-+ assert(segs[2].start == 5);
-+ assert(segs[2].end == 6);
-+
-+ assert(segs[1].log == 2);
-+ assert(segs[1].start == 2);
-+ assert(segs[1].end == 5);
-+ free(segs);
-+}
-+
-+void test_suggest_compaction_segment(void)
-+{
-+ {
-+ uint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 };
-+ /* .................0 1 2 3 4 5 6 */
-+ struct segment min =
-+ suggest_compaction_segment(sizes, ARRAYSIZE(sizes));
-+ assert(min.start == 2);
-+ assert(min.end == 7);
-+ }
-+
-+ {
-+ uint64_t sizes[] = { 64, 32, 16, 8, 4, 2 };
-+ struct segment result =
-+ suggest_compaction_segment(sizes, ARRAYSIZE(sizes));
-+ assert(result.start == result.end);
-+ }
-+}
-+
-+void test_reflog_expire(void)
-+{
-+ char dir[256] = "/tmp/stack.test_reflog_expire.XXXXXX";
-+ assert(mkdtemp(dir));
-+ printf("%s\n", dir);
-+ char fn[256] = "";
-+ strcat(fn, dir);
-+ strcat(fn, "/refs");
-+
-+ struct write_options cfg = {};
-+ struct stack *st = NULL;
-+ int err = new_stack(&st, dir, fn, cfg);
-+ assert_err(err);
-+
-+ struct log_record logs[20] = {};
-+ int N = ARRAYSIZE(logs) - 1;
-+ int i = 0;
-+ for (i = 1; i <= N; i++) {
-+ char buf[256];
-+ snprintf(buf, sizeof(buf), "branch%02d", i);
-+
-+ logs[i].ref_name = strdup(buf);
-+ logs[i].update_index = i;
-+ logs[i].time = i;
-+ logs[i].new_hash = malloc(SHA1_SIZE);
-+ logs[i].email = strdup("identity@invalid");
-+ set_test_hash(logs[i].new_hash, i);
-+ }
-+
-+ for (i = 1; i <= N; i++) {
-+ int err = stack_add(st, &write_test_log, &logs[i]);
-+ assert_err(err);
-+ }
-+
-+ err = stack_compact_all(st, NULL);
-+ assert_err(err);
-+
-+ struct log_expiry_config expiry = {
-+ .time = 10,
-+ };
-+ err = stack_compact_all(st, &expiry);
-+ assert_err(err);
-+
-+ struct log_record log = {};
-+ err = stack_read_log(st, logs[9].ref_name, &log);
-+ assert(err == 1);
-+
-+ err = stack_read_log(st, logs[11].ref_name, &log);
-+ assert_err(err);
-+
-+ expiry.min_update_index = 15;
-+ err = stack_compact_all(st, &expiry);
-+ assert_err(err);
-+
-+ err = stack_read_log(st, logs[14].ref_name, &log);
-+ assert(err == 1);
-+
-+ err = stack_read_log(st, logs[16].ref_name, &log);
-+ assert_err(err);
-+
-+ /* cleanup */
-+ stack_destroy(st);
-+ for (i = 0; i < N; i++) {
-+ log_record_clear(&logs[i]);
-+ }
-+}
-+
-+int main()
-+{
-+ add_test_case("test_reflog_expire", test_reflog_expire);
-+ add_test_case("test_suggest_compaction_segment",
-+ &test_suggest_compaction_segment);
-+ add_test_case("test_sizes_to_segments", &test_sizes_to_segments);
-+ add_test_case("test_log2", &test_log2);
-+ add_test_case("test_parse_names", &test_parse_names);
-+ add_test_case("test_read_file", &test_read_file);
-+ add_test_case("test_names_equal", &test_names_equal);
-+ add_test_case("test_stack_add", &test_stack_add);
-+ test_main();
-+}
-
diff --git a/reftable/system.h b/reftable/system.h
new file mode 100644
--- /dev/null
@@ -7173,8 +5576,12 @@
+#include "config.h"
+
+#ifndef REFTABLE_STANDALONE
++
+#include "git-compat-util.h"
++#include <zlib.h>
++
+#else
++
+#include <assert.h>
+#include <errno.h>
+#include <fcntl.h>
@@ -7191,151 +5598,25 @@
+#define PRIuMAX "lu"
+#define PRIdMAX "ld"
+#define PRIxMAX "lx"
-+
-+#endif
-+
-+#endif
-
- diff --git a/reftable/test_framework.c b/reftable/test_framework.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/test_framework.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "test_framework.h"
-+
-+#include "system.h"
-+
-+#include "constants.h"
-+
-+struct test_case **test_cases;
-+int test_case_len;
-+int test_case_cap;
-+
-+struct test_case *new_test_case(const char *name, void (*testfunc)())
-+{
-+ struct test_case *tc = malloc(sizeof(struct test_case));
-+ tc->name = name;
-+ tc->testfunc = testfunc;
-+ return tc;
-+}
-+
-+struct test_case *add_test_case(const char *name, void (*testfunc)())
-+{
-+ struct test_case *tc = new_test_case(name, testfunc);
-+ if (test_case_len == test_case_cap) {
-+ test_case_cap = 2 * test_case_cap + 1;
-+ test_cases = realloc(test_cases,
-+ sizeof(struct test_case) * test_case_cap);
-+ }
-+
-+ test_cases[test_case_len++] = tc;
-+ return tc;
-+}
-+
-+void test_main()
-+{
-+ int i = 0;
-+ for (i = 0; i < test_case_len; i++) {
-+ printf("case %s\n", test_cases[i]->name);
-+ test_cases[i]->testfunc();
-+ }
-+}
-+
-+void set_test_hash(byte *p, int i)
-+{
-+ memset(p, (byte)i, SHA1_SIZE);
-+}
-+
-+void print_names(char **a)
-+{
-+ if (a == NULL || *a == NULL) {
-+ puts("[]");
-+ return;
-+ }
-+ puts("[");
-+ char **p = a;
-+ while (*p) {
-+ puts(*p);
-+ p++;
-+ }
-+ puts("]");
-+}
-
- diff --git a/reftable/test_framework.h b/reftable/test_framework.h
- new file mode 100644
- --- /dev/null
- +++ b/reftable/test_framework.h
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#ifndef TEST_FRAMEWORK_H
-+#define TEST_FRAMEWORK_H
-+
-+#include "system.h"
-+
-+#include "reftable.h"
-+
-+#ifdef NDEBUG
-+#undef NDEBUG
-+#endif
-+
-+#include "system.h"
-+
-+#ifdef assert
-+#undef assert
++#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])
++#define FREE_AND_NULL(x) \
++ do { \
++ free(x); \
++ (x) = NULL; \
++ } while (0)
++#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)
++#define SWAP(a, b) \
++ { \
++ char tmp[sizeof(a)]; \
++ assert(sizeof(a)==sizeof(b)); \
++ memcpy(&tmp[0], &a, sizeof(a)); \
++ memcpy(&a, &b, sizeof(a)); \
++ memcpy(&b, &tmp[0], sizeof(a)); \
++ }
+#endif
+
-+#define assert_err(c) \
-+ if (c != 0) { \
-+ fflush(stderr); \
-+ fflush(stdout); \
-+ fprintf(stderr, "%s: %d: error == %d (%s), want 0\n", \
-+ __FILE__, __LINE__, c, error_str(c)); \
-+ abort(); \
-+ }
-+
-+#define assert_streq(a, b) \
-+ if (strcmp(a, b)) { \
-+ fflush(stderr); \
-+ fflush(stdout); \
-+ fprintf(stderr, "%s:%d: %s (%s) != %s (%s)\n", __FILE__, \
-+ __LINE__, #a, a, #b, b); \
-+ abort(); \
-+ }
-+
-+#define assert(c) \
-+ if (!(c)) { \
-+ fflush(stderr); \
-+ fflush(stdout); \
-+ fprintf(stderr, "%s: %d: failed assertion %s\n", __FILE__, \
-+ __LINE__, #c); \
-+ abort(); \
-+ }
-+
-+struct test_case {
-+ const char *name;
-+ void (*testfunc)();
-+};
-+
-+struct test_case *new_test_case(const char *name, void (*testfunc)());
-+struct test_case *add_test_case(const char *name, void (*testfunc)());
-+void test_main();
-+
-+void set_test_hash(byte *p, int i);
++int uncompress_return_consumed(Bytef *dest, uLongf *destLen,
++ const Bytef *source, uLong *sourceLen);
+
+#endif
@@ -7441,73 +5722,6 @@
+
+#endif
- diff --git a/reftable/tree_test.c b/reftable/tree_test.c
- new file mode 100644
- --- /dev/null
- +++ b/reftable/tree_test.c
-@@
-+/*
-+Copyright 2020 Google LLC
-+
-+Use of this source code is governed by a BSD-style
-+license that can be found in the LICENSE file or at
-+https://developers.google.com/open-source/licenses/bsd
-+*/
-+
-+#include "tree.h"
-+
-+#include "basics.h"
-+#include "record.h"
-+#include "reftable.h"
-+#include "test_framework.h"
-+
-+static int test_compare(const void *a, const void *b)
-+{
-+ return a - b;
-+}
-+
-+struct curry {
-+ void *last;
-+};
-+
-+void check_increasing(void *arg, void *key)
-+{
-+ struct curry *c = (struct curry *)arg;
-+ if (c->last != NULL) {
-+ assert(test_compare(c->last, key) < 0);
-+ }
-+ c->last = key;
-+}
-+
-+void test_tree()
-+{
-+ struct tree_node *root = NULL;
-+
-+ void *values[11] = {};
-+ struct tree_node *nodes[11] = {};
-+ int i = 1;
-+ do {
-+ nodes[i] = tree_search(values + i, &root, &test_compare, 1);
-+ i = (i * 7) % 11;
-+ } while (i != 1);
-+
-+ for (i = 1; i < ARRAYSIZE(nodes); i++) {
-+ assert(values + i == nodes[i]->key);
-+ assert(nodes[i] ==
-+ tree_search(values + i, &root, &test_compare, 0));
-+ }
-+
-+ struct curry c = {};
-+ infix_walk(root, check_increasing, &c);
-+ tree_free(root);
-+}
-+
-+int main()
-+{
-+ add_test_case("test_tree", &test_tree);
-+ test_main();
-+}
-
diff --git a/reftable/writer.c b/reftable/writer.c
new file mode 100644
--- /dev/null
@@ -7764,7 +5978,7 @@
+{
+ int err = 0;
+ int i = 0;
-+ qsort(refs, n, sizeof(struct ref_record), ref_record_compare_name);
++ QSORT(refs, n, ref_record_compare_name);
+ for (i = 0; err == 0 && i < n; i++) {
+ err = writer_add_ref(w, &refs[i]);
+ }
@@ -7801,7 +6015,7 @@
+{
+ int err = 0;
+ int i = 0;
-+ qsort(logs, n, sizeof(struct log_record), log_record_compare_key);
++ QSORT(logs, n, log_record_compare_key);
+ for (i = 0; err == 0 && i < n; i++) {
+ err = writer_add_log(w, &logs[i]);
+ }
@@ -7948,8 +6162,7 @@
+{
+ struct obj_index_tree_node *entry = (struct obj_index_tree_node *)key;
+
-+ free(entry->offsets);
-+ entry->offsets = NULL;
++ FREE_AND_NULL(entry->offsets);
+ free(slice_yield(&entry->hash));
+ free(entry);
+}
@@ -8053,8 +6266,7 @@
+ free(slice_yield(&w->index[i].last_key));
+ }
+
-+ free(w->index);
-+ w->index = NULL;
++ FREE_AND_NULL(w->index);
+ w->index_len = 0;
+ w->index_cap = 0;
+}
@@ -8189,3 +6401,101 @@
+int writer_finish_public_section(struct writer *w);
+
+#endif
+
+ diff --git a/reftable/zlib-compat.c b/reftable/zlib-compat.c
+ new file mode 100644
+ --- /dev/null
+ +++ b/reftable/zlib-compat.c
+@@
++/* taken from zlib's uncompr.c
++
++ commit cacf7f1d4e3d44d871b605da3b647f07d718623f
++ Author: Mark Adler <[email protected]>
++ Date: Sun Jan 15 09:18:46 2017 -0800
++
++ zlib 1.2.11
++
++*/
++
++/*
++ * Copyright (C) 1995-2003, 2010, 2014, 2016 Jean-loup Gailly, Mark Adler
++ * For conditions of distribution and use, see copyright notice in zlib.h
++ */
++
++#include "system.h"
++
++/* clang-format off */
++
++/* ===========================================================================
++ Decompresses the source buffer into the destination buffer. *sourceLen is
++ the byte length of the source buffer. Upon entry, *destLen is the total size
++ of the destination buffer, which must be large enough to hold the entire
++ uncompressed data. (The size of the uncompressed data must have been saved
++ previously by the compressor and transmitted to the decompressor by some
++ mechanism outside the scope of this compression library.) Upon exit,
++ *destLen is the size of the decompressed data and *sourceLen is the number
++ of source bytes consumed. Upon return, source + *sourceLen points to the
++ first unused input byte.
++
++ uncompress returns Z_OK if success, Z_MEM_ERROR if there was not enough
++ memory, Z_BUF_ERROR if there was not enough room in the output buffer, or
++ Z_DATA_ERROR if the input data was corrupted, including if the input data is
++ an incomplete zlib stream.
++*/
++int ZEXPORT uncompress_return_consumed (
++ Bytef *dest,
++ uLongf *destLen,
++ const Bytef *source,
++ uLong *sourceLen) {
++ z_stream stream;
++ int err;
++ const uInt max = (uInt)-1;
++ uLong len, left;
++ Byte buf[1]; /* for detection of incomplete stream when *destLen == 0 */
++
++ len = *sourceLen;
++ if (*destLen) {
++ left = *destLen;
++ *destLen = 0;
++ }
++ else {
++ left = 1;
++ dest = buf;
++ }
++
++ stream.next_in = (z_const Bytef *)source;
++ stream.avail_in = 0;
++ stream.zalloc = (alloc_func)0;
++ stream.zfree = (free_func)0;
++ stream.opaque = (voidpf)0;
++
++ err = inflateInit(&stream);
++ if (err != Z_OK) return err;
++
++ stream.next_out = dest;
++ stream.avail_out = 0;
++
++ do {
++ if (stream.avail_out == 0) {
++ stream.avail_out = left > (uLong)max ? max : (uInt)left;
++ left -= stream.avail_out;
++ }
++ if (stream.avail_in == 0) {
++ stream.avail_in = len > (uLong)max ? max : (uInt)len;
++ len -= stream.avail_in;
++ }
++ err = inflate(&stream, Z_NO_FLUSH);
++ } while (err == Z_OK);
++
++ *sourceLen -= len + stream.avail_in;
++ if (dest != buf)
++ *destLen = stream.total_out;
++ else if (stream.total_out && err == Z_BUF_ERROR)
++ left = 1;
++
++ inflateEnd(&stream);
++ return err == Z_STREAM_END ? Z_OK :
++ err == Z_NEED_DICT ? Z_DATA_ERROR :
++ err == Z_BUF_ERROR && left + stream.avail_out ? Z_DATA_ERROR :
++ err;
++}
5: 721201269d ! 6: 0b2a1a81d6 Reftable support for git-core
@@ -6,12 +6,14 @@
TODO:
+ * Make CI on gitgitgadget compile.
+ * Redo interaction between repo config, repo storage version
* Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
Example use:
- $ ~/vc/git/git init
+ $ ~/vc/git/git init --reftable
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
@@ -36,7 +38,6 @@
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}
- Change-Id: I225ee6317b7911edf9aa95f43299f6c7c4511914
Signed-off-by: Han-Wen Nienhuys <[email protected]>
diff --git a/Makefile b/Makefile
@@ -84,6 +85,7 @@
+REFTABLE_OBJS += reftable/stack.o
+REFTABLE_OBJS += reftable/tree.o
+REFTABLE_OBJS += reftable/writer.o
++REFTABLE_OBJS += reftable/zlib-compat.o
+
+
TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS))
@@ -106,6 +108,114 @@ $^
Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
+ diff --git a/builtin/init-db.c b/builtin/init-db.c
+ --- a/builtin/init-db.c
+ +++ b/builtin/init-db.c
+@@
+ }
+
+ static int create_default_files(const char *template_path,
+- const char *original_git_dir)
++ const char *original_git_dir, int flags)
+ {
+ struct stat st1;
+ struct strbuf buf = STRBUF_INIT;
+@@
+ * Create the default symlink from ".git/HEAD" to the "master"
+ * branch, if it does not exist yet.
+ */
+- path = git_path_buf(&buf, "HEAD");
+- reinit = (!access(path, R_OK)
+- || readlink(path, junk, sizeof(junk)-1) != -1);
++ if (flags & INIT_DB_REFTABLE) {
++ reinit = 0; /* XXX - how do we recognize a reinit,
++ * and what should we do? */
++ } else {
++ path = git_path_buf(&buf, "HEAD");
++ reinit = (!access(path, R_OK) ||
++ readlink(path, junk, sizeof(junk) - 1) != -1);
++ }
++
+ if (!reinit) {
+ if (create_symref("HEAD", "refs/heads/master", NULL) < 0)
+ exit(1);
+ }
+
+ /* This forces creation of new config file */
+- xsnprintf(repo_version_string, sizeof(repo_version_string),
+- "%d", GIT_REPO_VERSION);
++ xsnprintf(repo_version_string, sizeof(repo_version_string), "%d",
++ flags & INIT_DB_REFTABLE ? GIT_REPO_VERSION_READ :
++ GIT_REPO_VERSION);
+ git_config_set("core.repositoryformatversion", repo_version_string);
+
+ /* Check filemode trustability */
+@@
+ */
+ check_repository_format();
+
+- reinit = create_default_files(template_dir, original_git_dir);
++ reinit = create_default_files(template_dir, original_git_dir, flags);
+
+ create_object_directory();
+
+@@
+ git_config_set("receive.denyNonFastforwards", "true");
+ }
+
++ if (flags & INIT_DB_REFTABLE) {
++ git_config_set("extensions.refStorage", "reftable");
++ }
++
+ if (!(flags & INIT_DB_QUIET)) {
+ int len = strlen(git_dir);
+
+@@
+ const char *template_dir = NULL;
+ unsigned int flags = 0;
+ const struct option init_db_options[] = {
+- OPT_STRING(0, "template", &template_dir, N_("template-directory"),
+- N_("directory from which templates will be used")),
++ OPT_STRING(0, "template", &template_dir,
++ N_("template-directory"),
++ N_("directory from which templates will be used")),
+ OPT_SET_INT(0, "bare", &is_bare_repository_cfg,
+- N_("create a bare repository"), 1),
++ N_("create a bare repository"), 1),
+ { OPTION_CALLBACK, 0, "shared", &init_shared_repository,
+- N_("permissions"),
+- N_("specify that the git repository is to be shared amongst several users"),
+- PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0},
++ N_("permissions"),
++ N_("specify that the git repository is to be shared amongst several users"),
++ PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },
+ OPT_BIT('q', "quiet", &flags, N_("be quiet"), INIT_DB_QUIET),
++ OPT_BIT(0, "reftable", &flags, N_("use reftable"),
++ INIT_DB_REFTABLE),
+ OPT_STRING(0, "separate-git-dir", &real_git_dir, N_("gitdir"),
+ N_("separate git dir from working tree")),
+ OPT_END()
+
+ diff --git a/cache.h b/cache.h
+ --- a/cache.h
+ +++ b/cache.h
+@@
+
+ #define INIT_DB_QUIET 0x0001
+ #define INIT_DB_EXIST_OK 0x0002
++#define INIT_DB_REFTABLE 0x0004
+
+ int init_db(const char *git_dir, const char *real_git_dir,
+ const char *template_dir, unsigned int flags);
+@@
+ int is_bare;
+ int hash_algo;
+ char *work_tree;
++ char *ref_storage;
+ struct string_list unknown_extensions;
+ };
+
+
diff --git a/refs.c b/refs.c
--- a/refs.c
+++ b/refs.c
@@ -119,43 +229,65 @@ $^
static struct ref_storage_be *find_ref_storage_backend(const char *name)
{
@@
- static struct ref_store *ref_store_init(const char *gitdir,
+ * Create, record, and return a ref_store instance for the specified
+ * gitdir.
+ */
+-static struct ref_store *ref_store_init(const char *gitdir,
++static struct ref_store *ref_store_init(const char *gitdir, const char *be_name,
unsigned int flags)
{
- const char *be_name = "files";
- struct ref_storage_be *be = find_ref_storage_backend(be_name);
-+ struct strbuf refs_path = STRBUF_INIT;
-+
-+ /* XXX this should probably come from a git config setting and not
-+ default to reftable. */
-+ const char *be_name = "reftable";
+ struct ref_storage_be *be;
struct ref_store *refs;
-+ strbuf_addstr(&refs_path, gitdir);
-+ strbuf_addstr(&refs_path, "/refs");
-+
-+ if (!is_directory(refs_path.buf)) {
-+ be_name = "reftable";
-+ }
-+ strbuf_release(&refs_path);
+ be = find_ref_storage_backend(be_name);
if (!be)
BUG("reference backend %s is unknown", be_name);
-
- diff --git a/refs.h b/refs.h
- --- a/refs.h
- +++ b/refs.h
@@
- const char *refname,
- each_reflog_ent_fn fn,
- void *cb_data);
-+
-+/* XXX which ordering are these? Newest or oldest first? */
- int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data);
- int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data);
+ if (!r->gitdir)
+ BUG("attempting to get main_ref_store outside of repository");
+- r->refs = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
++ r->refs = ref_store_init(r->gitdir,
++ /* XXX r->ref_storage_format == NULL. Where
++ * should the config file be parsed out? */
++ r->ref_storage_format ? r->ref_storage_format :
++ "reftable",
++ REF_STORE_ALL_CAPS);
+ return r->refs;
+ }
+
+@@
+ goto done;
+
+ /* assume that add_submodule_odb() has been called */
+- refs = ref_store_init(submodule_sb.buf,
++ refs = ref_store_init(submodule_sb.buf, "files", /* XXX */
+ REF_STORE_READ | REF_STORE_ODB);
+ register_ref_store_map(&submodule_ref_stores, "submodule",
+ refs, submodule);
+@@
+
+ struct ref_store *get_worktree_ref_store(const struct worktree *wt)
+ {
++ const char *format = "files"; /* XXX */
+ struct ref_store *refs;
+ const char *id;
+
+@@
+
+ if (wt->id)
+ refs = ref_store_init(git_common_path("worktrees/%s", wt->id),
+- REF_STORE_ALL_CAPS);
++ format, REF_STORE_ALL_CAPS);
+ else
+- refs = ref_store_init(get_git_common_dir(),
++ refs = ref_store_init(get_git_common_dir(), format,
+ REF_STORE_ALL_CAPS);
+
+ if (refs)
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
--- a/refs/refs-internal.h
@@ -184,6 +316,11 @@ $^
+
+#include "../reftable/reftable.h"
+
++/*
++ The reftable v1 spec only supports 20-byte binary SHA1s. A new format version
++ will be needed to support SHA256.
++ */
++
+extern struct ref_storage_be refs_be_reftable;
+
+struct reftable_ref_store {
@@ -244,16 +381,28 @@ $^
+ base_ref_store_init(ref_store, &refs_be_reftable);
+ refs->store_flags = store_flags;
+
-+ strbuf_addf(&sb, "%s/refs", path);
-+ refs->table_list_file = xstrdup(sb.buf);
-+ strbuf_reset(&sb);
+ strbuf_addf(&sb, "%s/reftable", path);
+ refs->reftable_dir = xstrdup(sb.buf);
-+ strbuf_release(&sb);
++ strbuf_reset(&sb);
++
++ strbuf_addf(&sb, "%s/reftable/tables.list", path);
++ refs->table_list_file = xstrdup(sb.buf);
++ strbuf_reset(&sb);
++
++ strbuf_addf(&sb, "%s/refs", path);
++ safe_create_dir(sb.buf, 1);
++ strbuf_reset(&sb);
++
++ strbuf_addf(&sb, "%s/HEAD", path);
++ write_file(sb.buf, "ref: refs/.invalid");
++ strbuf_reset(&sb);
++
++ strbuf_addf(&sb, "%s/refs/heads", path);
++ write_file(sb.buf, "this repository uses the reftable format");
+
+ refs->err = new_stack(&refs->stack, refs->reftable_dir,
+ refs->table_list_file, cfg);
-+
++ strbuf_release(&sb);
+ return ref_store;
+}
+
@@ -261,13 +410,13 @@ $^
+{
+ struct reftable_ref_store *refs =
+ (struct reftable_ref_store *)ref_store;
++
++ safe_create_dir(refs->reftable_dir, 1);
+ FILE *f = fopen(refs->table_list_file, "a");
+ if (f == NULL) {
+ return -1;
+ }
+ fclose(f);
-+
-+ safe_create_dir(refs->reftable_dir, 1);
+ return 0;
+}
+
@@ -277,31 +426,33 @@ $^
+ struct ref_record ref;
+ struct object_id oid;
+ struct ref_store *ref_store;
++ unsigned int flags;
++ int err;
+ char *prefix;
+};
+
+static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
+{
-+ while (1) {
-+ struct reftable_iterator *ri =
-+ (struct reftable_iterator *)ref_iterator;
-+ int err = iterator_next_ref(ri->iter, &ri->ref);
-+ if (err > 0) {
-+ return ITER_DONE;
-+ }
-+ if (err < 0) {
-+ return ITER_ERROR;
++ struct reftable_iterator *ri = (struct reftable_iterator *)ref_iterator;
++ while (ri->err == 0) {
++ ri->err = iterator_next_ref(ri->iter, &ri->ref);
++ if (ri->err) {
++ break;
+ }
+
+ ri->base.refname = ri->ref.ref_name;
+ if (ri->prefix != NULL &&
+ strncmp(ri->prefix, ri->ref.ref_name, strlen(ri->prefix))) {
-+ return ITER_DONE;
++ ri->err = 1;
++ break;
+ }
++ if (ri->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&
++ ref_type(ri->base.refname) != REF_TYPE_PER_WORKTREE)
++ continue;
+
+ ri->base.flags = 0;
+ if (ri->ref.value != NULL) {
-+ memcpy(ri->oid.hash, ri->ref.value, GIT_SHA1_RAWSZ);
++ hashcpy(ri->oid.hash, ri->ref.value);
+ } else if (ri->ref.target != NULL) {
+ int out_flags = 0;
+ const char *resolved = refs_resolve_ref_unsafe(
@@ -309,14 +460,30 @@ $^
+ RESOLVE_REF_READING, &ri->oid, &out_flags);
+ ri->base.flags = out_flags;
+ if (resolved == NULL &&
-+ !(ri->base.flags & DO_FOR_EACH_INCLUDE_BROKEN) &&
++ !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&
+ (ri->base.flags & REF_ISBROKEN)) {
+ continue;
+ }
+ }
++
+ ri->base.oid = &ri->oid;
-+ return ITER_OK;
++ if (!(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&
++ !ref_resolves_to_object(ri->base.refname, ri->base.oid,
++ ri->base.flags)) {
++ continue;
++ }
++
++ break;
++ }
++
++ if (ri->err > 0) {
++ return ITER_DONE;
+ }
++ if (ri->err < 0) {
++ return ITER_ERROR;
++ }
++
++ return ITER_OK;
+}
+
+static int reftable_ref_iterator_peel(struct ref_iterator *ref_iterator,
@@ -324,7 +491,7 @@ $^
+{
+ struct reftable_iterator *ri = (struct reftable_iterator *)ref_iterator;
+ if (ri->ref.target_value != NULL) {
-+ memcpy(peeled->hash, ri->ref.target_value, GIT_SHA1_RAWSZ);
++ hashcpy(peeled->hash, ri->ref.target_value);
+ return 0;
+ }
+
@@ -352,20 +519,15 @@ $^
+ (struct reftable_ref_store *)ref_store;
+ struct reftable_iterator *ri = xcalloc(1, sizeof(*ri));
+ struct merged_table *mt = NULL;
-+ int err = 0;
-+ if (refs->err) {
-+ /* how to propagate errors? */
-+ return NULL;
-+ }
+
+ mt = stack_merged_table(refs->stack);
-+
-+ /* XXX something with flags? */
-+ err = merged_table_seek_ref(mt, &ri->iter, prefix);
-+ /* XXX what to do with err? */
-+ assert(err == 0);
++ ri->err = refs->err;
++ if (ri->err == 0) {
++ ri->err = merged_table_seek_ref(mt, &ri->iter, prefix);
++ }
+ base_ref_iterator_init(&ri->base, &reftable_ref_iterator_vtable, 1);
+ ri->base.oid = &ri->oid;
++ ri->flags = flags;
+ ri->ref_store = ref_store;
+ return &ri->base;
+}
@@ -387,12 +549,6 @@ $^
+ return 0;
+}
+
-+static int ref_update_cmp(const void *a, const void *b)
-+{
-+ return strcmp(((struct ref_update *)a)->refname,
-+ ((struct ref_update *)b)->refname);
-+}
-+
+static int reftable_check_old_oid(struct ref_store *refs, const char *refname,
+ struct object_id *want_oid)
+{
@@ -411,6 +567,12 @@ $^
+ return 0;
+}
+
++static int ref_update_cmp(const void *a, const void *b)
++{
++ return strcmp(((struct ref_update *)a)->refname,
++ ((struct ref_update *)b)->refname);
++}
++
+static int write_transaction_table(struct writer *writer, void *arg)
+{
+ struct ref_transaction *transaction = (struct ref_transaction *)arg;
@@ -418,13 +580,14 @@ $^
+ (struct reftable_ref_store *)transaction->ref_store;
+ uint64_t ts = stack_next_update_index(refs->stack);
+ int err = 0;
-+ /* XXX - are we allowed to mutate the input data? */
-+ qsort(transaction->updates, transaction->nr,
-+ sizeof(struct ref_update *), ref_update_cmp);
++ struct ref_update **sorted =
++ malloc(transaction->nr * sizeof(struct ref_update *));
++ COPY_ARRAY(sorted, transaction->updates, transaction->nr);
++ QSORT(sorted, transaction->nr, ref_update_cmp);
+ writer_set_limits(writer, ts, ts);
+
+ for (int i = 0; i < transaction->nr; i++) {
-+ struct ref_update *u = transaction->updates[i];
++ struct ref_update *u = sorted[i];
+ if (u->flags & REF_HAVE_OLD) {
+ err = reftable_check_old_oid(transaction->ref_store,
+ u->refname, &u->old_oid);
@@ -435,11 +598,12 @@ $^
+ }
+
+ for (int i = 0; i < transaction->nr; i++) {
-+ struct ref_update *u = transaction->updates[i];
++ struct ref_update *u = sorted[i];
+ if (u->flags & REF_HAVE_NEW) {
+ struct object_id out_oid = {};
+ int out_flags = 0;
-+ /* XXX who owns the memory here? */
++ /* Memory owned by refs_resolve_ref_unsafe, no need to
++ * free(). */
+ const char *resolved = refs_resolve_ref_unsafe(
+ transaction->ref_store, u->refname, 0, &out_oid,
+ &out_flags);
@@ -456,7 +620,7 @@ $^
+ }
+
+ for (int i = 0; i < transaction->nr; i++) {
-+ struct ref_update *u = transaction->updates[i];
++ struct ref_update *u = sorted[i];
+ struct log_record log = {};
+ fill_log_record(&log);
+
@@ -473,6 +637,7 @@ $^
+ }
+ }
+exit:
++ free(sorted);
+ return err;
+}
+
@@ -527,16 +692,21 @@ $^
+
+ for (int i = 0; i < arg->refnames->nr; i++) {
+ struct log_record log = {};
++ struct ref_record current = {};
+ fill_log_record(&log);
+ log.message = xstrdup(arg->logmsg);
+ log.new_hash = NULL;
-+
-+ /* XXX should lookup old oid. */
+ log.old_hash = NULL;
+ log.update_index = ts;
+ log.ref_name = (char *)arg->refnames->items[i].string;
+
++ if (stack_read_ref(arg->stack, log.ref_name, ¤t) == 0) {
++ log.old_hash = current.value;
++ }
+ err = writer_add_log(writer, &log);
++ log.old_hash = NULL;
++ ref_record_clear(¤t);
++
+ clear_log_record(&log);
+ if (err < 0) {
+ return err;
@@ -564,12 +734,11 @@ $^
+{
+ struct reftable_ref_store *refs =
+ (struct reftable_ref_store *)ref_store;
-+ /* XXX reflog expiry. */
+ return stack_compact_all(refs->stack, NULL);
+}
+
+struct write_create_symref_arg {
-+ struct stack *stack;
++ struct reftable_ref_store *refs;
+ const char *refname;
+ const char *target;
+ const char *logmsg;
@@ -579,7 +748,7 @@ $^
+{
+ struct write_create_symref_arg *create =
+ (struct write_create_symref_arg *)arg;
-+ uint64_t ts = stack_next_update_index(create->stack);
++ uint64_t ts = stack_next_update_index(create->refs->stack);
+ int err = 0;
+
+ struct ref_record ref = {
@@ -593,8 +762,36 @@ $^
+ return err;
+ }
+
-+ /* XXX reflog? */
++ {
++ struct log_record log = {};
++ struct object_id new_oid = {};
++ struct object_id old_oid = {};
++ struct ref_record current = {};
++ stack_read_ref(create->refs->stack, create->refname, ¤t);
++
++ fill_log_record(&log);
++ log.ref_name = current.ref_name;
++ if (refs_resolve_ref_unsafe(
++ (struct ref_store *)create->refs, create->refname,
++ RESOLVE_REF_READING, &old_oid, NULL) != NULL) {
++ log.old_hash = old_oid.hash;
++ }
++
++ /* XXX should the resolution be done relative or absolute? */
++ if (refs_resolve_ref_unsafe((struct ref_store *)create->refs,
++ create->target, RESOLVE_REF_READING,
++ &new_oid, NULL) != NULL) {
++ log.new_hash = new_oid.hash;
++ }
+
++ if (log.old_hash != NULL || log.new_hash != NULL) {
++ writer_add_log(writer, &log);
++ }
++ log.ref_name = NULL;
++ log.old_hash = NULL;
++ log.new_hash = NULL;
++ clear_log_record(&log);
++ }
+ return 0;
+}
+
@@ -604,7 +801,7 @@ $^
+{
+ struct reftable_ref_store *refs =
+ (struct reftable_ref_store *)ref_store;
-+ struct write_create_symref_arg arg = { .stack = refs->stack,
++ struct write_create_symref_arg arg = { .refs = refs,
+ .refname = refname,
+ .target = target,
+ .logmsg = logmsg };
@@ -628,7 +825,11 @@ $^
+ goto exit;
+ }
+
-+ /* XXX should check that dest doesn't exist? */
++ /* XXX do ref renames overwrite the target? */
++ if (stack_read_ref(arg->stack, arg->newname, &ref) == 0) {
++ goto exit;
++ }
++
+ free(ref.ref_name);
+ ref.ref_name = strdup(arg->newname);
+ writer_set_limits(writer, ts, ts);
@@ -638,6 +839,7 @@ $^
+ struct ref_record todo[2] = {};
+ todo[0].ref_name = (char *)arg->oldname;
+ todo[0].update_index = ts;
++ /* leave todo[0] empty */
+ todo[1] = ref;
+ todo[1].update_index = ts;
+
@@ -735,8 +937,7 @@ $^
+
+ free(ri->last_name);
+ ri->last_name = xstrdup(ri->log.ref_name);
-+ /* XXX const? */
-+ memcpy(&ri->oid.hash, ri->log.new_hash, GIT_SHA1_RAWSZ);
++ hashcpy(ri->oid.hash, ri->log.new_hash);
+ return ITER_OK;
+ }
+}
@@ -808,12 +1009,16 @@ $^
+ {
+ struct object_id old_oid = {};
+ struct object_id new_oid = {};
++ const char *full_committer = "";
+
-+ memcpy(&old_oid.hash, log.old_hash, GIT_SHA1_RAWSZ);
-+ memcpy(&new_oid.hash, log.new_hash, GIT_SHA1_RAWSZ);
++ hashcpy(old_oid.hash, log.old_hash);
++ hashcpy(new_oid.hash, log.new_hash);
+
-+ /* XXX committer = email? name? */
-+ if (fn(&old_oid, &new_oid, log.name, log.time,
++ full_committer = fmt_ident(log.name, log.email,
++ WANT_COMMITTER_IDENT,
++ /*date*/ NULL,
++ IDENT_NO_DATE);
++ if (fn(&old_oid, &new_oid, full_committer, log.time,
+ log.tz_offset, log.message, cb_data)) {
+ err = -1;
+ break;
@@ -844,7 +1049,6 @@ $^
+ int cap = 0;
+ int len = 0;
+
-+ printf("oldest first\n");
+ while (err == 0) {
+ struct log_record log = {};
+ err = iterator_next_log(it, &log);
@@ -868,12 +1072,15 @@ $^
+ struct log_record *log = &logs[i];
+ struct object_id old_oid = {};
+ struct object_id new_oid = {};
++ const char *full_committer = "";
+
-+ memcpy(&old_oid.hash, log->old_hash, GIT_SHA1_RAWSZ);
-+ memcpy(&new_oid.hash, log->new_hash, GIT_SHA1_RAWSZ);
++ hashcpy(old_oid.hash, log->old_hash);
++ hashcpy(new_oid.hash, log->new_hash);
+
-+ /* XXX committer = email? name? */
-+ if (!fn(&old_oid, &new_oid, log->name, log->time,
++ full_committer = fmt_ident(log->name, log->email,
++ WANT_COMMITTER_IDENT, NULL,
++ IDENT_NO_DATE);
++ if (!fn(&old_oid, &new_oid, full_committer, log->time,
+ log->tz_offset, log->message, cb_data)) {
+ err = -1;
+ break;
@@ -921,18 +1128,11 @@ $^
+ reflog_expiry_cleanup_fn cleanup_fn,
+ void *policy_cb_data)
+{
++ BUG("per ref reflog expiry is not implemented for reftable backend.");
+ /*
-+ XXX
-+
-+ This doesn't fit with the reftable API. If we are expiring for space
-+ reasons, the expiry should be combined with a compaction, and there
-+ should be a policy that can be called for all refnames, not for a
-+ single ref name.
-+
-+ If this is for cleaning up individual entries, we'll have to write
-+ extra data to create tombstones.
++ TODO: should iterate over the reflog, and write tombstones. The
++ tombstones will be removed on compaction.
+ */
-+ return 0;
+}
+
+static int reftable_read_raw_ref(struct ref_store *ref_store,
@@ -947,13 +1147,13 @@ $^
+ goto exit;
+ }
+ if (ref.target != NULL) {
++ /* XXX recurse? */
+ strbuf_reset(referent);
+ strbuf_addstr(referent, ref.target);
+ *type |= REF_ISSYMREF;
+ } else {
-+ memcpy(oid->hash, ref.value, GIT_SHA1_RAWSZ);
++ hashcpy(oid->hash, ref.value);
+ }
-+
+exit:
+ ref_record_clear(&ref);
+ return err;
@@ -986,3 +1186,58 @@ $^
+ reftable_delete_reflog,
+ reftable_reflog_expire
+};
+
+ diff --git a/repository.c b/repository.c
+ --- a/repository.c
+ +++ b/repository.c
+@@
+ if (worktree)
+ repo_set_worktree(repo, worktree);
+
++ repo->ref_storage_format = format.ref_storage != NULL ?
++ xstrdup(format.ref_storage) :
++ "files"; /* XXX */
++
+ clear_repository_format(&format);
+ return 0;
+
+
+ diff --git a/repository.h b/repository.h
+ --- a/repository.h
+ +++ b/repository.h
+@@
+ /* The store in which the refs are held. */
+ struct ref_store *refs;
+
++ /* The format to use for the ref database. */
++ char *ref_storage_format;
++
+ /*
+ * Contains path to often used file names.
+ */
+
+ diff --git a/setup.c b/setup.c
+ --- a/setup.c
+ +++ b/setup.c
+@@
+ if (!value)
+ return config_error_nonbool(var);
+ data->partial_clone = xstrdup(value);
+- } else if (!strcmp(ext, "worktreeconfig"))
++ } else if (!strcmp(ext, "worktreeconfig")) {
+ data->worktree_config = git_config_bool(var, value);
+- else
++ } else if (!strcmp(ext, "refStorage")) {
++ data->ref_storage = xstrdup(value);
++ } else
+ string_list_append(&data->unknown_extensions, ext);
+ }
+
+@@
+ string_list_clear(&format->unknown_extensions, 0);
+ free(format->work_tree);
+ free(format->partial_clone);
++ free(format->ref_storage);
+ init_repository_format(format);
+ }
+ Signature · 1 line
gitgitgadget
PATCH v3 1/6 refs.h: clarify reflog iteration order
From: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
refs.h +4 -1
@@ -444,18 +444,21 @@ int delete_refs(const char *msg, struct string_list *refnames,int refs_delete_reflog (struct ref_store *refs,const char *refname);int delete_reflog (const char *refname); -/* iterate over reflog entries */ +/* Iterate over reflog entries. */ typedef int each_reflog_ent_fn (struct object_id *old_oid,struct object_id *new_oid,const char *committer,timestamp_t timestamp,int tz,const char *msg,void *cb_data); +/* Iterate in over reflog entries, oldest entry first. */ int refs_for_each_reflog_ent (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data);int refs_for_each_reflog_ent_reverse (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data); + +/* Call a function for each reflog entry, oldest entry first. */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
Signature · 1 line
gitgitgadget
PATCH v3 2/6 setup.c: enable repo detection for reftable
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
* only check R_OK for .git/refs.
In the reftable format, tables are stored under $GITDIR/reftable/ and the list of tables is managed in a file $GITDIR/refs. Generally, this file does not have the +x bit set
* allow missing HEAD if there is a reftable/
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- setup.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
setup.c +14 -6
@@ -304,22 +304,23 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)* - either an objects/ directory _or_ the proper * GIT_OBJECT_DIRECTORY environment variable * - a refs/ directory -* - either a HEAD symlink or a HEAD file that is formatted as -* a proper "ref:", or a regular file HEAD that has a properly -* formatted sha1 object name. +* - a reftable/ director or, either a HEAD ref. +* The HEAD ref should be a HEAD symlink or a HEAD file that is formatted as +* a proper "ref:", or a regular file HEAD that has a properly formatted sha1 +* object name. */ int is_git_directory (const char *suspect) {struct strbuf path =STRBUF_INIT ;int ret =0 ; +int have_head =0 ;size_t len;/* Check worktree-related signatures */ strbuf_addstr (&path, suspect);strbuf_complete (&path,'/' );strbuf_addstr (&path,"HEAD" ); -if (validate_headref (path.buf)) - goto done; + have_head = !validate_headref (path.buf);strbuf_reset (&path);get_common_dir (&path, suspect);
@@ -339,9 +340,16 @@ int is_git_directory(const char *suspect)strbuf_setlen (&path, len);strbuf_addstr (&path,"/refs" ); -if (access (path.buf,X_OK )) +if (access (path.buf,R_OK )) goto done; +if (!have_head) { +strbuf_setlen (&path, len); +strbuf_addstr (&path,"/reftable" ); +if (access (path.buf,X_OK )) + goto done; + } + ret =1 ; done:strbuf_release (&path);
Signature · 1 line
gitgitgadget
Re: [PATCH v3 2/6] setup.c: enable repo detection for reftable
Quoted 4 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
* only check
R_OKfor .git/refs.
this patch can go now.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
PATCH v3 3/6 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- builtin/init-db.c | 2 -- refs/files-backend.c | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +5 -0
@@ -3158,6 +3158,11 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */
Signature · 1 line
gitgitgadget
Re: [PATCH v3 3/6] create .git/refs in files-backend.c
Quoted 29 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
Signed-off-by Han-Wen Nienhuys <[email protected]>--- builtin/init-db.c | 2 -- refs/files-backend.c | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)builtin/init-db.c +0
-2@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);refs/files-backend.c +0
-0
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3161
Quoted 4 lines
refs/files-backend.c +1
-0@@ -3158,6 +3158,11 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" );
Have you run "git diff --check" before committing?
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3163
Quoted 2 lines
refs/files-backend.c +2
-0+safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */
whitespace error here, but this comment is really appreciated. It wasn't immediately clear why we do this again in the original.
Quoted 5 lines
refs/files-backend.c +2
-0+adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */
Re: [PATCH v3 3/6] create .git/refs in files-backend.c
Quoted 3 lines · Junio C Hamano
Quoted 1 line
+
files_ref_path(refs, &sb, "refs");Have you run "git diff --check" before committing?
let me install the clang-format pre-commit hook.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v3 3/6] create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetQuoted 4 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
This breaks 'git init', and, consequently, the whole test suite:
$ ./git init /tmp/foo /tmp/foo/.git/refs/tmp/foo/.git/refs/heads: No such file or directory
Quoted 36 lines
builtin/init-db.c | 2 -- refs/files-backend.c | 5 +++++ 2 files changed, 5 insertions(+), 2 deletions(-)builtin/init-db.c +0
-2@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);refs/files-backend.c +5
-0@@ -3158,6 +3158,11 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */ Signature · 1 line
gitgitgadget
Re: [PATCH v3 3/6] create .git/refs in files-backend.c
Quoted 10 lines · SZEDER Gábor
Quoted 4 lines · Han-Wen Nienhuys via GitGitGadget
From: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
This breaks 'git init', and, consequently, the whole test suite:
$ ./git init /tmp/foo /tmp/foo/.git/refs/tmp/foo/.git/refs/heads: No such file or directory
Yeah, this is one of the fixes in the patch I sent earlier in the thread. The issue is here:
Quoted 4 lines
Quoted 4 lines
refs/files-backend.c +0
-0@@ -3158,6 +3158,11 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ;
Quoted 8 lines
refs/files-backend.c static int files_init_db(struct ref_store *ref_store, struct strbuf *err) · line 3168
Quoted 8 lines
refs/files-backend.c +5
-0+files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */
Right after this context, we call files_ref_path() with "sb" again. There needs to be a strbuf_reset() beforehand.
-Peff
PATCH v3 4/6 refs: document how ref_iterator_advance_fn should handle symrefs
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 5 +++++ 1 file changed, 5 insertions(+)
refs/refs-internal.h +5 -0
@@ -438,6 +438,11 @@ void base_ref_iterator_free(struct ref_iterator *iter);/* Virtual function declarations for ref_iterators: */ +/* +* backend-specific implementation of ref_iterator_advance. +* For symrefs, the function should set REF_ISSYMREF, and it should also dereference +* the symref to provide the OID referent. +*/ typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator); typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,
Signature · 1 line
gitgitgadget
PATCH v3 5/6 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Reftable is a new format for storing the ref database. It provides the following benefits:
* Simple and fast atomic ref transactions, including multiple refs and reflogs.
* Compact storage of ref data.
* Fast look ups of ref data.
* Case-sensitive ref names on Windows/OSX, regardless of file system
* Eliminates file/directory conflicts in ref names
Further context and motivation can be found in background reading:
* Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
* Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
* First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
* Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
* First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
* libgit2 support issue: https://github.com/libgit2/libgit2/issues
* GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 401 +++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 +++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 286 +++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 708 +++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 ++++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 399 +++++++++++++++ reftable/slice.c | 199 ++++++++ reftable/slice.h | 39 ++ reftable/stack.c | 983 +++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 57 +++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 622 ++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ 34 files changed, 6268 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
reftable/LICENSE new file +31 -0
@@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
reftable/README.md new file +19 -0
@@ -0,0 +1,19 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + echo '/* empty */' > reftable/config.h + rm reftable/*_test.c reftable/test_framework.* + git add reftable/*.[ch] + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +5 -0
@@ -0,0 +1,5 @@ +commit e54326f73d95bfe8b17f264c400f4c365dbd5e5e +Author: Han-Wen Nienhuys <[email protected]> +Date: Tue Feb 4 19:48:17 2020 +0100 + + C: PRI?MAX use; clang-format.
reftable/basics.c new file +196 -0
196 lines not on this page · raw message
reftable/basics.h new file +37 -0
@@ -0,0 +1,37 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +401 -0
401 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/config.h new file +1 -0
@@ -0,0 +1 @@ +/* empty */
reftable/constants.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +97 -0
97 lines not on this page · raw message
reftable/file.c new file +97 -0
97 lines not on this page · raw message
reftable/iter.c new file +229 -0
229 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +286 -0
286 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/pq.c new file +114 -0
114 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +708 -0
708 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1107 -0
485 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/reftable.h new file +399 -0
399 lines not on this page · raw message
reftable/slice.c new file +199 -0
199 lines not on this page · raw message
reftable/stack.c new file +983 -0
983 lines not on this page · raw message
reftable/system.h new file +57 -0
57 lines not on this page · raw message
reftable/tree.c new file +66 -0
66 lines not on this page · raw message
reftable/writer.c new file +622 -0
622 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
reftable/zlib-compat.c new file +92 -0
92 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
PATCH v3 6/6 Reftable support for git-core
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
For background, see the previous commit introducing the library.
TODO:
* Make CI on gitgitgadget compile.
* Redo interaction between repo config, repo storage version
* Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
Example use:
$ ~/vc/git/git init --reftable
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
$ ~/vc/git/git add a
$ ~/vc/git/git commit -mx
fatal: not a git repository (or any of the parent directories): .git
[master (root-commit) 373d969] x
1 file changed, 1 insertion(+)
create mode 100644 a
$ ~/vc/git/git show-ref
373d96972fca9b63595740bba3898a762778ba20 HEAD
373d96972fca9b63595740bba3898a762778ba20 refs/heads/master
$ ls -l .git/reftable/
total 12
-rw------- 1 hanwen primarygroup 126 Jan 23 20:08 000000000001-000000000001.ref
-rw------- 1 hanwen primarygroup 4277 Jan 23 20:08 000000000002-000000000002.ref
$ go run ~/vc/reftable/cmd/dump.go -table /tmp/qz/.git/reftable/000000000002-000000000002.ref
** DEBUG **
name /tmp/qz/.git/reftable/000000000002-000000000002.ref, sz 4209: 'r' reftable.readerOffsets{Present:true, Offset:0x0, IndexOffset:0x0}, 'o' reftable.readerOffsets{Present:false, Offset:0x0, IndexOffset:0x0} 'g' reftable.readerOffsets{Present:true, Offset:0x1000, IndexOffset:0x0}
** REFS **
reftable.RefRecord{RefName:"refs/heads/master", UpdateIndex:0x2, Value:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, TargetValue:[]uint8(nil), Target:""}
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}Signed-off-by Han-Wen Nienhuys <[email protected]>
--- Makefile | 24 +- builtin/init-db.c | 40 +- cache.h | 2 + refs.c | 22 +- refs/refs-internal.h | 1 + refs/reftable-backend.c | 880 ++++++++++++++++++++++++++++++++++++++++ repository.c | 4 + repository.h | 3 + setup.c | 7 +- 9 files changed, 959 insertions(+), 24 deletions(-) create mode 100644 refs/reftable-backend.c
Makefile +23 -1
@@ -814,6 +814,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += command-list.h
@@ -959,6 +960,7 @@ LIB_OBJS += rebase-interactive.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
@@ -1163,7 +1165,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION)
@@ -2347,11 +2349,28 @@ VCSSVN_OBJS += vcs-svn/fast_export.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o +REFTABLE_OBJS += reftable/zlib-compat.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL
@@ -2488,6 +2507,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
builtin/init-db.c +27 -13
@@ -177,7 +177,7 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree) }static int create_default_files (const char *template_path, -const char *original_git_dir) +const char *original_git_dir,int flags) {struct stat st1;struct strbuf buf =STRBUF_INIT ;
@@ -234,17 +234,24 @@ static int create_default_files(const char *template_path,* Create the default symlink from ".git/HEAD" to the "master" * branch, if it does not exist yet. */ - path =git_path_buf (&buf,"HEAD" ); - reinit = (!access (path,R_OK ) - ||readlink (path, junk,sizeof (junk)-1 ) !=-1 ); +if (flags &INIT_DB_REFTABLE ) { + reinit =0 ;/* XXX - how do we recognize a reinit, +* and what should we do? */ + }else { + path =git_path_buf (&buf,"HEAD" ); + reinit = (!access (path,R_OK ) || +readlink (path, junk,sizeof (junk) -1 ) !=-1 ); + } +if (!reinit) {if (create_symref ("HEAD" ,"refs/heads/master" ,NULL ) <0 )exit (1 ); }/* This forces creation of new config file */ -xsnprintf (repo_version_string,sizeof (repo_version_string), -"%d" ,GIT_REPO_VERSION ); +xsnprintf (repo_version_string,sizeof (repo_version_string),"%d" , + flags &INIT_DB_REFTABLE ?GIT_REPO_VERSION_READ : +GIT_REPO_VERSION );git_config_set ("core.repositoryformatversion" , repo_version_string);/* Check filemode trustability */
@@ -378,7 +385,7 @@ int init_db(const char *git_dir, const char *real_git_dir,*/ check_repository_format (); - reinit =create_default_files (template_dir, original_git_dir); + reinit =create_default_files (template_dir, original_git_dir, flags);create_object_directory ();
@@ -403,6 +410,10 @@ int init_db(const char *git_dir, const char *real_git_dir,git_config_set ("receive.denyNonFastforwards" ,"true" ); } +if (flags &INIT_DB_REFTABLE ) { +git_config_set ("extensions.refStorage" ,"reftable" ); + } +if (!(flags &INIT_DB_QUIET )) {int len =strlen (git_dir);
@@ -481,15 +492,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)const char *template_dir =NULL ;unsigned int flags =0 ;const struct option init_db_options[] = { -OPT_STRING (0 ,"template" , &template_dir,N_ ("template-directory" ), -N_ ("directory from which templates will be used" )), +OPT_STRING (0 ,"template" , &template_dir, +N_ ("template-directory" ), +N_ ("directory from which templates will be used" )),OPT_SET_INT (0 ,"bare" , &is_bare_repository_cfg, -N_ ("create a bare repository" ),1 ), +N_ ("create a bare repository" ),1 ), {OPTION_CALLBACK ,0 ,"shared" , &init_shared_repository, -N_ ("permissions" ), -N_ ("specify that the git repository is to be shared amongst several users" ), -PARSE_OPT_OPTARG |PARSE_OPT_NONEG , shared_callback,0 }, +N_ ("permissions" ), +N_ ("specify that the git repository is to be shared amongst several users" ), +PARSE_OPT_OPTARG |PARSE_OPT_NONEG , shared_callback,0 },OPT_BIT ('q' ,"quiet" , &flags,N_ ("be quiet" ),INIT_DB_QUIET ), +OPT_BIT (0 ,"reftable" , &flags,N_ ("use reftable" ), +INIT_DB_REFTABLE ),OPT_STRING (0 ,"separate-git-dir" , &real_git_dir,N_ ("gitdir" ),N_ ("separate git dir from working tree" )),OPT_END ()
cache.h +2 -0
@@ -625,6 +625,7 @@ int path_inside_repo(const char *prefix, const char *path);#define INIT_DB_QUIET 0x0001#define INIT_DB_EXIST_OK 0x0002 +#define INIT_DB_REFTABLE 0x0004int init_db (const char *git_dir,const char *real_git_dir,const char *template_dir,unsigned int flags);
@@ -1041,6 +1042,7 @@ struct repository_format {int is_bare;int hash_algo;char *work_tree; +char *ref_storage;struct string_list unknown_extensions; };
refs.c +14 -8
@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1836,13 +1836,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,* Create, record, and return a ref_store instance for the specified * gitdir. */ -static struct ref_store *ref_store_init (const char *gitdir, +static struct ref_store *ref_store_init (const char *gitdir,const char *be_name,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct ref_storage_be *be;struct ref_store *refs; + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
@@ -1858,7 +1858,12 @@ struct ref_store *get_main_ref_store(struct repository *r)if (!r->gitdir)BUG ("attempting to get main_ref_store outside of repository" ); - r->refs =ref_store_init (r->gitdir,REF_STORE_ALL_CAPS ); + r->refs =ref_store_init (r->gitdir, +/* XXX r->ref_storage_format == NULL. Where +* should the config file be parsed out? */ + r->ref_storage_format ? r->ref_storage_format : +"reftable" , +REF_STORE_ALL_CAPS );return r->refs; }
@@ -1913,7 +1918,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,"files" ,/* XXX */ REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);
@@ -1927,6 +1932,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format ="files" ;/* XXX */ struct ref_store *refs;const char *id;
@@ -1940,9 +1946,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)if (wt->id) refs =ref_store_init (git_common_path ("worktrees/%s" , wt->id), -REF_STORE_ALL_CAPS ); + format,REF_STORE_ALL_CAPS );else - refs =ref_store_init (get_git_common_dir (), + refs =ref_store_init (get_git_common_dir (), format,REF_STORE_ALL_CAPS );if (refs)
refs/refs-internal.h +1 -0
@@ -661,6 +661,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +880 -0
880 lines not on this page · raw message
repository.c +4 -0
@@ -174,6 +174,10 @@ int repo_init(struct repository *repo,if (worktree)repo_set_worktree (repo, worktree); + repo->ref_storage_format = format.ref_storage !=NULL ? +xstrdup (format.ref_storage) : +"files" ;/* XXX */ +clear_repository_format (&format);return 0 ;
repository.h +3 -0
@@ -70,6 +70,9 @@ struct repository {/* The store in which the refs are held. */ struct ref_store *refs; +/* The format to use for the ref database. */ +char *ref_storage_format; +/* * Contains path to often used file names. */
setup.c +5 -2
@@ -448,9 +448,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata)if (!value)return config_error_nonbool (var); data->partial_clone =xstrdup (value); - }else if (!strcmp (ext,"worktreeconfig" )) + }else if (!strcmp (ext,"worktreeconfig" )) { data->worktree_config =git_config_bool (var, value); -else + }else if (!strcmp (ext,"refStorage" )) { + data->ref_storage =xstrdup (value); + }else string_list_append (&data->unknown_extensions, ext); }
@@ -539,6 +541,7 @@ void clear_repository_format(struct repository_format *format)string_list_clear (&format->unknown_extensions,0 );free (format->work_tree);free (format->partial_clone); +free (format->ref_storage);init_repository_format (format); }
Signature · 1 line
gitgitgadget
v4 cover Reftable support git-core
In reply to Han-Wen Nienhuys via GitGitGadgetThis adds the reftable library, and hooks it up as a ref backend.
At this point, I am mainly interested in feedback on the spots marked with XXX in the Git source code.
v3
* passes gitgitgadget CI.
Han-Wen Nienhuys (5):
refs.h: clarify reflog iteration order
create .git/refs in files-backend.c
refs: document how ref_iterator_advance_fn should handle symrefs Add reftable library
Reftable support for git-core
Makefile | 24 +- builtin/init-db.c | 49 +- cache.h | 2 + refs.c | 20 +- refs.h | 5 +- refs/files-backend.c | 6 + refs/refs-internal.h | 6 + refs/reftable-backend.c | 880 +++++++++++++++++++++++++++++++ reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 401 ++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 ++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 286 ++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 ++++ reftable/pq.h | 34 ++ reftable/reader.c | 708 +++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 +++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 399 ++++++++++++++ reftable/slice.c | 199 +++++++ reftable/slice.h | 39 ++ reftable/stack.c | 983 ++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 55 ++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 623 ++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- 45 files changed, 7248 insertions(+), 28 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
base-commit: 5b0ca878e008e82f91300091e793427205ce3544
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Range-diff vs v3:
1: c00403c94d = 1: c00403c94d refs.h: clarify reflog iteration order
2: 57c7342319 < -: ---------- setup.c: enable repo detection for reftable
3: 5b7060cb2f ! 2: 4d6da9bc47 create .git/refs in files-backend.c
@@ -29,9 +29,13 @@
+ files_ref_path(refs, &sb, "refs");
+ safe_create_dir(sb.buf, 1);
-+ /* adjust permissions even if directory already exists. */
++ /* adjust permissions even if directory already exists. */
+ adjust_shared_perm(sb.buf);
+
/*
* Create .git/refs/{heads,tags}
*/
++ strbuf_reset(&sb);
+ files_ref_path(refs, &sb, "refs/heads");
+ safe_create_dir(sb.buf, 1);
+
4: 1b01c735a9 = 3: fbdcdccc88 refs: document how ref_iterator_advance_fn should handle symrefs
5: eb0df10068 ! 4: 02d2ca8b87 Add reftable library
@@ -98,11 +98,11 @@
--- /dev/null
+++ b/reftable/VERSION
@@
-+commit e54326f73d95bfe8b17f264c400f4c365dbd5e5e
++commit e7c3fc3099d9999bc8d895f84027b0e36348d5e6
+Author: Han-Wen Nienhuys <[email protected]>
-+Date: Tue Feb 4 19:48:17 2020 +0100
++Date: Thu Feb 6 20:17:40 2020 +0100
+
-+ C: PRI?MAX use; clang-format.
++ C: use inttypes.h header definitions
diff --git a/reftable/basics.c b/reftable/basics.c
new file mode 100644
@@ -2882,7 +2882,7 @@
+{
+ char hex[SHA256_SIZE + 1] = {};
+
-+ printf("ref{%s(%" PRIdMAX ") ", ref->ref_name, ref->update_index);
++ printf("ref{%s(%" PRIu64 ") ", ref->ref_name, ref->update_index);
+ if (ref->value != NULL) {
+ hex_format(hex, ref->value, hash_size);
+ printf("%s", hex);
@@ -3274,9 +3274,9 @@
+{
+ char hex[SHA256_SIZE + 1] = {};
+
-+ printf("log{%s(%" PRIdMAX ") %s <%s> %" PRIuMAX " %04d\n",
-+ log->ref_name, log->update_index, log->name, log->email,
-+ log->time, log->tz_offset);
++ printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n", log->ref_name,
++ log->update_index, log->name, log->email, log->time,
++ log->tz_offset);
+ hex_format(hex, log->old_hash, hash_size);
+ printf("%s => ", hex);
+ hex_format(hex, log->new_hash, hash_size);
@@ -4860,7 +4860,7 @@
+static void format_name(struct slice *dest, uint64_t min, uint64_t max)
+{
+ char buf[100];
-+ snprintf(buf, sizeof(buf), "%012" PRIxMAX "-%012" PRIxMAX, min, max);
++ snprintf(buf, sizeof(buf), "%012" PRIx64 "-%012" PRIx64, min, max);
+ slice_set_string(dest, buf);
+}
+
@@ -5585,6 +5585,7 @@
+#include <assert.h>
+#include <errno.h>
+#include <fcntl.h>
++#include <inttypes.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
@@ -5595,9 +5596,6 @@
+#include <unistd.h>
+#include <zlib.h>
+
-+#define PRIuMAX "lu"
-+#define PRIdMAX "ld"
-+#define PRIxMAX "lx"
+#define ARRAY_SIZE(a) sizeof((a)) / sizeof((a)[0])
+#define FREE_AND_NULL(x) \
+ do { \
@@ -5605,14 +5603,14 @@
+ (x) = NULL; \
+ } while (0)
+#define QSORT(arr, n, cmp) qsort(arr, n, sizeof(arr[0]), cmp)
-+#define SWAP(a, b) \
-+ { \
-+ char tmp[sizeof(a)]; \
-+ assert(sizeof(a)==sizeof(b)); \
-+ memcpy(&tmp[0], &a, sizeof(a)); \
-+ memcpy(&a, &b, sizeof(a)); \
-+ memcpy(&b, &tmp[0], sizeof(a)); \
-+ }
++#define SWAP(a, b) \
++ { \
++ char tmp[sizeof(a)]; \
++ assert(sizeof(a) == sizeof(b)); \
++ memcpy(&tmp[0], &a, sizeof(a)); \
++ memcpy(&a, &b, sizeof(a)); \
++ memcpy(&b, &tmp[0], sizeof(a)); \
++ }
+#endif
+
+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,
@@ -5758,6 +5756,7 @@
+ return &w->stats.log_stats;
+ }
+ assert(false);
++ return NULL;
+}
+
+/* write data, queuing the padding for the next write. Returns negative for
@@ -6299,7 +6298,7 @@
+ w->stats.blocks++;
+
+ if (debug) {
-+ fprintf(stderr, "block %c off %" PRIuMAX " sz %d (%d)\n", typ,
++ fprintf(stderr, "block %c off %" PRIu64 " sz %d (%d)\n", typ,
+ w->next, raw_bytes,
+ get_u24(w->block + w->block_writer->header_off + 1));
+ }
6: 0b2a1a81d6 ! 5: 2786a6bf61 Reftable support for git-core
@@ -39,6 +39,7 @@
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}
Signed-off-by: Han-Wen Nienhuys <[email protected]>
+ Co-authored-by: Jeff King <[email protected]>
diff --git a/Makefile b/Makefile
--- a/Makefile
@@ -120,6 +121,31 @@ $^
{
struct stat st1;
struct strbuf buf = STRBUF_INIT;
+@@
+ is_bare_repository_cfg = init_is_bare_repository;
+ if (init_shared_repository != -1)
+ set_shared_repository(init_shared_repository);
++ if (flags & INIT_DB_REFTABLE)
++ the_repository->ref_storage_format = xstrdup("reftable");
+
+ /*
+ * We would have created the above under user's umask -- under
+@@
+ adjust_shared_perm(get_git_dir());
+ }
+
++ /*
++ * Check to see if .git/HEAD exists; this must happen before
++ * initializing the ref db, because we want to see if there is an
++ * existing HEAD.
++ */
++ path = git_path_buf(&buf, "HEAD");
++ reinit = (!access(path, R_OK) ||
++ readlink(path, junk, sizeof(junk) - 1) != -1);
++
+ /*
+ * We need to create a "refs" dir in any case so that older
+ * versions of git can tell that this is a repository.
@@
* Create the default symlink from ".git/HEAD" to the "master"
* branch, if it does not exist yet.
@@ -127,18 +153,14 @@ $^
- path = git_path_buf(&buf, "HEAD");
- reinit = (!access(path, R_OK)
- || readlink(path, junk, sizeof(junk)-1) != -1);
-+ if (flags & INIT_DB_REFTABLE) {
-+ reinit = 0; /* XXX - how do we recognize a reinit,
-+ * and what should we do? */
-+ } else {
-+ path = git_path_buf(&buf, "HEAD");
-+ reinit = (!access(path, R_OK) ||
-+ readlink(path, junk, sizeof(junk) - 1) != -1);
-+ }
-+
if (!reinit) {
if (create_symref("HEAD", "refs/heads/master", NULL) < 0)
exit(1);
++ } else {
++ /*
++ * XXX should check whether our ref backend matches the
++ * original one; if not, either die() or convert
++ */
}
/* This forces creation of new config file */
@@ -251,10 +273,8 @@ $^
- r->refs = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
+ r->refs = ref_store_init(r->gitdir,
-+ /* XXX r->ref_storage_format == NULL. Where
-+ * should the config file be parsed out? */
+ r->ref_storage_format ? r->ref_storage_format :
-+ "reftable",
++ "files",
+ REF_STORE_ALL_CAPS);
return r->refs;
}
@@ -410,13 +430,14 @@ $^
+{
+ struct reftable_ref_store *refs =
+ (struct reftable_ref_store *)ref_store;
-+
++ FILE *file;
+ safe_create_dir(refs->reftable_dir, 1);
-+ FILE *f = fopen(refs->table_list_file, "a");
-+ if (f == NULL) {
++
++ file = fopen(refs->table_list_file, "a");
++ if (file == NULL) {
+ return -1;
+ }
-+ fclose(f);
++ fclose(file);
+ return 0;
+}
+
@@ -777,7 +798,6 @@ $^
+ log.old_hash = old_oid.hash;
+ }
+
-+ /* XXX should the resolution be done relative or absolute? */
+ if (refs_resolve_ref_unsafe((struct ref_store *)create->refs,
+ create->target, RESOLVE_REF_READING,
+ &new_oid, NULL) != NULL) {
@@ -1194,9 +1214,7 @@ $^
if (worktree)
repo_set_worktree(repo, worktree);
-+ repo->ref_storage_format = format.ref_storage != NULL ?
-+ xstrdup(format.ref_storage) :
-+ "files"; /* XXX */
++ repo->ref_storage_format = xstrdup_or_null(format.ref_storage);
+
clear_repository_format(&format);
return 0;
@@ -1227,7 +1245,7 @@ $^
+ } else if (!strcmp(ext, "worktreeconfig")) {
data->worktree_config = git_config_bool(var, value);
- else
-+ } else if (!strcmp(ext, "refStorage")) {
++ } else if (!strcmp(ext, "refstorage")) {
+ data->ref_storage = xstrdup(value);
+ } else
string_list_append(&data->unknown_extensions, ext);
@@ -1241,3 +1259,16 @@ $^
init_repository_format(format);
}
+@@
+ gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
+ setup_git_env(gitdir);
+ }
+- if (startup_info->have_repository)
++ if (startup_info->have_repository) {
+ repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
++ the_repository->ref_storage_format =
++ xstrdup_or_null(repo_fmt.ref_storage);
++ }
+ }
+
+ strbuf_release(&dir);Signature · 1 line
gitgitgadget
PATCH v4 1/5 refs.h: clarify reflog iteration order
From: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
refs.h +4 -1
@@ -444,18 +444,21 @@ int delete_refs(const char *msg, struct string_list *refnames,int refs_delete_reflog (struct ref_store *refs,const char *refname);int delete_reflog (const char *refname); -/* iterate over reflog entries */ +/* Iterate over reflog entries. */ typedef int each_reflog_ent_fn (struct object_id *old_oid,struct object_id *new_oid,const char *committer,timestamp_t timestamp,int tz,const char *msg,void *cb_data); +/* Iterate in over reflog entries, oldest entry first. */ int refs_for_each_reflog_ent (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data);int refs_for_each_reflog_ent_reverse (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data); + +/* Call a function for each reflog entry, oldest entry first. */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
Signature · 1 line
gitgitgadget
PATCH v4 2/5 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- builtin/init-db.c | 2 -- refs/files-backend.c | 6 ++++++ 2 files changed, 6 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +6 -0
@@ -3158,9 +3158,15 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */ +strbuf_reset (&sb);files_ref_path (refs, &sb,"refs/heads" );safe_create_dir (sb.buf,1 );
Signature · 1 line
gitgitgadget
PATCH v4 3/5 refs: document how ref_iterator_advance_fn should handle symrefs
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 5 +++++ 1 file changed, 5 insertions(+)
refs/refs-internal.h +5 -0
@@ -438,6 +438,11 @@ void base_ref_iterator_free(struct ref_iterator *iter);/* Virtual function declarations for ref_iterators: */ +/* +* backend-specific implementation of ref_iterator_advance. +* For symrefs, the function should set REF_ISSYMREF, and it should also dereference +* the symref to provide the OID referent. +*/ typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator); typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,
Signature · 1 line
gitgitgadget
PATCH v4 4/5 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Reftable is a new format for storing the ref database. It provides the following benefits:
* Simple and fast atomic ref transactions, including multiple refs and reflogs.
* Compact storage of ref data.
* Fast look ups of ref data.
* Case-sensitive ref names on Windows/OSX, regardless of file system
* Eliminates file/directory conflicts in ref names
Further context and motivation can be found in background reading:
* Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
* Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
* First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
* Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
* First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
* libgit2 support issue: https://github.com/libgit2/libgit2/issues
* GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 401 +++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 +++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 286 +++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 708 +++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 ++++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 399 +++++++++++++++ reftable/slice.c | 199 ++++++++ reftable/slice.h | 39 ++ reftable/stack.c | 983 +++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 55 ++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 623 ++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ 34 files changed, 6267 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
reftable/LICENSE new file +31 -0
@@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
reftable/README.md new file +19 -0
@@ -0,0 +1,19 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + echo '/* empty */' > reftable/config.h + rm reftable/*_test.c reftable/test_framework.* + git add reftable/*.[ch] + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +5 -0
@@ -0,0 +1,5 @@ +commit e7c3fc3099d9999bc8d895f84027b0e36348d5e6 +Author: Han-Wen Nienhuys <[email protected]> +Date: Thu Feb 6 20:17:40 2020 +0100 + + C: use inttypes.h header definitions
reftable/basics.c new file +196 -0
196 lines not on this page · raw message
reftable/basics.h new file +37 -0
@@ -0,0 +1,37 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +401 -0
401 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/config.h new file +1 -0
@@ -0,0 +1 @@ +/* empty */
reftable/constants.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +97 -0
97 lines not on this page · raw message
reftable/file.c new file +97 -0
97 lines not on this page · raw message
reftable/iter.c new file +229 -0
229 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +286 -0
286 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/pq.c new file +114 -0
114 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +708 -0
708 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1107 -0
485 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/reftable.h new file +399 -0
399 lines not on this page · raw message
reftable/slice.c new file +199 -0
199 lines not on this page · raw message
reftable/stack.c new file +983 -0
983 lines not on this page · raw message
reftable/system.h new file +55 -0
55 lines not on this page · raw message
reftable/tree.c new file +66 -0
66 lines not on this page · raw message
reftable/writer.c new file +623 -0
623 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
reftable/zlib-compat.c new file +92 -0
92 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
Re: [PATCH v4 4/5] Add reftable library
Quoted 5 lines · Han-Wen Nienhuys via GitGitGadget
reftable/writer.c new file +5
-0+static struct block_stats *writer_block_stats (struct writer *w,byte typ) +{ +switch (typ) { +case 'r' : +return &w->stats.ref_stats;
reftable/writer.c line 33
Quoted 10 lines
reftable/writer.c new file +10
-0+case 'o' : +return &w->stats.obj_stats; +case 'i' : +return &w->stats.idx_stats; +case 'g' : +return &w->stats.log_stats; + } +assert (false); +return NULL ; +}
As assert() turns into nothing, this is not a particularly good way to document that "if control reaches here, that means we found a programming error".
We intead would use BUG("message") in our codebase, which is marked as NORETURN (so "return NULL" after it would be a dead code).
Thanks.
Re: [PATCH v4 4/5] Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetQuoted 15 lines · Han-Wen Nienhuys via GitGitGadget
reftable/README.md new file +9
-0@@ -0,0 +1,19 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \
reftable/README.md line 19
Quoted 10 lines
reftable/README.md new file +10
-0+ cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + echo '/* empty */' > reftable/config.h + rm reftable/*_test.c reftable/test_framework.* + git add reftable/*.[ch] + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
I think this particular statement may be problematic. The upstream project requires a CLA, which is a non-starter for many folks. I don't think we can expect folks who are working on Git to necessarily send patches upstream with that condition.
For example, if I end up needing to patch this series to work with SHA-256, I'm happy for Google to have my changes under the BSD License, but I won't be signing a CLA.
reftable/README.md
Quoted 2 lines
reftable/README.md new file +2
-0+#define SHA1_SIZE 20 +#define SHA256_SIZE 32
I'd rather we didn't hard-code these values here. If you need a size suitable for allocation, we have GIT_MAX_RAWSZ and GIT_MAX_HEXSZ. Those are guaranteed to be the right size for any future hash.
reftable/README.md
Quoted 7 lines
reftable/README.md new file +7
-0+ { + struct merged_table m = { + .stack = stack, + .stack_len = n, + .min = first_min, + .max = last_max, + .hash_size = SHA1_SIZE,
This is going to cause problems with SHA-256. We'd want to write this as the_hash_algo->rawsz, since it can change at runtime.
Quoted 1 line
+static byte zero[SHA256_SIZE] = {};
It would be better to refer to null_oid here.
reftable/README.md
Quoted 6 lines
reftable/README.md new file +6
-0+#ifndef REFTABLE_H +#define REFTABLE_H + +#include "system.h" + +typedef uint8_t byte;
I think we typically prefer writing "unsigned char" or "uint8_t" instead of "byte".
Quoted 1 line
+typedef byte bool;
I suspect this is going to cause a whole host of sadness if somebody ever includes stdbool.h, or if that header ever gets included by an OS-specific header, since bool is a #define constant for the built-in type _Bool.
We typically use int for booleans, but I'm not opposed to using stdbool.h for it on those systems that we know have it (which, in 2020, is probably all of them).
reftable/README.md
Quoted 10 lines
reftable/README.md new file +10
-0+struct writer *new_writer(int (*writer_func)(void *, byte *, int), + void *writer_arg, struct write_options *opts) +{ + struct writer *wp = calloc(sizeof(struct writer), 1); + options_set_defaults(opts); + if (opts->block_size >= (1 << 24)) { + /* TODO - error return? */ + abort(); + } + wp->hash_size = SHA1_SIZE;
Another SHA-256 problem.
Once this series has good test coverage, I'm happy to send a patch that squashes in SHA-256 support if you don't want to implement it yourself and you CC me. If you do want to implement it yourself, you can grab the transition-stage-4 branch from https://github.com/bk2204/git.git,
rebase your series on top of it, and run the testsuite with GIT_TEST_DEFAULT_HASH=sha256 to see what breaks.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Re: [PATCH v4 4/5] Add reftable library
Quoted 9 lines · brian m. carlson
reftable/README.md line 19
Quoted 3 lines
reftable/README.md new file +3
-0+ +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.I think this particular statement may be problematic. The upstream project requires a CLA, which is a non-starter for many folks. I don't think we can expect folks who are working on Git to necessarily send patches upstream with that condition.
Quoted 3 lines
For example, if I end up needing to patch this series to work with SHA-256, I'm happy for Google to have my changes under the BSD License, but I won't be signing a CLA.
The CLA requirement is something our Open Source lawyers insist on, and it's not for me to renege on. AFAIU, the CLA is essentially a legally sound formulation of "I'm happy for Google to have my changes under the BSD License"
Quoted 6 lines
reftable/constants.h line 13
Quoted 2 lines
reftable/constants.h new file +2
-0+#define SHA1_SIZE 20 +#define SHA256_SIZE 32I'd rather we didn't hard-code these values here. If you need a size suitable for allocation, we have
GIT_MAX_RAWSZandGIT_MAX_HEXSZ. Those are guaranteed to be the right size for any future hash.
this code is structured as a standalone library so the same code can be integrated into libgit2 unchanged, and can be unit-tested without needing complex scaffolding.
As a result some data (eg. SHA1_SIZE) is duplicate. It also comes with its own strbuf library.
Quoted 10 lines
reftable/merged.c line 169
Quoted 7 lines
reftable/merged.c new file +7
-0+ { +struct merged_table m = { + .stack = stack, + .stack_len = n, + .min = first_min, + .max = last_max, + .hash_size =SHA1_SIZE ,This is going to cause problems with SHA-256. We'd want to write this as
the_hash_algo->rawsz, since it can change at runtime.
As this is within the library, the hash size should be injected somewhere. I don't see huge problems here (the hash size could be part of the 'struct write_options'), but until we have properly defined how reftable will mark the on-disk files as SHA-256, it will only supports SHA1. I intend to work set the file format in motion, once this SHA1 version is accepted into git-core.
There was one question you might be able to answer, though. Can repositories exist in dual hash configuration, i.e. where the ref database must store both SHA1 and SHA256 hashes?
Quoted 3 lines
Quoted 1 line
+static byte zero[SHA256_SIZE] = {};
It would be better to refer to
null_oidhere.
See above.
Quoted 13 lines
reftable/reftable.h line 14
Quoted 7 lines
reftable/reftable.h new file +6
-0+#ifndef REFTABLE_H +#define REFTABLE_H + +#include "system.h" + +typedef uint8_t byte ;I think we typically prefer writing "unsigned char" or "
uint8_t" instead of "byte".Quoted 1 line
+typedef byte bool;
Quoted 8 lines
I suspect this is going to cause a whole host of sadness if somebody ever includes
stdbool.h, or if that header ever gets included by an OS-specific header, since bool is a #define constant for the built-in type_Bool.We typically use int for booleans, but I'm not opposed to using
stdbool.hfor it on those systems that we know have it (which, in 2020, is probably all of them).
fixed in https://github.com/google/reftable/commit/a0c794609f2ce8114f014f7260093d4ffdde0d5d
Quoted 13 lines
reftable/writer.c line 103
Quoted 10 lines
reftable/writer.c new file +10
-0+struct writer *new_writer (int (*writer_func)(void *,byte *,int ), +void *writer_arg,struct write_options *opts) +{ +struct writer *wp =calloc (sizeof (struct writer ),1 ); +options_set_defaults (opts); +if (opts->block_size >= (1 <<24 )) { +/* TODO - error return? */ +abort (); + } + wp->hash_size =SHA1_SIZE ;Another SHA-256 problem.
Quoted 2 lines
Once this series has good test coverage, I'm happy to send a patch that squashes in SHA-256 support if you don't want to implement it yourself
There is really nothing to implement. The C library will accept hashes of arbitrary size (hence the hash_size field you commented on before), but it just needs a tiny bit of plumbing for the library user to specify the wanted hash size, and a change to the documented format to store it in the file format header.
Signature · 7 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 4/5] Add reftable library
Quoted 13 lines · Han-Wen Nienhuys
Quoted 10 lines · brian m. carlson
reftable/merged.c line 169
Quoted 7 lines
reftable/merged.c new file +7
-0+ { +struct merged_table m = { + .stack = stack, + .stack_len = n, + .min = first_min, + .max = last_max, + .hash_size =SHA1_SIZE ,This is going to cause problems with SHA-256. We'd want to write this as
the_hash_algo->rawsz, since it can change at runtime.
Quoted 6 lines
As this is within the library, the hash size should be injected somewhere. I don't see huge problems here (the hash size could be part of the 'struct
write_options'), but until we have properly defined how reftable will mark the on-disk files as SHA-256, it will only supports SHA1. I intend to work set the file format in motion, once this SHA1 version is accepted into git-core.
It's not required to mark the on-disk files. A repository's storage is either entirely SHA-1 or entirely SHA-256, except for the translation tables. The entire marking of the repository is in the config file, and everything in the .git directory is assumed to be in that format.
I expect full stage 4 SHA-256 support to land in the next couple of months. Currently, the entire codebase is ready and is hash agnostic except for a small number of remaining tests, and having reftable support land without support for SHA-256 will be a significant regression to that state and delay future series. So when you have a series with tests, I'll send a patch that can be added to the series to add SHA-256 support so we don't have that happen.
Quoted 3 lines
There was one question you might be able to answer, though. Can repositories exist in dual hash configuration, i.e. where the ref database must store both SHA1 and SHA256 hashes?
No. Refs are always stored in the native format with the rest of the repository. When we gain support for using both SHA-1 and SHA-256 to refer to objects in the same repository, there will be either pack index v3 or a loose object lookup table which has both and is used to translate. We won't store both values in the ref database at once.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Re: [PATCH v4 4/5] Add reftable library
Quoted 11 lines · brian m. carlson
Quoted 6 lines
As this is within the library, the hash size should be injected somewhere. I don't see huge problems here (the hash size could be part of the 'struct
write_options'), but until we have properly defined how reftable will mark the on-disk files as SHA-256, it will only supports SHA1. I intend to work set the file format in motion, once this SHA1 version is accepted into git-core.It's not required to mark the on-disk files. A repository's storage is either entirely SHA-1 or entirely SHA-256, except for the translation tables. The entire marking of the repository is in the config file, and everything in the .git directory is assumed to be in that format.
I disagree. I think it should be possible to determine if a file in reftable format is well-formed or not. Since parsing requires knowing the hash size, we have to encode it somewhere.
I've uploaded https://git.eclipse.org/r/c/157501/ that proposes a way to encode the hash size. I look forward to feedback.
Signature · 7 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 4/5] Add reftable library
Quoted 2 lines · Han-Wen Nienhuys
I've uploaded https://git.eclipse.org/r/c/157501/ that proposes a way to encode the hash size. I look forward to feedback.
Development and design discussion happens here. Please do not require people to go click at an external website---once people start doing so, we'd have to go around 47 different places to piece one discussion together, which is just crazy.
Here is what I saw:
A 24-byte header appears at the beginning of the file:
'REFT'
uint8( format_version )
uint24( block_size )
uint64( min_update_index )
uint64( max_update_index )
The format_version is a byte, and it indicates both the version of the on-disk format, as well as the size of the hash. The hash size is indicated in the MSB of the format_version. For the SHA1 hash, format_version & 0x80 == 0 and all hash values are 20 bytes. For SHA256, format_version & 0x80 == 1, and all hash values are 32 bytes. Future hash functions may be added by using more bits at the right.
The file format version can be extract as format_version & 0x7f. Currently, only version 1 is defined.
If you cast in stone that "& 0x7f is the way to extract the version", then you cannot promise that you may steal more bits at the right of MSB to support more hash functions, as you've reserved the rightmost 7 bits already for the version number with 0x7f and there are only 8 bits in your byte.
It seems that you are trying to make the format too dense? Is it too much a waste to use a separate word or a byte for hash? Or perhaps declare that format version 1 uses SHA-1, format version 2 uses SHA-256, etc. (in other words, do we want to support both SHA-1 and SHA-256 when we are at format version 7)?
Thanks.
Re: [PATCH v4 4/5] Add reftable library
Quoted 26 lines · Junio C Hamano
Here is what I saw:
A 24-byte header appears at the beginning of the file:
'REFT'
uint8( format_version )
uint24( block_size )
uint64( min_update_index )
uint64( max_update_index )The
format_versionis a byte, and it indicates both the version of the on-disk format, as well as the size of the hash. The hash size is indicated in the MSB of theformat_version. For the SHA1 hash,format_version & 0x80 == 0and all hash values are 20 bytes. For SHA256,format_version & 0x80 == 1, and all hash values are 32 bytes. Future hash functions may be added by using more bits at the right.The file format version can be extract as
format_version & 0x7f. Currently, only version 1 is defined.If you cast in stone that "&
0x7fis the way to extract the version", then you cannot promise that you may steal more bits at the right of MSB to support more hash functions, as you've reserved the rightmost 7 bits already for the version number with0x7fand there are only 8 bits in your byte.
Quoted 5 lines
It seems that you are trying to make the format too dense? Is it too much a waste to use a separate word or a byte for hash? Or perhaps declare that format version 1 uses SHA-1, format version 2 uses SHA-256, etc. (in other words, do we want to support both SHA-1 and SHA-256 when we are at format version 7)?
I can see a future where we have a different format that allows for more metadata so we can encode the hash size separately. But maybe that can be for format v3 and up.
Let's do format v2 = format v1 but with 32-byte hashes.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 4/5] Add reftable library
Quoted 5 lines · Han-Wen Nienhuys
I can see a future where we have a different format that allows for more metadata so we can encode the hash size separately. But maybe that can be for format v3 and up.
Let's do format v2 = format v1 but with 32-byte hashes.
The way we've preferred to do this in Git is to write a four-byte hash identifier into the file, but we have legacy concerns here, so I think switching to v2 for SHA-256 is fine.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Re: [PATCH v4 4/5] Add reftable library
Quoted 6 lines · brian m. carlson
Quoted 4 lines · Han-Wen Nienhuys
I can see a future where we have a different format that allows for more metadata so we can encode the hash size separately. But maybe that can be for format v3 and up.
Quoted 8 lines
Quoted 1 line
Let's do format v2 = format v1 but with 32-byte hashes.
The way we've preferred to do this in Git is to write a four-byte hash identifier into the file, but we have legacy concerns here, so I think switching to v2 for SHA-256 is fine.
--
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
I've implemented this in the last version of the series.
Signature · 7 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 4/5] Add reftable library
In reply to Junio C HamanoQuoted 10 lines · Junio C Hamano
Quoted 2 lines · Han-Wen Nienhuys
I've uploaded https://git.eclipse.org/r/c/157501/ that proposes a way to encode the hash size. I look forward to feedback.
Development and design discussion happens here. Please do not require people to go click at an external website---once people start doing so, we'd have to go around 47 different places to piece one discussion together, which is just crazy.
The ground truth for the format is the spec. The spec is part of the JGit repository, so it seems reasonable for it to be reviewed there. Do you want to move it somewhere else?
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 4/5] Add reftable library
Quoted 3 lines · Han-Wen Nienhuys
The ground truth for the format is the spec. The spec is part of the JGit repository, so it seems reasonable for it to be reviewed there. Do you want to move it somewhere else?
Let's stick a copy in Documentation/technical. I can send a patch to Git to do that today, and we can update the doc in JGit to point to Git's version as the canonical one.
Thanks,
Jonathan
PATCH v4 5/5 Reftable support for git-core
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
For background, see the previous commit introducing the library.
TODO:
* Make CI on gitgitgadget compile.
* Redo interaction between repo config, repo storage version
* Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
Example use:
$ ~/vc/git/git init --reftable
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
$ ~/vc/git/git add a
$ ~/vc/git/git commit -mx
fatal: not a git repository (or any of the parent directories): .git
[master (root-commit) 373d969] x
1 file changed, 1 insertion(+)
create mode 100644 a
$ ~/vc/git/git show-ref
373d96972fca9b63595740bba3898a762778ba20 HEAD
373d96972fca9b63595740bba3898a762778ba20 refs/heads/master
$ ls -l .git/reftable/
total 12
-rw------- 1 hanwen primarygroup 126 Jan 23 20:08 000000000001-000000000001.ref
-rw------- 1 hanwen primarygroup 4277 Jan 23 20:08 000000000002-000000000002.ref
$ go run ~/vc/reftable/cmd/dump.go -table /tmp/qz/.git/reftable/000000000002-000000000002.ref
** DEBUG **
name /tmp/qz/.git/reftable/000000000002-000000000002.ref, sz 4209: 'r' reftable.readerOffsets{Present:true, Offset:0x0, IndexOffset:0x0}, 'o' reftable.readerOffsets{Present:false, Offset:0x0, IndexOffset:0x0} 'g' reftable.readerOffsets{Present:true, Offset:0x1000, IndexOffset:0x0}
** REFS **
reftable.RefRecord{RefName:"refs/heads/master", UpdateIndex:0x2, Value:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, TargetValue:[]uint8(nil), Target:""}
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}Signed-off-by Han-Wen Nienhuys <[email protected]>
Co-authored-by Jeff King <[email protected]>
--- Makefile | 24 +- builtin/init-db.c | 47 ++- cache.h | 2 + refs.c | 20 +- refs/refs-internal.h | 1 + refs/reftable-backend.c | 880 ++++++++++++++++++++++++++++++++++++++++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- 9 files changed, 966 insertions(+), 25 deletions(-) create mode 100644 refs/reftable-backend.c
Makefile +23 -1
@@ -814,6 +814,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += command-list.h
@@ -959,6 +960,7 @@ LIB_OBJS += rebase-interactive.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
@@ -1163,7 +1165,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION)
@@ -2347,11 +2349,28 @@ VCSSVN_OBJS += vcs-svn/fast_export.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o +REFTABLE_OBJS += reftable/zlib-compat.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL
@@ -2488,6 +2507,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
builtin/init-db.c +34 -13
98 lines not on this page · raw message
cache.h +2 -0
@@ -625,6 +625,7 @@ int path_inside_repo(const char *prefix, const char *path);#define INIT_DB_QUIET 0x0001#define INIT_DB_EXIST_OK 0x0002 +#define INIT_DB_REFTABLE 0x0004int init_db (const char *git_dir,const char *real_git_dir,const char *template_dir,unsigned int flags);
@@ -1041,6 +1042,7 @@ struct repository_format {int is_bare;int hash_algo;char *work_tree; +char *ref_storage;struct string_list unknown_extensions; };
refs.c +12 -8
@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1836,13 +1836,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,* Create, record, and return a ref_store instance for the specified * gitdir. */ -static struct ref_store *ref_store_init (const char *gitdir, +static struct ref_store *ref_store_init (const char *gitdir,const char *be_name,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct ref_storage_be *be;struct ref_store *refs; + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
@@ -1858,7 +1858,10 @@ struct ref_store *get_main_ref_store(struct repository *r)if (!r->gitdir)BUG ("attempting to get main_ref_store outside of repository" ); - r->refs =ref_store_init (r->gitdir,REF_STORE_ALL_CAPS ); + r->refs =ref_store_init (r->gitdir, + r->ref_storage_format ? r->ref_storage_format : +"files" , +REF_STORE_ALL_CAPS );return r->refs; }
@@ -1913,7 +1916,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,"files" ,/* XXX */ REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);
@@ -1927,6 +1930,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format ="files" ;/* XXX */ struct ref_store *refs;const char *id;
@@ -1940,9 +1944,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)if (wt->id) refs =ref_store_init (git_common_path ("worktrees/%s" , wt->id), -REF_STORE_ALL_CAPS ); + format,REF_STORE_ALL_CAPS );else - refs =ref_store_init (get_git_common_dir (), + refs =ref_store_init (get_git_common_dir (), format,REF_STORE_ALL_CAPS );if (refs)
refs/refs-internal.h +1 -0
@@ -661,6 +661,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +880 -0
880 lines not on this page · raw message
repository.c +2 -0
@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,if (worktree)repo_set_worktree (repo, worktree); + repo->ref_storage_format =xstrdup_or_null (format.ref_storage); +clear_repository_format (&format);return 0 ;
repository.h +3 -0
@@ -70,6 +70,9 @@ struct repository {/* The store in which the refs are held. */ struct ref_store *refs; +/* The format to use for the ref database. */ +char *ref_storage_format; +/* * Contains path to often used file names. */
setup.c +9 -3
@@ -440,9 +440,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata)if (!value)return config_error_nonbool (var); data->partial_clone =xstrdup (value); - }else if (!strcmp (ext,"worktreeconfig" )) + }else if (!strcmp (ext,"worktreeconfig" )) { data->worktree_config =git_config_bool (var, value); -else + }else if (!strcmp (ext,"refstorage" )) { + data->ref_storage =xstrdup (value); + }else string_list_append (&data->unknown_extensions, ext); }
@@ -531,6 +533,7 @@ void clear_repository_format(struct repository_format *format)string_list_clear (&format->unknown_extensions,0 );free (format->work_tree);free (format->partial_clone); +free (format->ref_storage);init_repository_format (format); }
@@ -1173,8 +1176,11 @@ const char *setup_git_directory_gently(int *nongit_ok) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ;setup_git_env (gitdir); } -if (startup_info->have_repository) +if (startup_info->have_repository) {repo_set_hash_algo (the_repository, repo_fmt.hash_algo); + the_repository->ref_storage_format = +xstrdup_or_null (repo_fmt.ref_storage); + } }strbuf_release (&dir);
Signature · 1 line
gitgitgadget
Re: [PATCH v4 5/5] Reftable support for git-core
builtin/init-db.c int init_db(const char *git_dir, const char *real_git_dir, · line 422
Quoted 7 lines · Han-Wen Nienhuys via GitGitGadget
builtin/init-db.c +3
-0@@ -403,6 +417,10 @@ int init_db(const char *git_dir, const char *real_git_dir,git_config_set ("receive.denyNonFastforwards" ,"true" ); } +if (flags &INIT_DB_REFTABLE ) { +git_config_set ("extensions.refStorage" ,"reftable" ); + }
This does seem like the best way to do this. Can we get an addition to Documentation/technical/repository-version.txt that documents this extension and the values it takes? I presume "reftable" and "files" are options, but it would be nice it folks didn't have to dig through the code to find that out.
One side note: we typically omit the braces for a single-line if.
Quoted 16 lines
builtin/init-db.c +5
-3+if (!(flags &INIT_DB_QUIET )) {int len =strlen (git_dir);@@ -481,15 +499,18 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)const char *template_dir =NULL ;unsigned int flags =0 ;const struct option init_db_options[] = { -OPT_STRING (0 ,"template" , &template_dir,N_ ("template-directory" ), -N_ ("directory from which templates will be used" )), +OPT_STRING (0 ,"template" , &template_dir, +N_ ("template-directory" ), +N_ ("directory from which templates will be used" )),OPT_SET_INT (0 ,"bare" , &is_bare_repository_cfg, -N_ ("create a bare repository" ),1 ), +N_ ("create a bare repository" ),1 ),
builtin/init-db.c int cmd_init_db(int argc, const char **argv, const char *prefix) · line 513
Quoted 10 lines
builtin/init-db.c +5
-3{OPTION_CALLBACK ,0 ,"shared" , &init_shared_repository, -N_ ("permissions" ), -N_ ("specify that the git repository is to be shared amongst several users" ), - PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0}, +N_ ("permissions" ), +N_ ("specify that the git repository is to be shared amongst several users" ), + PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },OPT_BIT ('q' ,"quiet" , &flags,N_ ("be quiet" ),INIT_DB_QUIET ), +OPT_BIT (0 ,"reftable" , &flags,N_ ("use reftable" ), + INIT_DB_REFTABLE),
I wonder if this might be better as --refs-backend={files|reftable}. If reftable becomes the default at some point in the future, it would be easier to let people explicitly specify that they want a non-reftable version. If we learn support for a hypothetical reftable v2 in the future, maybe we'd want to let folks specify that instead.
Quoted 28 lines
builtin/init-db.c +0
-0OPT_STRING (0 ,"separate-git-dir" , &real_git_dir,N_ ("gitdir" ),N_ ("separate git dir from working tree" )),OPT_END ()cache.h +2
-0@@ -625,6 +625,7 @@ int path_inside_repo(const char *prefix, const char *path);#define INIT_DB_QUIET 0x0001#define INIT_DB_EXIST_OK 0x0002 +#define INIT_DB_REFTABLE 0x0004int init_db (const char *git_dir,const char *real_git_dir,const char *template_dir,unsigned int flags);@@ -1041,6 +1042,7 @@ struct repository_format {int is_bare;int hash_algo;char *work_tree; +char *ref_storage;struct string_list unknown_extensions; };refs.c +0
-0
refs.c line 23
Quoted 5 lines
refs.c +1
-1@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;
I'm not sure why we're making this change. It doesn't look like the default is changing.
Quoted 10 lines
refs.c +1
-1@@ -1913,7 +1916,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,"files" ,/* XXX */ REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);
refs.c struct ref_store *get_submodule_ref_store(const char *submodule) · line 1933
Quoted 4 lines
refs.c +1
-0@@ -1927,6 +1930,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format ="files" ;/* XXX */
If the question is whether we still want to default to "files" as the default, then yes, I think we do. Suddenly changing things would be a problem if for some reason someone needed to downgrade Git versions.
Since we have two instances of "files" above, it might be better to create a #define like DEFAULT_REF_BACKEND or some such.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
Re: [PATCH v4 5/5] Reftable support for git-core
Quoted 6 lines · brian m. carlson
Quoted 4 lines · Han-Wen Nienhuys via GitGitGadget
builtin/init-db.c +0
-0@@ -403,6 +417,10 @@ int init_db(const char *git_dir, const char *real_git_dir,git_config_set ("receive.denyNonFastforwards" ,"true" ); }
Quoted 9 lines
builtin/init-db.c int init_db(const char *git_dir, const char *real_git_dir, · line 422
Quoted 3 lines
builtin/init-db.c +3
-0+if (flags &INIT_DB_REFTABLE ) { +git_config_set ("extensions.refStorage" ,"reftable" ); + }This does seem like the best way to do this. Can we get an addition to
Documentation/technical/repository-version.txtthat documents this extension and the values it takes? I presume "reftable" and "files" are options, but it would be nice it folks didn't have to dig through the code to find that out.
done.
Quoted 5 lines
I wonder if this might be better as --refs-backend={files|reftable}. If reftable becomes the default at some point in the future, it would be easier to let people explicitly specify that they want a non-reftable version. If we learn support for a hypothetical reftable v2 in the future, maybe we'd want to let folks specify that instead.
done.
Quoted 5 lines
refs.c line 23
Quoted 2 lines
refs.c +1
-1-static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;I'm not sure why we're making this change. It doesn't look like the default is changing.
This is adding the reftable backend to the ref_storage_be linked list.
Quoted 20 lines
Quoted 11 lines
refs.c +1
-1@@ -1913,7 +1916,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,"files" ,/* XXX */ REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);refs.c struct ref_store *get_submodule_ref_store(const char *submodule) · line 1933
Quoted 4 lines
refs.c +1
-0@@ -1927,6 +1930,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format ="files" ;/* XXX */ If the question is whether we still want to default to "files" as the default, then yes, I think we do. Suddenly changing things would be a problem if for some reason someone needed to downgrade Git versions.
Quoted 2 lines
Since we have two instances of "files" above, it might be better to create a #define like
DEFAULT_REF_BACKENDor some such.
I did this. I stuck it in refs.h which is probably the wrong place. Suggestions welcome.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Quoted 8 lines · Han-Wen Nienhuys via GitGitGadget
This adds the reftable library, and hooks it up as a ref backend.
At this point, I am mainly interested in feedback on the spots marked with XXX in the Git source code.
v3
* passes gitgitgadget CI.
It's interesting you should say that, because one thing I'm missing here is some tests. Since SHA-256 support is rapidly coming to Git, I'd like to make sure that this series works with SHA-256 repositories, but I can't do that if there are no tests.
Signature · 2 lines
brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204
v5 cover Reftable support git-core
In reply to Han-Wen Nienhuys via GitGitGadgetThis adds the reftable library, and hooks it up as a ref backend.
At this point, I am mainly interested in feedback on the spots marked with XXX in the Git source code.
v3
* passes gitgitgadget CI.
Han-Wen Nienhuys (5):
refs.h: clarify reflog iteration order
create .git/refs in files-backend.c
refs: document how ref_iterator_advance_fn should handle symrefs Add reftable library
Reftable support for git-core
.../technical/repository-version.txt | 7 + Makefile | 24 +- builtin/clone.c | 4 +- builtin/init-db.c | 57 +- cache.h | 4 +- refs.c | 20 +- refs.h | 8 +- refs/files-backend.c | 6 + refs/refs-internal.h | 6 + refs/reftable-backend.c | 880 +++++++++++++ reftable/LICENSE | 31 + reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++ reftable/basics.h | 37 + reftable/block.c | 401 ++++++ reftable/block.h | 71 ++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++ reftable/file.c | 97 ++ reftable/iter.c | 229 ++++ reftable/iter.h | 56 + reftable/merged.c | 286 +++++ reftable/merged.h | 34 + reftable/pq.c | 114 ++ reftable/pq.h | 34 + reftable/reader.c | 708 +++++++++++ reftable/reader.h | 52 + reftable/record.c | 1107 +++++++++++++++++ reftable/record.h | 79 ++ reftable/reftable.h | 394 ++++++ reftable/slice.c | 199 +++ reftable/slice.h | 39 + reftable/stack.c | 983 +++++++++++++++ reftable/stack.h | 40 + reftable/system.h | 58 + reftable/tree.c | 66 + reftable/tree.h | 24 + reftable/writer.c | 623 ++++++++++ reftable/writer.h | 46 + reftable/zlib-compat.c | 92 ++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- 47 files changed, 7266 insertions(+), 32 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
base-commit: 5b0ca878e008e82f91300091e793427205ce3544
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Range-diff vs v4:
1: c00403c94d = 1: c00403c94d refs.h: clarify reflog iteration order
2: 4d6da9bc47 = 2: 4d6da9bc47 create .git/refs in files-backend.c
3: fbdcdccc88 = 3: fbdcdccc88 refs: document how ref_iterator_advance_fn should handle symrefs
4: 02d2ca8b87 ! 4: 546b82fe79 Add reftable library
@@ -98,11 +98,11 @@
--- /dev/null
+++ b/reftable/VERSION
@@
-+commit e7c3fc3099d9999bc8d895f84027b0e36348d5e6
++commit 6115b50fdb9bc662be39b05f5589bc109282ae7f
+Author: Han-Wen Nienhuys <[email protected]>
-+Date: Thu Feb 6 20:17:40 2020 +0100
++Date: Mon Feb 10 13:59:52 2020 +0100
+
-+ C: use inttypes.h header definitions
++ README: add a note about the Java implementation
diff --git a/reftable/basics.c b/reftable/basics.c
new file mode 100644
@@ -3885,9 +3885,6 @@
+
+#include "system.h"
+
-+typedef uint8_t byte;
-+typedef byte bool;
-+
+/* block_source is a generic wrapper for a seekable readable file.
+ It is generally passed around by value.
+ */
@@ -3900,7 +3897,7 @@
+ so it can return itself into the pool.
+*/
+struct block {
-+ byte *data;
++ uint8_t *data;
+ int len;
+ struct block_source source;
+};
@@ -3926,14 +3923,14 @@
+
+/* write_options sets options for writing a single reftable. */
+struct write_options {
-+ /* do not pad out blocks to block size. */
-+ bool unpadded;
++ /* boolean: do not pad out blocks to block size. */
++ int unpadded;
+
+ /* the blocksize. Should be less than 2^24. */
+ uint32_t block_size;
+
-+ /* do not generate a SHA1 => ref index. */
-+ bool skip_index_objects;
++ /* boolean: do not generate a SHA1 => ref index. */
++ int skip_index_objects;
+
+ /* how often to write complete keys in each block. */
+ int restart_interval;
@@ -3944,13 +3941,13 @@
+ char *ref_name; /* Name of the ref, malloced. */
+ uint64_t update_index; /* Logical timestamp at which this value is
+ written */
-+ byte *value; /* SHA1, or NULL. malloced. */
-+ byte *target_value; /* peeled annotated tag, or NULL. malloced. */
++ uint8_t *value; /* SHA1, or NULL. malloced. */
++ uint8_t *target_value; /* peeled annotated tag, or NULL. malloced. */
+ char *target; /* symref, or NULL. malloced. */
+};
+
+/* returns whether 'ref' represents a deletion */
-+bool ref_record_is_deletion(const struct ref_record *ref);
++int ref_record_is_deletion(const struct ref_record *ref);
+
+/* prints a ref_record onto stdout */
+void ref_record_print(struct ref_record *ref, int hash_size);
@@ -3959,15 +3956,14 @@
+void ref_record_clear(struct ref_record *ref);
+
+/* returns whether two ref_records are the same */
-+bool ref_record_equal(struct ref_record *a, struct ref_record *b,
-+ int hash_size);
++int ref_record_equal(struct ref_record *a, struct ref_record *b, int hash_size);
+
+/* log_record holds a reflog entry */
+struct log_record {
+ char *ref_name;
+ uint64_t update_index;
-+ byte *new_hash;
-+ byte *old_hash;
++ uint8_t *new_hash;
++ uint8_t *old_hash;
+ char *name;
+ char *email;
+ uint64_t time;
@@ -3976,14 +3972,13 @@
+};
+
+/* returns whether 'ref' represents the deletion of a log record. */
-+bool log_record_is_deletion(const struct log_record *log);
++int log_record_is_deletion(const struct log_record *log);
+
+/* frees and nulls all pointer values. */
+void log_record_clear(struct log_record *log);
+
+/* returns whether two records are equal. */
-+bool log_record_equal(struct log_record *a, struct log_record *b,
-+ int hash_size);
++int log_record_equal(struct log_record *a, struct log_record *b, int hash_size);
+
+void log_record_print(struct log_record *log, int hash_size);
+
@@ -4076,11 +4071,11 @@
+const char *error_str(int err);
+
+/* new_writer creates a new writer */
-+struct writer *new_writer(int (*writer_func)(void *, byte *, int),
++struct writer *new_writer(int (*writer_func)(void *, uint8_t *, int),
+ void *writer_arg, struct write_options *opts);
+
+/* write to a file descriptor. fdp should be an int* pointing to the fd. */
-+int fd_writer(void *fdp, byte *data, int size);
++int fd_writer(void *fdp, uint8_t *data, int size);
+
+/* Set the range of update indices for the records we will add. When
+ writing a table into a stack, the min should be at least
@@ -4163,7 +4158,7 @@
+void reader_free(struct reader *);
+
+/* return an iterator for the refs pointing to oid */
-+int reader_refs_for(struct reader *r, struct iterator *it, byte *oid,
++int reader_refs_for(struct reader *r, struct iterator *it, uint8_t *oid,
+ int oid_len);
+
+/* return the max_update_index for a table */
@@ -5613,6 +5608,9 @@
+ }
+#endif
+
++typedef uint8_t byte;
++typedef int bool;
++
+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,
+ const Bytef *source, uLong *sourceLen);
+
5: 2786a6bf61 ! 5: 702fb89871 Reftable support for git-core
@@ -41,6 +41,21 @@
Signed-off-by: Han-Wen Nienhuys <[email protected]>
Co-authored-by: Jeff King <[email protected]>
+ diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt
+ --- a/Documentation/technical/repository-version.txt
+ +++ b/Documentation/technical/repository-version.txt
+@@
+ multiple working directory mode, "config" file is shared while
+ "config.worktree" is per-working directory (i.e., it's in
+ GIT_COMMON_DIR/worktrees/<id>/config.worktree)
++
++==== `refStorage`
++
++Specifies the file format for the ref database. Values are `files`
++(for the traditional packed + loose ref format) and `reftable` for the
++binary reftable format. See https://github.com/google/reftable for
++more information.
+
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -109,6 +124,21 @@ $^
Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
+ diff --git a/builtin/clone.c b/builtin/clone.c
+ --- a/builtin/clone.c
+ +++ b/builtin/clone.c
+@@
+ }
+ }
+
+- init_db(git_dir, real_git_dir, option_template, INIT_DB_QUIET);
++ init_db(git_dir, real_git_dir, option_template,
++ DEFAULT_REF_STORAGE, /* XXX */
++ INIT_DB_QUIET);
+
+ if (real_git_dir)
+ git_dir = real_git_dir;
+
diff --git a/builtin/init-db.c b/builtin/init-db.c
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -117,7 +147,8 @@ $^
static int create_default_files(const char *template_path,
- const char *original_git_dir)
-+ const char *original_git_dir, int flags)
++ const char *original_git_dir,
++ const char *ref_storage_format, int flags)
{
struct stat st1;
struct strbuf buf = STRBUF_INIT;
@@ -125,8 +156,7 @@ $^
is_bare_repository_cfg = init_is_bare_repository;
if (init_shared_repository != -1)
set_shared_repository(init_shared_repository);
-+ if (flags & INIT_DB_REFTABLE)
-+ the_repository->ref_storage_format = xstrdup("reftable");
++ the_repository->ref_storage_format = xstrdup(ref_storage_format);
/*
* We would have created the above under user's umask -- under
@@ -167,17 +197,29 @@ $^
- xsnprintf(repo_version_string, sizeof(repo_version_string),
- "%d", GIT_REPO_VERSION);
+ xsnprintf(repo_version_string, sizeof(repo_version_string), "%d",
-+ flags & INIT_DB_REFTABLE ? GIT_REPO_VERSION_READ :
-+ GIT_REPO_VERSION);
++ !strcmp(ref_storage_format, "reftable") ?
++ GIT_REPO_VERSION_READ :
++ GIT_REPO_VERSION);
git_config_set("core.repositoryformatversion", repo_version_string);
/* Check filemode trustability */
+@@
+ }
+
+ int init_db(const char *git_dir, const char *real_git_dir,
+- const char *template_dir, unsigned int flags)
++ const char *template_dir, const char *ref_storage_format,
++ unsigned int flags)
+ {
+ int reinit;
+ int exist_ok = flags & INIT_DB_EXIST_OK;
@@
*/
check_repository_format();
- reinit = create_default_files(template_dir, original_git_dir);
-+ reinit = create_default_files(template_dir, original_git_dir, flags);
++ reinit = create_default_files(template_dir, original_git_dir,
++ ref_storage_format, flags);
create_object_directory();
@@ -185,14 +227,18 @@ $^
git_config_set("receive.denyNonFastforwards", "true");
}
-+ if (flags & INIT_DB_REFTABLE) {
-+ git_config_set("extensions.refStorage", "reftable");
-+ }
++ git_config_set("extensions.refStorage", ref_storage_format);
+
if (!(flags & INIT_DB_QUIET)) {
int len = strlen(git_dir);
@@
+ int cmd_init_db(int argc, const char **argv, const char *prefix)
+ {
+ const char *git_dir;
++ const char *ref_storage_format = DEFAULT_REF_STORAGE;
+ const char *real_git_dir = NULL;
+ const char *work_tree;
const char *template_dir = NULL;
unsigned int flags = 0;
const struct option init_db_options[] = {
@@ -212,23 +258,38 @@ $^
+ N_("specify that the git repository is to be shared amongst several users"),
+ PARSE_OPT_OPTARG | PARSE_OPT_NONEG, shared_callback, 0 },
OPT_BIT('q', "quiet", &flags, N_("be quiet"), INIT_DB_QUIET),
-+ OPT_BIT(0, "reftable", &flags, N_("use reftable"),
-+ INIT_DB_REFTABLE),
++ OPT_STRING(0, "ref-storage", &ref_storage_format, N_("backend"),
++ N_("the ref storage format to use")),
OPT_STRING(0, "separate-git-dir", &real_git_dir, N_("gitdir"),
N_("separate git dir from working tree")),
OPT_END()
+@@
+ }
+
+ UNLEAK(real_git_dir);
++ UNLEAK(ref_storage_format);
+ UNLEAK(git_dir);
+ UNLEAK(work_tree);
+
+ flags |= INIT_DB_EXIST_OK;
+- return init_db(git_dir, real_git_dir, template_dir, flags);
++ return init_db(git_dir, real_git_dir, template_dir, ref_storage_format,
++ flags);
+ }
diff --git a/cache.h b/cache.h
--- a/cache.h
+++ b/cache.h
@@
-
- #define INIT_DB_QUIET 0x0001
#define INIT_DB_EXIST_OK 0x0002
-+#define INIT_DB_REFTABLE 0x0004
int init_db(const char *git_dir, const char *real_git_dir,
- const char *template_dir, unsigned int flags);
+- const char *template_dir, unsigned int flags);
++ const char *template_dir, const char *ref_storage_format,
++ unsigned int flags);
+
+ void sanitize_stdfds(void);
+ int daemonize(void);
@@
int is_bare;
int hash_algo;
@@ -274,7 +335,7 @@ $^
- r->refs = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
+ r->refs = ref_store_init(r->gitdir,
+ r->ref_storage_format ? r->ref_storage_format :
-+ "files",
++ DEFAULT_REF_STORAGE,
+ REF_STORE_ALL_CAPS);
return r->refs;
}
@@ -284,7 +345,7 @@ $^
/* assume that add_submodule_odb() has been called */
- refs = ref_store_init(submodule_sb.buf,
-+ refs = ref_store_init(submodule_sb.buf, "files", /* XXX */
++ refs = ref_store_init(submodule_sb.buf, DEFAULT_REF_STORAGE, /* XXX */
REF_STORE_READ | REF_STORE_ODB);
register_ref_store_map(&submodule_ref_stores, "submodule",
refs, submodule);
@@ -292,7 +353,7 @@ $^
struct ref_store *get_worktree_ref_store(const struct worktree *wt)
{
-+ const char *format = "files"; /* XXX */
++ const char *format = DEFAULT_REF_STORAGE; /* XXX */
struct ref_store *refs;
const char *id;
@@ -309,6 +370,20 @@ $^
if (refs)
+ diff --git a/refs.h b/refs.h
+ --- a/refs.h
+ +++ b/refs.h
+@@
+ struct string_list_item;
+ struct worktree;
+
++/* XXX where should this be? */
++#define DEFAULT_REF_STORAGE "files"
++
+ /*
+ * Resolve a reference, recursively following symbolic refererences.
+ *
+
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.hSignature · 1 line
gitgitgadget
PATCH v5 1/5 refs.h: clarify reflog iteration order
From: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
refs.h +4 -1
@@ -444,18 +444,21 @@ int delete_refs(const char *msg, struct string_list *refnames,int refs_delete_reflog (struct ref_store *refs,const char *refname);int delete_reflog (const char *refname); -/* iterate over reflog entries */ +/* Iterate over reflog entries. */ typedef int each_reflog_ent_fn (struct object_id *old_oid,struct object_id *new_oid,const char *committer,timestamp_t timestamp,int tz,const char *msg,void *cb_data); +/* Iterate in over reflog entries, oldest entry first. */ int refs_for_each_reflog_ent (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data);int refs_for_each_reflog_ent_reverse (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data); + +/* Call a function for each reflog entry, oldest entry first. */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
Signature · 1 line
gitgitgadget
PATCH v5 2/5 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- builtin/init-db.c | 2 -- refs/files-backend.c | 6 ++++++ 2 files changed, 6 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +6 -0
@@ -3158,9 +3158,15 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */ +strbuf_reset (&sb);files_ref_path (refs, &sb,"refs/heads" );safe_create_dir (sb.buf,1 );
Signature · 1 line
gitgitgadget
PATCH v5 3/5 refs: document how ref_iterator_advance_fn should handle symrefs
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 5 +++++ 1 file changed, 5 insertions(+)
refs/refs-internal.h +5 -0
@@ -438,6 +438,11 @@ void base_ref_iterator_free(struct ref_iterator *iter);/* Virtual function declarations for ref_iterators: */ +/* +* backend-specific implementation of ref_iterator_advance. +* For symrefs, the function should set REF_ISSYMREF, and it should also dereference +* the symref to provide the OID referent. +*/ typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator); typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,
Signature · 1 line
gitgitgadget
PATCH v5 4/5 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Reftable is a new format for storing the ref database. It provides the following benefits:
* Simple and fast atomic ref transactions, including multiple refs and reflogs.
* Compact storage of ref data.
* Fast look ups of ref data.
* Case-sensitive ref names on Windows/OSX, regardless of file system
* Eliminates file/directory conflicts in ref names
Further context and motivation can be found in background reading:
* Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
* Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
* First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
* Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
* First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
* libgit2 support issue: https://github.com/libgit2/libgit2/issues
* GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 5 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 401 +++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 27 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 +++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 286 +++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 708 +++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 ++++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 394 ++++++++++++++ reftable/slice.c | 199 ++++++++ reftable/slice.h | 39 ++ reftable/stack.c | 983 +++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 58 +++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 623 ++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ 34 files changed, 6265 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
reftable/LICENSE new file +31 -0
@@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
reftable/README.md new file +19 -0
@@ -0,0 +1,19 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + echo '/* empty */' > reftable/config.h + rm reftable/*_test.c reftable/test_framework.* + git add reftable/*.[ch] + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +5 -0
@@ -0,0 +1,5 @@ +commit 6115b50fdb9bc662be39b05f5589bc109282ae7f +Author: Han-Wen Nienhuys <[email protected]> +Date: Mon Feb 10 13:59:52 2020 +0100 + + README: add a note about the Java implementation
reftable/basics.c new file +196 -0
196 lines not on this page · raw message
reftable/basics.h new file +37 -0
@@ -0,0 +1,37 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +401 -0
401 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/config.h new file +1 -0
@@ -0,0 +1 @@ +/* empty */
reftable/constants.h new file +27 -0
@@ -0,0 +1,27 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define SHA1_SIZE 20 +#define SHA256_SIZE 32 +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +97 -0
97 lines not on this page · raw message
reftable/file.c new file +97 -0
97 lines not on this page · raw message
reftable/iter.c new file +229 -0
229 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +286 -0
286 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/pq.c new file +114 -0
114 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +708 -0
708 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1107 -0
485 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/reftable.h new file +394 -0
394 lines not on this page · raw message
reftable/slice.c new file +199 -0
199 lines not on this page · raw message
reftable/stack.c new file +983 -0
983 lines not on this page · raw message
reftable/system.h new file +58 -0
58 lines not on this page · raw message
reftable/tree.c new file +66 -0
66 lines not on this page · raw message
reftable/writer.c new file +623 -0
623 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
reftable/zlib-compat.c new file +92 -0
92 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
PATCH v5 5/5 Reftable support for git-core
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
For background, see the previous commit introducing the library.
TODO:
* Make CI on gitgitgadget compile.
* Redo interaction between repo config, repo storage version
* Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
Example use:
$ ~/vc/git/git init --reftable
warning: templates not found in /usr/local/google/home/hanwen/share/git-core/templates
Initialized empty Git repository in /tmp/qz/.git/
$ echo q > a
$ ~/vc/git/git add a
$ ~/vc/git/git commit -mx
fatal: not a git repository (or any of the parent directories): .git
[master (root-commit) 373d969] x
1 file changed, 1 insertion(+)
create mode 100644 a
$ ~/vc/git/git show-ref
373d96972fca9b63595740bba3898a762778ba20 HEAD
373d96972fca9b63595740bba3898a762778ba20 refs/heads/master
$ ls -l .git/reftable/
total 12
-rw------- 1 hanwen primarygroup 126 Jan 23 20:08 000000000001-000000000001.ref
-rw------- 1 hanwen primarygroup 4277 Jan 23 20:08 000000000002-000000000002.ref
$ go run ~/vc/reftable/cmd/dump.go -table /tmp/qz/.git/reftable/000000000002-000000000002.ref
** DEBUG **
name /tmp/qz/.git/reftable/000000000002-000000000002.ref, sz 4209: 'r' reftable.readerOffsets{Present:true, Offset:0x0, IndexOffset:0x0}, 'o' reftable.readerOffsets{Present:false, Offset:0x0, IndexOffset:0x0} 'g' reftable.readerOffsets{Present:true, Offset:0x1000, IndexOffset:0x0}
** REFS **
reftable.RefRecord{RefName:"refs/heads/master", UpdateIndex:0x2, Value:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, TargetValue:[]uint8(nil), Target:""}
** LOGS **
reftable.LogRecord{RefName:"HEAD", UpdateIndex:0x2, New:[]uint8{0x37, 0x3d, 0x96, 0x97, 0x2f, 0xca, 0x9b, 0x63, 0x59, 0x57, 0x40, 0xbb, 0xa3, 0x89, 0x8a, 0x76, 0x27, 0x78, 0xba, 0x20}, Old:[]uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Name:"Han-Wen Nienhuys", Email:"[email protected]", Time:0x5e29ef27, TZOffset:100, Message:"commit (initial): x\n"}Signed-off-by Han-Wen Nienhuys <[email protected]>
Co-authored-by Jeff King <[email protected]>
--- .../technical/repository-version.txt | 7 + Makefile | 24 +- builtin/clone.c | 4 +- builtin/init-db.c | 55 +- cache.h | 4 +- refs.c | 20 +- refs.h | 3 + refs/refs-internal.h | 1 + refs/reftable-backend.c | 880 ++++++++++++++++++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- 12 files changed, 986 insertions(+), 29 deletions(-) create mode 100644 refs/reftable-backend.c
Documentation/technical/repository-version.txt +7 -0
@@ -100,3 +100,10 @@ If set, by default "git config" reads from both "config" and multiple working directory mode, "config" file is shared while "config.worktree" is per-working directory (i.e., it's in GIT_COMMON_DIR/worktrees/<id>/config.worktree) + +==== `refStorage` + +Specifies the file format for the ref database. Values are `files` +(for the traditional packed + loose ref format) and `reftable` for the +binary reftable format. See https://github.com/google/reftable for +more information.
Makefile +23 -1
@@ -814,6 +814,7 @@ TEST_SHELL_PATH = $(SHELL_PATH) LIB_FILE = libgit.a XDIFF_LIB = xdiff/lib.a VCSSVN_LIB = vcs-svn/lib.a +REFTABLE_LIB = reftable/libreftable.a GENERATED_H += command-list.h
@@ -959,6 +960,7 @@ LIB_OBJS += rebase-interactive.o LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
@@ -1163,7 +1165,7 @@ THIRD_PARTY_SOURCES += compat/regex/% THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% -GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) +GITLIBS = common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) EXTLIBS = GIT_USER_AGENT = git/$(GIT_VERSION)
@@ -2347,11 +2349,28 @@ VCSSVN_OBJS += vcs-svn/fast_export.o VCSSVN_OBJS += vcs-svn/svndiff.o VCSSVN_OBJS += vcs-svn/svndump.o +REFTABLE_OBJS += reftable/basics.o +REFTABLE_OBJS += reftable/block.o +REFTABLE_OBJS += reftable/bytes.o +REFTABLE_OBJS += reftable/file.o +REFTABLE_OBJS += reftable/iter.o +REFTABLE_OBJS += reftable/merged.o +REFTABLE_OBJS += reftable/pq.o +REFTABLE_OBJS += reftable/reader.o +REFTABLE_OBJS += reftable/record.o +REFTABLE_OBJS += reftable/slice.o +REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/tree.o +REFTABLE_OBJS += reftable/writer.o +REFTABLE_OBJS += reftable/zlib-compat.o + + TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ $(FUZZ_OBJS) \ + $(REFTABLE_OBJS) \ common-main.o \ git.o ifndef NO_CURL
@@ -2488,6 +2507,9 @@ $(XDIFF_LIB): $(XDIFF_OBJS) $(VCSSVN_LIB): $(VCSSVN_OBJS) $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ +$(REFTABLE_LIB): $(REFTABLE_OBJS) + $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ + export DEFAULT_EDITOR DEFAULT_PAGER Documentation/GIT-EXCLUDED-PROGRAMS: FORCE
builtin/clone.c +3 -1
@@ -1096,7 +1096,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } } -init_db (git_dir, real_git_dir, option_template,INIT_DB_QUIET ); +init_db (git_dir, real_git_dir, option_template, +DEFAULT_REF_STORAGE ,/* XXX */ +INIT_DB_QUIET );if (real_git_dir) git_dir = real_git_dir;
builtin/init-db.c +40 -15
125 lines not on this page · raw message
cache.h +3 -1
@@ -627,7 +627,8 @@ int path_inside_repo(const char *prefix, const char *path);#define INIT_DB_EXIST_OK 0x0002int init_db (const char *git_dir,const char *real_git_dir, -const char *template_dir,unsigned int flags); +const char *template_dir,const char *ref_storage_format, +unsigned int flags);void sanitize_stdfds (void );int daemonize (void );
@@ -1041,6 +1042,7 @@ struct repository_format {int is_bare;int hash_algo;char *work_tree; +char *ref_storage;struct string_list unknown_extensions; };
refs.c +12 -8
@@ -20,7 +20,7 @@/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1836,13 +1836,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,* Create, record, and return a ref_store instance for the specified * gitdir. */ -static struct ref_store *ref_store_init (const char *gitdir, +static struct ref_store *ref_store_init (const char *gitdir,const char *be_name,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct ref_storage_be *be;struct ref_store *refs; + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
@@ -1858,7 +1858,10 @@ struct ref_store *get_main_ref_store(struct repository *r)if (!r->gitdir)BUG ("attempting to get main_ref_store outside of repository" ); - r->refs =ref_store_init (r->gitdir,REF_STORE_ALL_CAPS ); + r->refs =ref_store_init (r->gitdir, + r->ref_storage_format ? r->ref_storage_format : +DEFAULT_REF_STORAGE , +REF_STORE_ALL_CAPS );return r->refs; }
@@ -1913,7 +1916,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,DEFAULT_REF_STORAGE ,/* XXX */ REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);
@@ -1927,6 +1930,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format =DEFAULT_REF_STORAGE ;/* XXX */ struct ref_store *refs;const char *id;
@@ -1940,9 +1944,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)if (wt->id) refs =ref_store_init (git_common_path ("worktrees/%s" , wt->id), -REF_STORE_ALL_CAPS ); + format,REF_STORE_ALL_CAPS );else - refs =ref_store_init (get_git_common_dir (), + refs =ref_store_init (get_git_common_dir (), format,REF_STORE_ALL_CAPS );if (refs)
refs.h +3 -0
@@ -9,6 +9,9 @@ struct string_list;struct string_list_item ;struct worktree ; +/* XXX where should this be? */ +#define DEFAULT_REF_STORAGE "files" +/* * Resolve a reference, recursively following symbolic refererences. *
refs/refs-internal.h +1 -0
@@ -661,6 +661,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +880 -0
880 lines not on this page · raw message
repository.c +2 -0
@@ -174,6 +174,8 @@ int repo_init(struct repository *repo,if (worktree)repo_set_worktree (repo, worktree); + repo->ref_storage_format =xstrdup_or_null (format.ref_storage); +clear_repository_format (&format);return 0 ;
repository.h +3 -0
@@ -70,6 +70,9 @@ struct repository {/* The store in which the refs are held. */ struct ref_store *refs; +/* The format to use for the ref database. */ +char *ref_storage_format; +/* * Contains path to often used file names. */
setup.c +9 -3
@@ -440,9 +440,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata)if (!value)return config_error_nonbool (var); data->partial_clone =xstrdup (value); - }else if (!strcmp (ext,"worktreeconfig" )) + }else if (!strcmp (ext,"worktreeconfig" )) { data->worktree_config =git_config_bool (var, value); -else + }else if (!strcmp (ext,"refstorage" )) { + data->ref_storage =xstrdup (value); + }else string_list_append (&data->unknown_extensions, ext); }
@@ -531,6 +533,7 @@ void clear_repository_format(struct repository_format *format)string_list_clear (&format->unknown_extensions,0 );free (format->work_tree);free (format->partial_clone); +free (format->ref_storage);init_repository_format (format); }
@@ -1173,8 +1176,11 @@ const char *setup_git_directory_gently(int *nongit_ok) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ;setup_git_env (gitdir); } -if (startup_info->have_repository) +if (startup_info->have_repository) {repo_set_hash_algo (the_repository, repo_fmt.hash_algo); + the_repository->ref_storage_format = +xstrdup_or_null (repo_fmt.ref_storage); + } }strbuf_release (&dir);
Signature · 1 line
gitgitgadget
v6 cover Reftable support git-core
In reply to Han-Wen Nienhuys via GitGitGadgetThis adds the reftable library, and hooks it up as a ref backend.
Feedback wanted:
* spots marked with XXX in the Git source code.
* how to write a unittest script? From the command-line, it works; inside the test framework, it says:
fatal: could not open '/usr/local/google/home/hanwen/vc/git/t/trash directory.t0031-reftable/.git/refs/heads' for writing: Is a directory
ie. the files backend already initialized.
* what is a good test strategy? Could we have a CI flavor where we flip the default to reftable, and see how it fares?
v6
* implement reflog expiry.
v7
* support SHA256 as version 2 of the format.
Han-Wen Nienhuys (5):
refs.h: clarify reflog iteration order
create .git/refs in files-backend.c
refs: document how ref_iterator_advance_fn should handle symrefs Add reftable library
Reftable support for git-core
.../technical/repository-version.txt | 7 + Makefile | 24 +- builtin/clone.c | 4 +- builtin/init-db.c | 57 +- cache.h | 4 +- refs.c | 20 +- refs.h | 8 +- refs/files-backend.c | 6 + refs/refs-internal.h | 6 + refs/reftable-backend.c | 971 +++++++++++++++ reftable/LICENSE | 31 + reftable/README.md | 19 + reftable/VERSION | 11 + reftable/basics.c | 196 +++ reftable/basics.h | 37 + reftable/block.c | 413 ++++++ reftable/block.h | 71 ++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 25 + reftable/dump.c | 97 ++ reftable/file.c | 97 ++ reftable/iter.c | 229 ++++ reftable/iter.h | 56 + reftable/merged.c | 290 +++++ reftable/merged.h | 34 + reftable/pq.c | 114 ++ reftable/pq.h | 34 + reftable/reader.c | 718 +++++++++++ reftable/reader.h | 52 + reftable/record.c | 1107 +++++++++++++++++ reftable/record.h | 79 ++ reftable/reftable.h | 405 ++++++ reftable/slice.c | 199 +++ reftable/slice.h | 39 + reftable/stack.c | 984 +++++++++++++++ reftable/stack.h | 40 + reftable/system.h | 59 + reftable/tree.c | 66 + reftable/tree.h | 24 + reftable/writer.c | 630 ++++++++++ reftable/writer.h | 46 + reftable/zlib-compat.c | 92 ++ repository.c | 2 + repository.h | 3 + setup.c | 12 +- 47 files changed, 7407 insertions(+), 32 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
base-commit: e68e29171cc2d6968902e0654b5687fbe1ccb903
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-539%2Fhanwen%2Freftable-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-539/hanwen/reftable-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/539
Range-diff vs v5:
1: c00403c94de = 1: e4b0773becd refs.h: clarify reflog iteration order
2: 4d6da9bc473 = 2: c25b6c601dc create .git/refs in files-backend.c
3: fbdcdccc885 = 3: e132e0f4e00 refs: document how ref_iterator_advance_fn should handle symrefs
4: 546b82fe798 ! 4: fe29a9db399 Add reftable library
@@ -86,7 +86,7 @@
+ cp reftable-repo/LICENSE reftable/ &&
+ git --git-dir reftable-repo/.git show --no-patch origin/master \
+ > reftable/VERSION && \
-+ echo '/* empty */' > reftable/config.h
++ sed -i~ 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' reftable/system.h
+ rm reftable/*_test.c reftable/test_framework.*
+ git add reftable/*.[ch]
+
@@ -98,11 +98,17 @@
--- /dev/null
+++ b/reftable/VERSION
@@
-+commit 6115b50fdb9bc662be39b05f5589bc109282ae7f
++commit 42aa04f301a682197f0698867b9771b78ce4fdaf
+Author: Han-Wen Nienhuys <[email protected]>
-+Date: Mon Feb 10 13:59:52 2020 +0100
-+
-+ README: add a note about the Java implementation
++Date: Mon Feb 17 16:05:22 2020 +0100
++
++ C: simplify system and config header handling
++
++ reftable.h only depends on <stdint.h>
++
++ system.h is only included from .c files. The git-core specific code is
++ guarded by a #if REFTABLE_IN_GITCORE, which we'll edit with a sed
++ script on the git-core side.
diff --git a/reftable/basics.c b/reftable/basics.c
new file mode 100644
@@ -485,19 +491,31 @@
+ if (block_writer_type(w) == BLOCK_TYPE_LOG) {
+ int block_header_skip = 4 + w->header_off;
+ struct slice compressed = {};
-+ uLongf dest_len = 0, src_len = 0;
-+ slice_resize(&compressed, w->next - block_header_skip);
++ int zresult = 0;
++ uLongf src_len = w->next - block_header_skip;
++ slice_resize(&compressed, src_len);
+
-+ dest_len = compressed.len;
-+ src_len = w->next - block_header_skip;
++ while (1) {
++ uLongf dest_len = compressed.len;
+
-+ if (Z_OK != compress2(compressed.buf, &dest_len,
-+ w->buf + block_header_skip, src_len, 9)) {
-+ free(slice_yield(&compressed));
-+ return ZLIB_ERROR;
++ zresult = compress2(compressed.buf, &dest_len,
++ w->buf + block_header_skip, src_len,
++ 9);
++ if (zresult == Z_BUF_ERROR) {
++ slice_resize(&compressed, 2 * compressed.len);
++ continue;
++ }
++
++ if (Z_OK != zresult) {
++ free(slice_yield(&compressed));
++ return ZLIB_ERROR;
++ }
++
++ memcpy(w->buf + block_header_skip, compressed.buf,
++ dest_len);
++ w->next = dest_len + block_header_skip;
++ break;
+ }
-+ memcpy(w->buf + block_header_skip, compressed.buf, dest_len);
-+ w->next = dest_len + block_header_skip;
+ }
+ return w->next;
+}
@@ -885,8 +903,6 @@
+#ifndef CONSTANTS_H
+#define CONSTANTS_H
+
-+#define SHA1_SIZE 20
-+#define SHA256_SIZE 32
+#define VERSION 1
+#define HEADER_SIZE 24
+#define FOOTER_SIZE 68
@@ -1555,13 +1571,17 @@
+ it->ops = &merged_iter_vtable;
+}
+
-+int new_merged_table(struct merged_table **dest, struct reader **stack, int n)
++int new_merged_table(struct merged_table **dest, struct reader **stack, int n,
++ int hash_size)
+{
+ uint64_t last_max = 0;
+ uint64_t first_min = 0;
+ int i = 0;
+ for (i = 0; i < n; i++) {
+ struct reader *r = stack[i];
++ if (reader_hash_size(r) != hash_size) {
++ return FORMAT_ERROR;
++ }
+ if (i > 0 && last_max >= reader_min_update_index(r)) {
+ return FORMAT_ERROR;
+ }
@@ -1578,7 +1598,7 @@
+ .stack_len = n,
+ .min = first_min,
+ .max = last_max,
-+ .hash_size = SHA1_SIZE,
++ .hash_size = hash_size,
+ };
+
+ *dest = calloc(sizeof(struct merged_table), 1);
@@ -1985,6 +2005,11 @@
+ block_source_return_block(r->source, p);
+}
+
++int reader_hash_size(struct reader *r)
++{
++ return r->hash_size;
++}
++
+const char *reader_name(struct reader *r)
+{
+ return r->name;
@@ -2005,8 +2030,13 @@
+ goto exit;
+ }
+
++ r->hash_size = SHA1_SIZE;
+ {
+ byte version = *f++;
++ if (version == 2) {
++ r->hash_size = SHA256_SIZE;
++ version = 1;
++ }
+ if (version != 1) {
+ err = FORMAT_ERROR;
+ goto exit;
@@ -2070,7 +2100,7 @@
+ r->size = block_source_size(source) - FOOTER_SIZE;
+ r->source = source;
+ r->name = strdup(name);
-+ r->hash_size = SHA1_SIZE;
++ r->hash_size = 0;
+
+ err = block_source_read_block(source, &footer, r->size, FOOTER_SIZE);
+ if (err != FOOTER_SIZE) {
@@ -3883,7 +3913,7 @@
+#ifndef REFTABLE_H
+#define REFTABLE_H
+
-+#include "system.h"
++#include <stdint.h>
+
+/* block_source is a generic wrapper for a seekable readable file.
+ It is generally passed around by value.
@@ -3934,6 +3964,10 @@
+
+ /* how often to write complete keys in each block. */
+ int restart_interval;
++
++ /* width of the hash. Should be 20 for SHA1 or 32 for SHA256. Defaults
++ * to SHA1 if unset */
++ int hash_size;
+};
+
+/* ref_record holds a ref database entry target_value */
@@ -4147,6 +4181,9 @@
+ */
+int reader_seek_ref(struct reader *r, struct iterator *it, const char *name);
+
++/* returns the hash size used in this table. */
++int reader_hash_size(struct reader *r);
++
+/* seek to logs for the given name, older than update_index. */
+int reader_seek_log_at(struct reader *r, struct iterator *it, const char *name,
+ uint64_t update_index);
@@ -4173,7 +4210,11 @@
+/* new_merged_table creates a new merged table. It takes ownership of the stack
+ array.
+*/
-+int new_merged_table(struct merged_table **dest, struct reader **stack, int n);
++int new_merged_table(struct merged_table **dest, struct reader **stack, int n,
++ int hash_size);
++
++/* returns the hash size used in this merged table. */
++int merged_hash_size(struct merged_table *mt);
+
+/* returns an iterator positioned just before 'name' */
+int merged_table_seek_ref(struct merged_table *mt, struct iterator *it,
@@ -4688,7 +4729,8 @@
+ }
+
+ /* success! */
-+ err = new_merged_table(&new_merged, new_tables, new_tables_len);
++ err = new_merged_table(&new_merged, new_tables, new_tables_len,
++ st->config.hash_size);
+ if (err < 0) {
+ goto exit;
+ }
@@ -5084,7 +5126,7 @@
+ writer_set_limits(wr, st->merged->stack[first]->min_update_index,
+ st->merged->stack[last]->max_update_index);
+
-+ err = new_merged_table(&mt, subtabs, subtabs_len);
++ err = new_merged_table(&mt, subtabs, subtabs_len, st->config.hash_size);
+ if (err < 0) {
+ free(subtabs);
+ goto exit;
@@ -5568,9 +5610,7 @@
+#ifndef SYSTEM_H
+#define SYSTEM_H
+
-+#include "config.h"
-+
-+#ifndef REFTABLE_STANDALONE
++#if 1 /* REFTABLE_IN_GITCORE */
+
+#include "git-compat-util.h"
+#include <zlib.h>
@@ -5606,7 +5646,7 @@
+ memcpy(&a, &b, sizeof(a)); \
+ memcpy(&b, &tmp[0], sizeof(a)); \
+ }
-+#endif
++#endif /* REFTABLE_IN_GITCORE */
+
+typedef uint8_t byte;
+typedef int bool;
@@ -5614,6 +5654,9 @@
+int uncompress_return_consumed(Bytef *dest, uLongf *destLen,
+ const Bytef *source, uLong *sourceLen);
+
++#define SHA1_SIZE 20
++#define SHA256_SIZE 32
++
+#endif
diff --git a/reftable/tree.c b/reftable/tree.c
@@ -5788,6 +5831,9 @@
+ opts->restart_interval = 16;
+ }
+
++ if (opts->hash_size == 0) {
++ opts->hash_size = SHA1_SIZE;
++ }
+ if (opts->block_size == 0) {
+ opts->block_size = DEFAULT_BLOCK_SIZE;
+ }
@@ -5796,7 +5842,8 @@
+static int writer_write_header(struct writer *w, byte *dest)
+{
+ memcpy((char *)dest, "REFT", 4);
-+ dest[4] = 1; /* version */
++ dest[4] = (w->hash_size == SHA1_SIZE) ? 1 : 2; /* version */
++
+ put_u24(dest + 5, w->opts.block_size);
+ put_u64(dest + 8, w->min_update_index);
+ put_u64(dest + 16, w->max_update_index);
@@ -5825,7 +5872,7 @@
+ /* TODO - error return? */
+ abort();
+ }
-+ wp->hash_size = SHA1_SIZE;
++ wp->hash_size = opts->hash_size;
+ wp->block = calloc(opts->block_size, 1);
+ wp->write = writer_func;
+ wp->write_arg = writer_arg;
@@ -6222,7 +6269,10 @@
+ byte footer[68];
+ byte *p = footer;
+
-+ writer_finish_public_section(w);
++ int err = writer_finish_public_section(w);
++ if (err != 0) {
++ return err;
++ }
+
+ writer_write_header(w, footer);
+ p += 24;
5: 702fb89871e ! 5: 8d95ab52f75 Reftable support for git-core
@@ -6,10 +6,8 @@
TODO:
- * Make CI on gitgitgadget compile.
- * Redo interaction between repo config, repo storage version
- * Resolve the design problem with reflog expiry.
* Resolve spots marked with XXX
+ * Test strategy?
Example use:
@@ -411,11 +409,6 @@ $^
+
+#include "../reftable/reftable.h"
+
-+/*
-+ The reftable v1 spec only supports 20-byte binary SHA1s. A new format version
-+ will be needed to support SHA256.
-+ */
-+
+extern struct ref_storage_be refs_be_reftable;
+
+struct reftable_ref_store {
@@ -470,6 +463,7 @@ $^
+ struct ref_store *ref_store = (struct ref_store *)refs;
+ struct write_options cfg = {
+ .block_size = 4096,
++ .hash_size = the_hash_algo->rawsz,
+ };
+ struct strbuf sb = STRBUF_INIT;
+
@@ -676,6 +670,7 @@ $^
+ (struct reftable_ref_store *)transaction->ref_store;
+ uint64_t ts = stack_next_update_index(refs->stack);
+ int err = 0;
++ struct log_record *logs = calloc(transaction->nr, sizeof(*logs));
+ struct ref_update **sorted =
+ malloc(transaction->nr * sizeof(struct ref_update *));
+ COPY_ARRAY(sorted, transaction->updates, transaction->nr);
@@ -695,6 +690,14 @@ $^
+
+ for (int i = 0; i < transaction->nr; i++) {
+ struct ref_update *u = sorted[i];
++ struct log_record *log = &logs[i];
++ fill_log_record(log);
++ log->ref_name = (char *)u->refname;
++ log->old_hash = u->old_oid.hash;
++ log->new_hash = u->new_oid.hash;
++ log->update_index = ts;
++ log->message = u->msg;
++
+ if (u->flags & REF_HAVE_NEW) {
+ struct object_id out_oid = {};
+ int out_flags = 0;
@@ -706,6 +709,7 @@ $^
+ struct ref_record ref = {};
+ ref.ref_name =
+ (char *)(resolved ? resolved : u->refname);
++ log->ref_name = ref.ref_name;
+ ref.value = u->new_oid.hash;
+ ref.update_index = ts;
+ err = writer_add_ref(writer, &ref);
@@ -716,23 +720,15 @@ $^
+ }
+
+ for (int i = 0; i < transaction->nr; i++) {
-+ struct ref_update *u = sorted[i];
-+ struct log_record log = {};
-+ fill_log_record(&log);
-+
-+ log.ref_name = (char *)u->refname;
-+ log.old_hash = u->old_oid.hash;
-+ log.new_hash = u->new_oid.hash;
-+ log.update_index = ts;
-+ log.message = u->msg;
-+
-+ err = writer_add_log(writer, &log);
-+ clear_log_record(&log);
++ err = writer_add_log(writer, &logs[i]);
++ clear_log_record(&logs[i]);
+ if (err < 0) {
-+ return err;
++ goto exit;
+ }
+ }
++
+exit:
++ free(logs);
+ free(sorted);
+ return err;
+}
@@ -1076,7 +1072,7 @@ $^
+ 1);
+ ri->base.oid = &ri->oid;
+
-+ return empty_ref_iterator_begin();
++ return (struct ref_iterator *)ri;
+}
+
+static int
@@ -1214,6 +1210,53 @@ $^
+ return 0;
+}
+
++struct reflog_expiry_arg {
++ struct reftable_ref_store *refs;
++ struct log_record *tombstones;
++ int len;
++ int cap;
++};
++
++static void clear_log_tombstones(struct reflog_expiry_arg *arg)
++{
++ int i = 0;
++ for (; i < arg->len; i++) {
++ log_record_clear(&arg->tombstones[i]);
++ }
++
++ FREE_AND_NULL(arg->tombstones);
++}
++
++static void add_log_tombstone(struct reflog_expiry_arg *arg,
++ const char *refname, uint64_t ts)
++{
++ struct log_record tombstone = {
++ .ref_name = xstrdup(refname),
++ .update_index = ts,
++ };
++ if (arg->len == arg->cap) {
++ arg->cap = 2 * arg->cap + 1;
++ arg->tombstones =
++ realloc(arg->tombstones, arg->cap * sizeof(tombstone));
++ }
++ arg->tombstones[arg->len++] = tombstone;
++}
++
++static int write_reflog_expiry_table(struct writer *writer, void *argv)
++{
++ struct reflog_expiry_arg *arg = (struct reflog_expiry_arg *)argv;
++ uint64_t ts = stack_next_update_index(arg->refs->stack);
++ int i = 0;
++ writer_set_limits(writer, ts, ts);
++ for (i = 0; i < arg->len; i++) {
++ int err = writer_add_log(writer, &arg->tombstones[i]);
++ if (err) {
++ return err;
++ }
++ }
++ return 0;
++}
++
+static int reftable_reflog_expire(struct ref_store *ref_store,
+ const char *refname,
+ const struct object_id *oid,
@@ -1223,11 +1266,57 @@ $^
+ reflog_expiry_cleanup_fn cleanup_fn,
+ void *policy_cb_data)
+{
-+ BUG("per ref reflog expiry is not implemented for reftable backend.");
+ /*
-+ TODO: should iterate over the reflog, and write tombstones. The
-+ tombstones will be removed on compaction.
-+ */
++ For log expiry, we write tombstones in place of the expired entries,
++ This means that the entries are still retrievable by delving into the
++ stack, and expiring entries paradoxically takes extra memory.
++
++ This memory is only reclaimed when some operation issues a
++ reftable_pack_refs(), which will compact the entire stack and get rid
++ of deletion entries.
++
++ It would be better if the refs backend supported an API that sets a
++ criterion for all refs, passing the criterion to pack_refs().
++ */
++ struct reftable_ref_store *refs =
++ (struct reftable_ref_store *)ref_store;
++ struct merged_table *mt = stack_merged_table(refs->stack);
++ struct reflog_expiry_arg arg = {
++ .refs = refs,
++ };
++ struct log_record log = {};
++ struct iterator it = {};
++ int err = merged_table_seek_log(mt, &it, refname);
++ if (err < 0) {
++ return err;
++ }
++
++ while (1) {
++ struct object_id ooid = {};
++ struct object_id noid = {};
++
++ int err = iterator_next_log(it, &log);
++ if (err < 0) {
++ return err;
++ }
++
++ if (err > 0 || strcmp(log.ref_name, refname)) {
++ break;
++ }
++ hashcpy(ooid.hash, log.old_hash);
++ hashcpy(noid.hash, log.new_hash);
++
++ if (should_prune_fn(&ooid, &noid, log.email,
++ (timestamp_t)log.time, log.tz_offset,
++ log.message, policy_cb_data)) {
++ add_log_tombstone(&arg, refname, log.update_index);
++ }
++ }
++ log_record_clear(&log);
++ iterator_destroy(&it);
++ err = stack_add(refs->stack, &write_reflog_expiry_table, &arg);
++ clear_log_tombstones(&arg);
++ return err;
+}
+
+static int reftable_read_raw_ref(struct ref_store *ref_store,Signature · 1 line
gitgitgadget
PATCH v6 1/5 refs.h: clarify reflog iteration order
From: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
refs.h +4 -1
@@ -444,18 +444,21 @@ int delete_refs(const char *msg, struct string_list *refnames,int refs_delete_reflog (struct ref_store *refs,const char *refname);int delete_reflog (const char *refname); -/* iterate over reflog entries */ +/* Iterate over reflog entries. */ typedef int each_reflog_ent_fn (struct object_id *old_oid,struct object_id *new_oid,const char *committer,timestamp_t timestamp,int tz,const char *msg,void *cb_data); +/* Iterate in over reflog entries, oldest entry first. */ int refs_for_each_reflog_ent (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data);int refs_for_each_reflog_ent_reverse (struct ref_store *refs,const char *refname,each_reflog_ent_fn fn,void *cb_data); + +/* Call a function for each reflog entry, oldest entry first. */ int for_each_reflog_ent (const char *refname,each_reflog_ent_fn fn,void *cb_data);int for_each_reflog_ent_reverse (const char *refname,each_reflog_ent_fn fn,void *cb_data);
Signature · 1 line
gitgitgadget
PATCH v6 2/5 create .git/refs in files-backend.c
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
This prepares for supporting the reftable format, which will want create its own file system layout in .git
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- builtin/init-db.c | 2 -- refs/files-backend.c | 6 ++++++ 2 files changed, 6 insertions(+), 2 deletions(-)
builtin/init-db.c +0 -2
@@ -226,8 +226,6 @@ static int create_default_files(const char *template_path,* We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository. */ -safe_create_dir (git_path ("refs" ),1 ); -adjust_shared_perm (git_path ("refs" ));if (refs_init_db (&err))die ("failed to set up refs db: %s" , err.buf);
refs/files-backend.c +6 -0
@@ -3157,9 +3157,15 @@ static int files_init_db(struct ref_store *ref_store, struct strbuf *err)files_downcast (ref_store,REF_STORE_WRITE ,"init_db" );struct strbuf sb =STRBUF_INIT ; +files_ref_path (refs, &sb,"refs" ); +safe_create_dir (sb.buf,1 ); +/* adjust permissions even if directory already exists. */ +adjust_shared_perm (sb.buf); +/* * Create .git/refs/{heads,tags} */ +strbuf_reset (&sb);files_ref_path (refs, &sb,"refs/heads" );safe_create_dir (sb.buf,1 );
Signature · 1 line
gitgitgadget
PATCH v6 3/5 refs: document how ref_iterator_advance_fn should handle symrefs
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- refs/refs-internal.h | 5 +++++ 1 file changed, 5 insertions(+)
refs/refs-internal.h +5 -0
@@ -438,6 +438,11 @@ void base_ref_iterator_free(struct ref_iterator *iter);/* Virtual function declarations for ref_iterators: */ +/* +* backend-specific implementation of ref_iterator_advance. +* For symrefs, the function should set REF_ISSYMREF, and it should also dereference +* the symref to provide the OID referent. +*/ typedef int ref_iterator_advance_fn (struct ref_iterator *ref_iterator); typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,
Signature · 1 line
gitgitgadget
PATCH v6 4/5 Add reftable library
In reply to Han-Wen Nienhuys via GitGitGadgetFrom: Han-Wen Nienhuys <[email protected]>
Reftable is a new format for storing the ref database. It provides the following benefits:
* Simple and fast atomic ref transactions, including multiple refs and reflogs.
* Compact storage of ref data.
* Fast look ups of ref data.
* Case-sensitive ref names on Windows/OSX, regardless of file system
* Eliminates file/directory conflicts in ref names
Further context and motivation can be found in background reading:
* Spec: https://github.com/eclipse/jgit/blob/master/Documentation/technical/reftable.md
* Original discussion on JGit-dev: https://www.eclipse.org/lists/jgit-dev/msg03389.html
* First design discussion on git@vger: https://public-inbox.org/git/CAJo=hJtTp2eA3z9wW9cHo-nA7kK40vVThqh6inXpbCcqfdMP9g@mail.gmail.com/
* Last design discussion on git@vger: https://public-inbox.org/git/CAJo=hJsZcAM9sipdVr7TMD-FD2V2W6_pvMQ791EGCDsDkQ033w@mail.gmail.com/
* First attempt at implementation: https://public-inbox.org/git/CAP8UFD0PPZSjBnxCA7ez91vBuatcHKQ+JUWvTD1iHcXzPBjPBg@mail.gmail.com/
* libgit2 support issue: https://github.com/libgit2/libgit2/issues
* GitLab support issue: https://gitlab.com/gitlab-org/git/issues/6
* go-git support issue: https://github.com/src-d/go-git/issues/1059
Signed-off-by Han-Wen Nienhuys <[email protected]>
--- reftable/LICENSE | 31 ++ reftable/README.md | 19 + reftable/VERSION | 11 + reftable/basics.c | 196 +++++++ reftable/basics.h | 37 ++ reftable/block.c | 413 +++++++++++++++ reftable/block.h | 71 +++ reftable/blocksource.h | 20 + reftable/bytes.c | 0 reftable/config.h | 1 + reftable/constants.h | 25 + reftable/dump.c | 97 ++++ reftable/file.c | 97 ++++ reftable/iter.c | 229 +++++++++ reftable/iter.h | 56 ++ reftable/merged.c | 290 +++++++++++ reftable/merged.h | 34 ++ reftable/pq.c | 114 +++++ reftable/pq.h | 34 ++ reftable/reader.c | 718 ++++++++++++++++++++++++++ reftable/reader.h | 52 ++ reftable/record.c | 1107 ++++++++++++++++++++++++++++++++++++++++ reftable/record.h | 79 +++ reftable/reftable.h | 405 +++++++++++++++ reftable/slice.c | 199 ++++++++ reftable/slice.h | 39 ++ reftable/stack.c | 984 +++++++++++++++++++++++++++++++++++ reftable/stack.h | 40 ++ reftable/system.h | 59 +++ reftable/tree.c | 66 +++ reftable/tree.h | 24 + reftable/writer.c | 630 +++++++++++++++++++++++ reftable/writer.h | 46 ++ reftable/zlib-compat.c | 92 ++++ 34 files changed, 6315 insertions(+) create mode 100644 reftable/LICENSE create mode 100644 reftable/README.md create mode 100644 reftable/VERSION create mode 100644 reftable/basics.c create mode 100644 reftable/basics.h create mode 100644 reftable/block.c create mode 100644 reftable/block.h create mode 100644 reftable/blocksource.h create mode 100644 reftable/bytes.c create mode 100644 reftable/config.h create mode 100644 reftable/constants.h create mode 100644 reftable/dump.c create mode 100644 reftable/file.c create mode 100644 reftable/iter.c create mode 100644 reftable/iter.h create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/reader.c create mode 100644 reftable/reader.h create mode 100644 reftable/record.c create mode 100644 reftable/record.h create mode 100644 reftable/reftable.h create mode 100644 reftable/slice.c create mode 100644 reftable/slice.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/system.h create mode 100644 reftable/tree.c create mode 100644 reftable/tree.h create mode 100644 reftable/writer.c create mode 100644 reftable/writer.h create mode 100644 reftable/zlib-compat.c
reftable/LICENSE new file +31 -0
@@ -0,0 +1,31 @@ +BSD License + +Copyright (c) 2020, Google LLC +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer in the +documentation and/or other materials provided with the distribution. + +* Neither the name of Google LLC nor the names of its contributors may +be used to endorse or promote products derived from this software +without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
reftable/README.md new file +19 -0
@@ -0,0 +1,19 @@ + +The source code in this directory comes from https://github.com/google/reftable. + +The VERSION file keeps track of the current version of the reftable library. + +To update the library, do: + + ((cd reftable-repo && git fetch origin && git checkout origin/master ) || + git clone https://github.com/google/reftable reftable-repo) && \ + cp reftable-repo/c/*.[ch] reftable/ && \ + cp reftable-repo/LICENSE reftable/ && + git --git-dir reftable-repo/.git show --no-patch origin/master \ + > reftable/VERSION && \ + sed -i~ 's|if REFTABLE_IN_GITCORE|if 1 /* REFTABLE_IN_GITCORE */|' reftable/system.h + rm reftable/*_test.c reftable/test_framework.* + git add reftable/*.[ch] + +Bugfixes should be accompanied by a test and applied to upstream project at +https://github.com/google/reftable.
reftable/VERSION new file +11 -0
@@ -0,0 +1,11 @@ +commit 42aa04f301a682197f0698867b9771b78ce4fdaf +Author: Han-Wen Nienhuys <[email protected]> +Date: Mon Feb 17 16:05:22 2020 +0100 + + C: simplify system and config header handling + + reftable.h only depends on <stdint.h> + + system.h is only included from .c files. The git-core specific code is + guarded by a #if REFTABLE_IN_GITCORE, which we'll edit with a sed + script on the git-core side.
reftable/basics.c new file +196 -0
196 lines not on this page · raw message
reftable/basics.h new file +37 -0
@@ -0,0 +1,37 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BASICS_H +#define BASICS_H + +#include "system.h" + +#include "reftable.h" + +#define true 1 +#define false 0 + +void put_u24 (byte *out,uint32_t i); +uint32_t get_u24 (byte *in); + +uint64_t get_u64 (byte *in); +void put_u64 (byte *out,uint64_t i); + +void put_u32 (byte *out,uint32_t i); +uint32_t get_u32 (byte *in); + +void put_u16 (byte *out,uint16_t i); +uint16_t get_u16 (byte *in); +int binsearch (int sz,int (*f)(int k,void *args),void *args); + +void free_names (char **a); +void parse_names (char *buf,int size,char ***namesp); +int names_equal (char **a,char **b); +int names_length (char **names); + +#endif
reftable/block.c new file +413 -0
413 lines not on this page · raw message
reftable/block.h new file +71 -0
71 lines not on this page · raw message
reftable/blocksource.h new file +20 -0
@@ -0,0 +1,20 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef BLOCKSOURCE_H +#define BLOCKSOURCE_H + +#include "reftable.h" + +uint64_t block_source_size (struct block_source source); +int block_source_read_block (struct block_source source,struct block *dest, +uint64_t off,uint32_t size); +void block_source_return_block (struct block_source source,struct block *ret); +void block_source_close (struct block_source source); + +#endif
reftable/bytes.c new file +0 -0
reftable/config.h new file +1 -0
@@ -0,0 +1 @@ +/* empty */
reftable/constants.h new file +25 -0
@@ -0,0 +1,25 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef CONSTANTS_H +#define CONSTANTS_H + +#define VERSION 1 +#define HEADER_SIZE 24 +#define FOOTER_SIZE 68 + +#define BLOCK_TYPE_LOG 'g' +#define BLOCK_TYPE_INDEX 'i' +#define BLOCK_TYPE_REF 'r' +#define BLOCK_TYPE_OBJ 'o' +#define BLOCK_TYPE_ANY 0 + +#define MAX_RESTARTS ((1 << 16) - 1) +#define DEFAULT_BLOCK_SIZE 4096 + +#endif
reftable/dump.c new file +97 -0
97 lines not on this page · raw message
reftable/file.c new file +97 -0
97 lines not on this page · raw message
reftable/iter.c new file +229 -0
229 lines not on this page · raw message
reftable/iter.h new file +56 -0
56 lines not on this page · raw message
reftable/merged.c new file +290 -0
290 lines not on this page · raw message
reftable/merged.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" +#include "reftable.h" + +struct merged_table { +struct reader **stack; +int stack_len; +int hash_size; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct iterator *stack; +int hash_size; +int stack_len; +byte typ; +struct merged_iter_pqueue pq; +} merged_iter; + +void merged_table_clear (struct merged_table *mt); + +#endif
reftable/pq.c new file +114 -0
114 lines not on this page · raw message
reftable/pq.h new file +34 -0
@@ -0,0 +1,34 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +struct record rec; +int index; +}; + +int pq_less (struct pq_entry a,struct pq_entry b); + +struct merged_iter_pqueue { +struct pq_entry *heap; +int len; +int cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +bool merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_clear (struct merged_iter_pqueue *pq); + +#endif
reftable/reader.c new file +718 -0
718 lines not on this page · raw message
reftable/reader.h new file +52 -0
52 lines not on this page · raw message
reftable/record.c new file +1107 -0
455 lines not on this page · raw message
reftable/record.h new file +79 -0
79 lines not on this page · raw message
reftable/reftable.h new file +405 -0
405 lines not on this page · raw message
reftable/slice.c new file +199 -0
199 lines not on this page · raw message
reftable/stack.c new file +984 -0
984 lines not on this page · raw message
reftable/system.h new file +59 -0
59 lines not on this page · raw message
reftable/tree.c new file +66 -0
66 lines not on this page · raw message
reftable/writer.c new file +630 -0
630 lines not on this page · raw message
reftable/writer.h new file +46 -0
46 lines not on this page · raw message
reftable/zlib-compat.c new file +92 -0
92 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
gitgitgadget
Part of the diff is not shown here. View the raw message.
Re: [PATCH v6 4/5] Add reftable library
reftable/basics.c line 47
Quoted 8 lines · Han-Wen Nienhuys via GitGitGadget
reftable/basics.c new file +8
-0+void put_u64 (byte *out,uint64_t v) +{ +int i =0 ; +for (i =sizeof (uint64_t ); i--;) { + out[i] = (byte)(v &0xff ); + v >>=8 ; + } +}
This looks OK, but ...
reftable/basics.c line 122
Quoted 6 lines
reftable/basics.c new file +6
-0+int names_length (char **names) +{ +int len =0 ; +for (char **p = names; *p; p++) { + len++; + }
... this will break the build with some compiler options (see for example https://travis-ci.org/git/git/jobs/651322628#L679). You may also want to lose {braces} around a single-statement block.
The incremental change between v5 and v6, especially the reflog expiration bit, looked sensible.
Thanks.
Re: [PATCH v6 4/5] Add reftable library
Quoted 12 lines · Junio C Hamano
reftable/basics.c line 47
Quoted 8 lines · Han-Wen Nienhuys via GitGitGadget
reftable/basics.c new file +8
-0+void put_u64 (byte *out,uint64_t v) +{ +int i =0 ; +for (i =sizeof (uint64_t ); i--;) { + out[i] = (byte)(v &0xff ); + v >>=8 ; + } +}This looks OK, but ...
There's a useless initialization of "i" there, I think?
It also seems weird to decrement "i" in the loop condition and leave the final field of the for-loop empty. Would:
int i = sizeof(uint64_t);
while (i--) {
...
}be more clear? But...
We also have put_be64() already (it's coded without a loop, though I'd imagine just about any compiler would unroll the loop above).
I wonder how possible it would be to use the Git names for these utility functions in the reftable library, and then push the common ones out into a compat file within the reftable library. Then we could toss out the compat file when importing the library to Git, but it could still compile standalone.
-Peff
Re: [PATCH v6 4/5] Add reftable library
Quoted 5 lines · Jeff King
I wonder how possible it would be to use the Git names for these utility functions in the reftable library, and then push the common ones out into a compat file within the reftable library. Then we could toss out the compat file when importing the library to Git, but it could still compile standalone.
I did this for {get,put}_u{64,32} and xstrdup. For some reason, git is missing put_u16.
Let me know if there are other functions I should look at.
Signature · 7 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado