Support reftable ref backend for Git
Subsystems (2)
PATCH v4 14/28 reftable: reftable file level tests
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
With support for reading and writing files in place, we can construct files (in memory) and attempt to read them back.
Because some sections of the format are optional (eg. indices, log entries), we have to exercise this code using multiple sizes of input data
--- Makefile | 1 + reftable/readwrite_test.c | 652 ++++++++++++++++++++++++++++++++++++++ reftable/reftable-tests.h | 2 +- t/helper/test-reftable.c | 1 + 4 files changed, 655 insertions(+), 1 deletion(-) create mode 100644 reftable/readwrite_test.c
Makefile +1 -0
@@ -2469,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o REFTABLE_TEST_OBJS += reftable/basics_test.o REFTABLE_TEST_OBJS += reftable/block_test.o REFTABLE_TEST_OBJS += reftable/record_test.o +REFTABLE_TEST_OBJS += reftable/readwrite_test.o REFTABLE_TEST_OBJS += reftable/test_framework.o REFTABLE_TEST_OBJS += reftable/tree_test.o
reftable/readwrite_test.c new file +652 -0
652 lines not on this page · raw message
reftable/reftable-tests.h +1 -1
@@ -14,7 +14,7 @@ int block_test_main(int argc, const char **argv);int merged_test_main (int argc,const char **argv);int record_test_main (int argc,const char **argv);int refname_test_main (int argc,const char **argv); -int reftable_test_main (int argc,const char **argv); +int readwrite_test_main (int argc,const char **argv);int stack_test_main (int argc,const char **argv);int tree_test_main (int argc,const char **argv);int reftable_dump_main (int argc,char *const *argv);
t/helper/test-reftable.c +1 -0
@@ -6,6 +6,7 @@ int cmd__reftable(int argc, const char **argv)basics_test_main (argc, argv);block_test_main (argc, argv);record_test_main (argc, argv); +readwrite_test_main (argc, argv);tree_test_main (argc, argv);return 0 ; }
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 15/28 reftable: add a heap-based priority queue for reftable records
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
This is needed to create a merged view multiple reftables
--- Makefile | 2 + reftable/pq.c | 115 ++++++++++++++++++++++++++++++++++++++ reftable/pq.h | 32 +++++++++++ reftable/pq_test.c | 72 ++++++++++++++++++++++++ reftable/reftable-tests.h | 1 + t/helper/test-reftable.c | 1 + 6 files changed, 223 insertions(+) create mode 100644 reftable/pq.c create mode 100644 reftable/pq.h create mode 100644 reftable/pq_test.c
Makefile +2 -0
@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o REFTABLE_OBJS += reftable/blocksource.o REFTABLE_OBJS += reftable/iter.o REFTABLE_OBJS += reftable/publicbasics.o +REFTABLE_OBJS += reftable/pq.o REFTABLE_OBJS += reftable/reader.o REFTABLE_OBJS += reftable/record.o REFTABLE_OBJS += reftable/refname.o
@@ -2468,6 +2469,7 @@ REFTABLE_OBJS += reftable/writer.o REFTABLE_TEST_OBJS += reftable/basics_test.o REFTABLE_TEST_OBJS += reftable/block_test.o +REFTABLE_TEST_OBJS += reftable/pq_test.o REFTABLE_TEST_OBJS += reftable/record_test.o REFTABLE_TEST_OBJS += reftable/readwrite_test.o REFTABLE_TEST_OBJS += reftable/test_framework.o
reftable/pq.c new file +115 -0
115 lines not on this page · raw message
reftable/pq.h new file +32 -0
@@ -0,0 +1,32 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef PQ_H +#define PQ_H + +#include "record.h" + +struct pq_entry { +int index; +struct reftable_record rec; +}; + +struct merged_iter_pqueue { +struct pq_entry *heap; +size_t len; +size_t cap; +}; + +struct pq_entry merged_iter_pqueue_top (struct merged_iter_pqueue pq); +int merged_iter_pqueue_is_empty (struct merged_iter_pqueue pq); +void merged_iter_pqueue_check (struct merged_iter_pqueue pq); +struct pq_entry merged_iter_pqueue_remove (struct merged_iter_pqueue *pq); +void merged_iter_pqueue_add (struct merged_iter_pqueue *pq,struct pq_entry e); +void merged_iter_pqueue_release (struct merged_iter_pqueue *pq); + +#endif
reftable/pq_test.c new file +72 -0
72 lines not on this page · raw message
reftable/reftable-tests.h +1 -0
@@ -12,6 +12,7 @@ license that can be found in the LICENSE file or atint basics_test_main (int argc,const char **argv);int block_test_main (int argc,const char **argv);int merged_test_main (int argc,const char **argv); +int pq_test_main (int argc,const char **argv);int record_test_main (int argc,const char **argv);int refname_test_main (int argc,const char **argv);int readwrite_test_main (int argc,const char **argv);
t/helper/test-reftable.c +1 -0
@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv) {basics_test_main (argc, argv);block_test_main (argc, argv); +pq_test_main (argc, argv);record_test_main (argc, argv);readwrite_test_main (argc, argv);tree_test_main (argc, argv);
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 16/28 reftable: add merged table view
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
This adds an abstract, read-only interface to the ref database.
This primitive is used to construct the read view of the ref database (the read view is constructed by merging several *.ref files). It also provides the mechanism to provide a unified view of the refs in the main repository and the per-worktree refs.
--- Makefile | 2 + reftable/merged.c | 362 +++++++++++++++++++++++++++++++++++++ reftable/merged.h | 35 ++++ reftable/merged_test.c | 292 ++++++++++++++++++++++++++++++ reftable/reftable-merged.h | 72 ++++++++ t/helper/test-reftable.c | 1 + 6 files changed, 764 insertions(+) create mode 100644 reftable/merged.c create mode 100644 reftable/merged.h create mode 100644 reftable/merged_test.c create mode 100644 reftable/reftable-merged.h
Makefile +2 -0
@@ -2458,6 +2458,7 @@ REFTABLE_OBJS += reftable/block.o REFTABLE_OBJS += reftable/blocksource.o REFTABLE_OBJS += reftable/iter.o REFTABLE_OBJS += reftable/publicbasics.o +REFTABLE_OBJS += reftable/merged.o REFTABLE_OBJS += reftable/pq.o REFTABLE_OBJS += reftable/reader.o REFTABLE_OBJS += reftable/record.o
@@ -2469,6 +2470,7 @@ REFTABLE_OBJS += reftable/writer.o REFTABLE_TEST_OBJS += reftable/basics_test.o REFTABLE_TEST_OBJS += reftable/block_test.o +REFTABLE_TEST_OBJS += reftable/merged_test.o REFTABLE_TEST_OBJS += reftable/pq_test.o REFTABLE_TEST_OBJS += reftable/record_test.o REFTABLE_TEST_OBJS += reftable/readwrite_test.o
reftable/merged.c new file +362 -0
362 lines not on this page · raw message
reftable/merged.h new file +35 -0
@@ -0,0 +1,35 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#ifndef MERGED_H +#define MERGED_H + +#include "pq.h" + +struct reftable_merged_table { +struct reftable_table *stack; +size_t stack_len; +uint32_t hash_id; +int suppress_deletions; + +uint64_t min; +uint64_t max; +}; + +struct merged_iter { +struct reftable_iterator *stack; +uint32_t hash_id; +size_t stack_len; +uint8_t typ; +int suppress_deletions; +struct merged_iter_pqueue pq; +}; + +void merged_table_release (struct reftable_merged_table *mt); + +#endif
reftable/merged_test.c new file +292 -0
292 lines not on this page · raw message
reftable/reftable-merged.h new file +72 -0
72 lines not on this page · raw message
t/helper/test-reftable.c +1 -0
@@ -5,6 +5,7 @@ int cmd__reftable(int argc, const char **argv) {basics_test_main (argc, argv);block_test_main (argc, argv); +merged_test_main (argc, argv);pq_test_main (argc, argv);record_test_main (argc, argv);readwrite_test_main (argc, argv);
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 17/28 reftable: implement refname validation
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
The packed/loose format has restrictions on refnames: a and a/b cannot coexist. This limitation does not apply to reftable per se, but must be maintained for interoperability. This code adds validation routines to abort transactions that are trying to add invalid names.
--- Makefile | 1 + reftable/refname.c | 209 +++++++++++++++++++++++++++++++++++++++ reftable/refname.h | 29 ++++++ reftable/refname_test.c | 102 +++++++++++++++++++ t/helper/test-reftable.c | 1 + 5 files changed, 342 insertions(+) create mode 100644 reftable/refname.c create mode 100644 reftable/refname.h create mode 100644 reftable/refname_test.c
Makefile +1 -0
@@ -2474,6 +2474,7 @@ REFTABLE_TEST_OBJS += reftable/merged_test.o REFTABLE_TEST_OBJS += reftable/pq_test.o REFTABLE_TEST_OBJS += reftable/record_test.o REFTABLE_TEST_OBJS += reftable/readwrite_test.o +REFTABLE_TEST_OBJS += reftable/refname_test.o REFTABLE_TEST_OBJS += reftable/test_framework.o REFTABLE_TEST_OBJS += reftable/tree_test.o
reftable/refname.c new file +209 -0
209 lines not on this page · raw message
reftable/refname.h new file +29 -0
@@ -0,0 +1,29 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ +#ifndef REFNAME_H +#define REFNAME_H + +#include "reftable-record.h" +#include "reftable-generic.h" + +struct modification { +struct reftable_table tab; + +char **add; +size_t add_len; + +char **del; +size_t del_len; +}; + +int validate_ref_record_addition (struct reftable_table tab, +struct reftable_ref_record *recs,size_t sz); + +int modification_validate (struct modification *mod); + +#endif
reftable/refname_test.c new file +102 -0
102 lines not on this page · raw message
t/helper/test-reftable.c +1 -0
@@ -8,6 +8,7 @@ int cmd__reftable(int argc, const char **argv)merged_test_main (argc, argv);pq_test_main (argc, argv);record_test_main (argc, argv); +refname_test_main (argc, argv);readwrite_test_main (argc, argv);tree_test_main (argc, argv);return 0 ;
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 18/28 reftable: implement stack, a mutable database of reftable files.
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
--- Makefile | 1 + reftable/reftable-stack.h | 128 ++++ reftable/stack.c | 1396 +++++++++++++++++++++++++++++++++++++ reftable/stack.h | 41 ++ reftable/stack_test.c | 948 +++++++++++++++++++++++++ t/helper/test-reftable.c | 1 + 6 files changed, 2515 insertions(+) create mode 100644 reftable/reftable-stack.h create mode 100644 reftable/stack.c create mode 100644 reftable/stack.h create mode 100644 reftable/stack_test.c
Makefile +1 -0
@@ -2475,6 +2475,7 @@ REFTABLE_TEST_OBJS += reftable/pq_test.o REFTABLE_TEST_OBJS += reftable/record_test.o REFTABLE_TEST_OBJS += reftable/readwrite_test.o REFTABLE_TEST_OBJS += reftable/refname_test.o +REFTABLE_TEST_OBJS += reftable/stack_test.o REFTABLE_TEST_OBJS += reftable/test_framework.o REFTABLE_TEST_OBJS += reftable/tree_test.o
reftable/reftable-stack.h new file +128 -0
128 lines not on this page · raw message
reftable/stack.c new file +1396 -0
1,396 lines not on this page · raw message
reftable/stack.h new file +41 -0
41 lines not on this page · raw message
reftable/stack_test.c new file +948 -0
948 lines not on this page · raw message
t/helper/test-reftable.c +1 -0
@@ -10,6 +10,7 @@ int cmd__reftable(int argc, const char **argv)record_test_main (argc, argv);refname_test_main (argc, argv);readwrite_test_main (argc, argv); +stack_test_main (argc, argv);tree_test_main (argc, argv);return 0 ; }
Signature · 1 line
2.33.0.662.gbaddc25a55e
Re: [PATCH v4 18/28] reftable: implement stack, a mutable database of reftable files.
This step does not compile under -pedantic because it assumes that __FUNCTION__ is available unconditionally (unlike trace.h where it allows __FUNCTION__ to be used iff compiler supports it).
Here is a workaround that needs to be split and squashed into the steps that introduce these test source files.
Subject: [PATCH] SQUASH???
https://github.com/git/git/runs/3439941236?check_suite_focus=true#step:5:700
--- reftable/basics_test.c | 10 ++--- reftable/block_test.c | 2 +- reftable/merged_test.c | 6 +-- reftable/pq_test.c | 2 +- reftable/readwrite_test.c | 30 +++++++-------- reftable/record_test.c | 18 ++++----- reftable/refname_test.c | 2 +- reftable/stack_test.c | 80 +++++++++++++++++++-------------------- reftable/test_framework.h | 2 +- reftable/tree_test.c | 2 +- 10 files changed, 77 insertions(+), 77 deletions(-)
reftable/basics_test.c +5 -5
@@ -24,7 +24,7 @@ static int binsearch_func(size_t i, void *void_args)return args->key < args->arr[i]; } -static void test_binsearch (void ) +static void test_binsearch (const char *fn_name) {int arr[] = {2 ,4 ,6 ,8 ,10 };size_t sz =ARRAY_SIZE (arr);
@@ -49,13 +49,13 @@ static void test_binsearch(void) } } -static void test_names_length (void ) +static void test_names_length (const char *fn_name) {char *a[] = {"a" ,"b" ,NULL };EXPECT (names_length (a) ==2 ); } -static void test_parse_names_normal (void ) +static void test_parse_names_normal (const char *fn_name) {char in[] ="a\nb\n" ;char **out =NULL ;
@@ -66,7 +66,7 @@ static void test_parse_names_normal(void)free_names (out); } -static void test_parse_names_drop_empty (void ) +static void test_parse_names_drop_empty (const char *fn_name) {char in[] ="a\n\n" ;char **out =NULL ;
@@ -76,7 +76,7 @@ static void test_parse_names_drop_empty(void)free_names (out); } -static void test_common_prefix (void ) +static void test_common_prefix (const char *fn_name) {struct strbuf s1 =STRBUF_INIT ;struct strbuf s2 =STRBUF_INIT ;
reftable/block_test.c +1 -1
@@ -16,7 +16,7 @@ license that can be found in the LICENSE file or at#include "test_framework.h" #include "reftable-tests.h" -static void test_block_read_write (void ) +static void test_block_read_write (const char *fn_name) {const int header_off =21 ;/* random */ char *names[30 ];
reftable/merged_test.c +3 -3
@@ -95,7 +95,7 @@ static void readers_destroy(struct reftable_reader **readers, size_t n)reftable_free (readers); } -static void test_merged_between (void ) +static void test_merged_between (const char *fn_name) {uint8_t hash1[GIT_SHA1_RAWSZ ] = {1 ,2 ,3 ,0 };
@@ -137,7 +137,7 @@ static void test_merged_between(void)reftable_free (bs); } -static void test_merged (void ) +static void test_merged (const char *fn_name) {uint8_t hash1[GIT_SHA1_RAWSZ ] = {1 };uint8_t hash2[GIT_SHA1_RAWSZ ] = {2 };
@@ -237,7 +237,7 @@ static void test_merged(void)reftable_free (bs); } -static void test_default_write_opts (void ) +static void test_default_write_opts (const char *fn_name) {struct reftable_write_options opts = {0 };struct strbuf buf =STRBUF_INIT ;
reftable/pq_test.c +1 -1
@@ -15,7 +15,7 @@ license that can be found in the LICENSE file or at#include "reftable-tests.h" #include "test_framework.h" -static void test_pq (void ) +static void test_pq (const char *fn_name) {char *names[54 ] = {NULL };int N =ARRAY_SIZE (names) -1 ;
reftable/readwrite_test.c +15 -15
@@ -20,7 +20,7 @@ license that can be found in the LICENSE file or atstatic const int update_index =5 ; -static void test_buffer (void ) +static void test_buffer (const char *fn_name) {struct strbuf buf =STRBUF_INIT ;struct reftable_block_source source = {NULL };
@@ -114,7 +114,7 @@ static void write_table(char ***names, struct strbuf *buf, int N,reftable_writer_free (w); } -static void test_log_buffer_size (void ) +static void test_log_buffer_size (const char *fn_name) {struct strbuf buf =STRBUF_INIT ;struct reftable_write_options opts = {
@@ -155,7 +155,7 @@ static void test_log_buffer_size(void)strbuf_release (&buf); } -static void test_log_write_read (void ) +static void test_log_write_read (const char *fn_name) {int N =2 ;char **names =reftable_calloc (sizeof (char *) * (N +1 ));
@@ -254,7 +254,7 @@ static void test_log_write_read(void)reader_close (&rd); } -static void test_table_read_write_sequential (void ) +static void test_table_read_write_sequential (const char *fn_name) {char **names;struct strbuf buf =STRBUF_INIT ;
@@ -296,7 +296,7 @@ static void test_table_read_write_sequential(void)reader_close (&rd); } -static void test_table_write_small_table (void ) +static void test_table_write_small_table (const char *fn_name) {char **names;struct strbuf buf =STRBUF_INIT ;
@@ -307,7 +307,7 @@ static void test_table_write_small_table(void)free_names (names); } -static void test_table_read_api (void ) +static void test_table_read_api (const char *fn_name) {char **names;struct strbuf buf =STRBUF_INIT ;
@@ -406,17 +406,17 @@ static void test_table_read_write_seek(int index, int hash_id)reader_close (&rd); } -static void test_table_read_write_seek_linear (void ) +static void test_table_read_write_seek_linear (const char *fn_name) {test_table_read_write_seek (0 ,GIT_SHA1_FORMAT_ID ); } -static void test_table_read_write_seek_linear_sha256 (void ) +static void test_table_read_write_seek_linear_sha256 (const char *fn_name) {test_table_read_write_seek (0 ,GIT_SHA256_FORMAT_ID ); } -static void test_table_read_write_seek_index (void ) +static void test_table_read_write_seek_index (const char *fn_name) {test_table_read_write_seek (1 ,GIT_SHA1_FORMAT_ID ); }
@@ -522,17 +522,17 @@ static void test_table_refs_for(int indexed)reader_close (&rd); } -static void test_table_refs_for_no_index (void ) +static void test_table_refs_for_no_index (const char *fn_name) {test_table_refs_for (0 ); } -static void test_table_refs_for_obj_index (void ) +static void test_table_refs_for_obj_index (const char *fn_name) {test_table_refs_for (1 ); } -static void test_write_empty_table (void ) +static void test_write_empty_table (const char *fn_name) {struct reftable_write_options opts = {0 };struct strbuf buf =STRBUF_INIT ;
@@ -568,7 +568,7 @@ static void test_write_empty_table(void)strbuf_release (&buf); } -static void test_write_key_order (void ) +static void test_write_key_order (const char *fn_name) {struct reftable_write_options opts = {0 };struct strbuf buf =STRBUF_INIT ;
@@ -604,7 +604,7 @@ static void test_write_key_order(void)strbuf_release (&buf); } -static void test_corrupt_table_empty (void ) +static void test_corrupt_table_empty (const char *fn_name) {struct strbuf buf =STRBUF_INIT ;struct reftable_block_source source = {NULL };
@@ -616,7 +616,7 @@ static void test_corrupt_table_empty(void)EXPECT (err ==REFTABLE_FORMAT_ERROR ); } -static void test_corrupt_table (void ) +static void test_corrupt_table (const char *fn_name) {uint8_t zeros[1024 ] = {0 };struct strbuf buf =STRBUF_INIT ;
reftable/record_test.c +9 -9
@@ -36,7 +36,7 @@ static void test_copy(struct reftable_record *rec)reftable_record_destroy (©); } -static void test_varint_roundtrip (void ) +static void test_varint_roundtrip (const char *fn_name) {uint64_t inputs[] = {0 ,1 ,
@@ -68,7 +68,7 @@ static void test_varint_roundtrip(void) } } -static void test_common_prefix (void ) +static void test_common_prefix (const char *fn_name) {struct {const char *a, *b;
@@ -101,7 +101,7 @@ static void set_hash(uint8_t *h, int j) } } -static void test_reftable_ref_record_roundtrip (void ) +static void test_reftable_ref_record_roundtrip (const char *fn_name) {int i =0 ;
@@ -164,7 +164,7 @@ static void test_reftable_ref_record_roundtrip(void) } } -static void test_reftable_log_record_equal (void ) +static void test_reftable_log_record_equal (const char *fn_name) {struct reftable_log_record in[2 ] = { {
@@ -184,7 +184,7 @@ static void test_reftable_log_record_equal(void)reftable_log_record_release (&in[1 ]); } -static void test_reftable_log_record_roundtrip (void ) +static void test_reftable_log_record_roundtrip (const char *fn_name) {int i;struct reftable_log_record in[2 ] = {
@@ -258,7 +258,7 @@ static void test_reftable_log_record_roundtrip(void) } } -static void test_u24_roundtrip (void ) +static void test_u24_roundtrip (const char *fn_name) {uint32_t in =0x112233 ;uint8_t dest[3 ];
@@ -268,7 +268,7 @@ static void test_u24_roundtrip(void)EXPECT (in == out); } -static void test_key_roundtrip (void ) +static void test_key_roundtrip (const char *fn_name) {uint8_t buffer[1024 ] = {0 };struct string_view dest = {
@@ -300,7 +300,7 @@ static void test_key_roundtrip(void)strbuf_release (&roundtrip); } -static void test_reftable_obj_record_roundtrip (void ) +static void test_reftable_obj_record_roundtrip (const char *fn_name) {uint8_t testHash1[GIT_SHA1_RAWSZ ] = {1 ,2 ,3 ,4 ,0 };uint64_t till9[] = {1 ,2 ,3 ,4 ,500 ,600 ,700 ,800 ,9000 };
@@ -358,7 +358,7 @@ static void test_reftable_obj_record_roundtrip(void) } } -static void test_reftable_index_record_roundtrip (void ) +static void test_reftable_index_record_roundtrip (const char *fn_name) {struct reftable_index_record in = { .offset =42 ,
reftable/refname_test.c +1 -1
@@ -26,7 +26,7 @@ struct testcase {int error_code; }; -static void test_conflict (void ) +static void test_conflict (const char *fn_name) {struct reftable_write_options opts = {0 };struct strbuf buf =STRBUF_INIT ;
reftable/stack_test.c +40 -40
266 lines not on this page · raw message
reftable/test_framework.h +1 -1
@@ -42,7 +42,7 @@ license that can be found in the LICENSE file or at#define RUN_TEST (f) \ fprintf(stderr, "running %s\n", #f); \ fflush(stderr); \ - f(); + f(#f);void set_test_hash (uint8_t *p,int i);
reftable/tree_test.c +1 -1
@@ -31,7 +31,7 @@ static void check_increasing(void *arg, void *key) c->last = key; } -static void test_tree (void ) +static void test_tree (const char *fn_name) {struct tree_node *root =NULL ;
Signature · 1 line
2.33.0-205-g1b5b9d966b
RFC PATCH 1/1 reftable: fixup for broken __FUNCTION__ use
The use of the __FUNCTION__ macro is non-standard, in this case all we're doing with it is generating a prettier unique filename based on the name of the function that makes it, presumably to make ad-hoc debugging easier.
For mkdtemp() and mkstemp() we don't need to pass anything like this in, since the "XXXXXX" part of the template will ensure that we get a unique filename, but to make finding what function created what tempfile easy let's just use __LINE__ here, it's not *as easy*, but at least this one uses standard behavior.
This can be tested under DEVOPTS=pedantic, i.e. before this change we'd emit errors like:
reftable/stack_test.c: In function ‘test_read_file’:
reftable/stack_test.c:67:30: error: ISO C does not support ‘__FUNCTION__’ predefined identifier [-Werror=pedantic]
char *fn = get_tmp_template(__FUNCTION__);
The current tip of "seen" is broken as a result, see
https://github.com/git/git/runs/3439941236
---
Quoted 9 lines · Junio C Hamano
This step does not compile under -pedantic because it assumes that
__FUNCTION__is available unconditionally (unliketrace.hwhere it allows__FUNCTION__to be used iff compiler supports it).Here is a workaround that needs to be split and squashed into the steps that introduce these test source files.
Subject: [PATCH] SQUASH???
https://github.com/git/git/runs/3439941236?check_suite_focus=true#step:5:700
First, thanks for the re-re-arrangement of the errno+reftable topics. It looks like the "seen" integration is good, except for this issue under -pedantic.
I can confirm your fix works, for what it's worth I came up with this alternate approach that I was about to send before I saw your proposed fixup.
It's smaller because it punts on the whole notion of adding the function name to the filename, as argued above I think __LINE__ should be sufficient here (and is probably already overkill). The only reason to add __FUNCTION__ or __LINE__ to the filename is presumably for one-off ad-hoc debugging.
I see you pushed out seen a few minutes ago as bce8679d69 with your proposed squash. Let's leave that in and not have the churn of re-replacing that fixup. I'm submitting this more for Han-Wen's consideration at this point.
reftable/stack_test.c | 43 ++++++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 21 deletions(-)
reftable/stack_test.c +22 -21
@@ -46,25 +46,26 @@ static int count_dir_entries(const char *dirname)return len; } -static char *get_tmp_template (const char *prefix) +static char *get_tmp_template (const int line) {const char *tmp =getenv ("TMPDIR" );static char template[1024 ]; -snprintf (template,sizeof (template) -1 ,"%s/%s.XXXXXX" , - tmp ? tmp :"/tmp" , prefix); +snprintf (template,sizeof (template) -1 ,"%s/stack-test.c-%d.XXXXXX" , + tmp ? tmp :"/tmp" , line); +fprintf (stderr,"have template %s\n" , template);return template; } -static char *get_tmp_dir (const char *prefix) +static char *get_tmp_dir (const int line) { -char *dir =get_tmp_template (prefix); +char *dir =get_tmp_template (line);EXPECT (mkdtemp (dir));return dir; }static void test_read_file (void ) { -char *fn =get_tmp_template (__FUNCTION__); +char *fn =get_tmp_template (__LINE__);int fd =mkstemp (fn);char out[1024 ] ="line1\n\nline2\nline3" ;int n, err;
@@ -133,7 +134,7 @@ static int write_test_log(struct reftable_writer *wr, void *arg)static void test_reftable_stack_add_one (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -174,7 +175,7 @@ static void test_reftable_stack_uptodate(void)struct reftable_write_options cfg = {0 };struct reftable_stack *st1 =NULL ;struct reftable_stack *st2 =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);int err;struct reftable_ref_record ref1 = {
@@ -218,7 +219,7 @@ static void test_reftable_stack_uptodate(void)static void test_reftable_stack_transaction_api (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -265,7 +266,7 @@ static void test_reftable_stack_validate_refname(void)struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;int err; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);int i;struct reftable_ref_record ref = {
@@ -305,7 +306,7 @@ static int write_error(struct reftable_writer *wr, void *arg)static void test_reftable_stack_update_index_check (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -337,7 +338,7 @@ static void test_reftable_stack_update_index_check(void)static void test_reftable_stack_lock_failure (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -362,7 +363,7 @@ static void test_reftable_stack_add(void) .exact_log_message = 1, };struct reftable_stack *st =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_ref_record refs[2 ] = { {NULL } };struct reftable_log_record logs[2 ] = { {NULL } };
@@ -443,7 +444,7 @@ static void test_reftable_stack_log_normalize(void) 0, };struct reftable_stack *st =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);uint8_t h1[GIT_SHA1_RAWSZ ] = {0x01 }, h2[GIT_SHA1_RAWSZ ] = {0x02 };
@@ -495,7 +496,7 @@ static void test_reftable_stack_log_normalize(void)static void test_reftable_stack_tombstone (void ) {int i =0 ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -577,7 +578,7 @@ static void test_reftable_stack_tombstone(void)static void test_reftable_stack_hash_id (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -685,7 +686,7 @@ static void test_suggest_compaction_segment_nothing(void)static void test_reflog_expire (void ) { -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;
@@ -766,7 +767,7 @@ static void test_empty_add(void)struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ;int err; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);struct reftable_stack *st2 =NULL ;
@@ -788,7 +789,7 @@ static void test_reftable_stack_auto_compaction(void) {struct reftable_write_options cfg = {0 };struct reftable_stack *st =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);int err, i;int N =100 ;
@@ -823,7 +824,7 @@ static void test_reftable_stack_compaction_concurrent(void) {struct reftable_write_options cfg = {0 };struct reftable_stack *st1 =NULL , *st2 =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);int err, i;int N =3 ;
@@ -873,7 +874,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) {struct reftable_write_options cfg = {0 };struct reftable_stack *st1 =NULL , *st2 =NULL , *st3 =NULL ; -char *dir =get_tmp_dir (__FUNCTION__); +char *dir =get_tmp_dir (__LINE__);int err, i;int N =3 ;
Signature · 1 line
2.33.0.736.g68690aaec9a
Re: [RFC PATCH] reftable: fixup for broken __FUNCTION__ use
Quoted 2 lines · Ævar Arnfjörð Bjarmason
The use of the
__FUNCTION__macro is non-standard
and is also problematic because it is a "magic" variable[1] instead of a string literal.
FWIW, __func__ has also the same problem but it is part of C99; I still like Junio's solution
better than my own that was using those.
Re: [RFC PATCH] reftable: fixup for broken __FUNCTION__ use
In reply to Ævar Arnfjörð BjarmasonQuoted 5 lines · Ævar Arnfjörð Bjarmason
The use of the
__FUNCTION__macro is non-standard, in this case all we're doing with it is generating a prettier unique filename based on the name of the function that makes it, presumably to make ad-hoc debugging easier.
..
Quoted 5 lines
It's smaller because it punts on the whole notion of adding the function name to the filename, as argued above I think
__LINE__should be sufficient here (and is probably already overkill). The only reason to add__FUNCTION__or__LINE__to the filename is presumably for one-off ad-hoc debugging.
It's there so it's easy to track down which tests forget to cleanup after they run, and __LINE__ works for this as well.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
PATCH v4 19/28 reftable: add dump utility
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
provide a command-line utility for inspecting individual tables, and inspecting a complete ref database
--- reftable/dump.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 reftable/dump.c
reftable/dump.c new file +107 -0
@@ -0,0 +1,107 @@ +/* +Copyright 2020 Google LLC + +Use of this source code is governed by a BSD-style +license that can be found in the LICENSE file or at +https://developers.google.com/open-source/licenses/bsd +*/ + +#include "git-compat-util.h" +#include "hash.h" + +#include "reftable-blocksource.h" +#include "reftable-error.h" +#include "reftable-merged.h" +#include "reftable-record.h" +#include "reftable-tests.h" +#include "reftable-writer.h" +#include "reftable-iterator.h" +#include "reftable-reader.h" +#include "reftable-stack.h" +#include "reftable-generic.h" + +#include <stddef.h> +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> +#include <string.h> + +static int compact_stack (const char *stackdir) +{ +struct reftable_stack *stack =NULL ; +struct reftable_write_options cfg = {0 }; + +int err =reftable_new_stack (&stack, stackdir, cfg); +if (err <0 ) + goto done; + + err =reftable_stack_compact_all (stack,NULL ); +if (err <0 ) + goto done; +done: +if (stack) { +reftable_stack_destroy (stack); + } +return err; +} + +static void print_help (void ) +{ +printf ("usage: dump [-cst] arg\n\n" +"options: \n" +" -c compact\n" +" -t dump table\n" +" -s dump stack\n" +" -6 sha256 hash format\n" +" -h this help\n" +"\n" ); +} + +int reftable_dump_main (int argc,char *const *argv) +{ +int err =0 ; +int opt_dump_table =0 ; +int opt_dump_stack =0 ; +int opt_compact =0 ; +uint32_t opt_hash_id =GIT_SHA1_FORMAT_ID ; +const char *arg =NULL , *argv0 = argv[0 ]; + +for (; argc >1 ; argv++, argc--) +if (*argv[1 ] !='-' ) +break ; +else if (!strcmp ("-t" , argv[1 ])) + opt_dump_table =1 ; +else if (!strcmp ("-6" , argv[1 ])) + opt_hash_id =GIT_SHA256_FORMAT_ID ; +else if (!strcmp ("-s" , argv[1 ])) + opt_dump_stack =1 ; +else if (!strcmp ("-c" , argv[1 ])) + opt_compact =1 ; +else if (!strcmp ("-?" , argv[1 ]) || !strcmp ("-h" , argv[1 ])) { +print_help (); +return 2 ; + } + +if (argc !=2 ) { +fprintf (stderr,"need argument\n" ); +print_help (); +return 2 ; + } + + arg = argv[1 ]; + +if (opt_dump_table) { + err =reftable_reader_print_file (arg); + }else if (opt_dump_stack) { + err =reftable_stack_print_directory (arg, opt_hash_id); + }else if (opt_compact) { + err =compact_stack (arg); + } + +if (err <0 ) { +fprintf (stderr,"%s: %s: %s\n" , argv0, arg, +reftable_error_str (err)); +return 1 ; + } +return 0 ; +}
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 20/28 refs: RFC: Reftable support for git-core
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
For background, see Documentation/technical/reftable.txt.
This introduces the file refs/reftable-backend.c containing a reftable-powered ref storage backend.
It can be activated by setting GIT_TEST_REFTABLE in the environment. When GIT_TEST_REFTABLE is set, the test prerequisite !REFFILES is set.
There is no option to git-init for now, as the test suite still shows failures with GIT_TEST_REFTABLE=1.
Example use: see t/t0031-reftable.sh
--- Documentation/config/extensions.txt | 9 + .../technical/repository-version.txt | 7 + Makefile | 1 + builtin/clone.c | 5 +- builtin/init-db.c | 39 +- builtin/stash.c | 8 +- builtin/worktree.c | 27 +- cache.h | 8 +- config.mak.uname | 2 +- contrib/buildsystems/Generators/Vcxproj.pm | 11 +- contrib/workdir/git-new-workdir | 2 +- refs.c | 26 +- refs.h | 3 + refs/refs-internal.h | 1 + refs/reftable-backend.c | 1691 +++++++++++++++++ repository.c | 2 + repository.h | 3 + setup.c | 8 + t/t0031-reftable.sh | 310 +++ t/t1409-avoid-packing-refs.sh | 6 + t/t1450-fsck.sh | 6 + t/t3210-pack-refs.sh | 6 + t/test-lib.sh | 7 +- 23 files changed, 2159 insertions(+), 29 deletions(-) create mode 100644 refs/reftable-backend.c create mode 100755 t/t0031-reftable.sh
Documentation/config/extensions.txt +9 -0
@@ -6,3 +6,12 @@ extensions.objectFormat:: Note that this setting should only be set by linkgit:git-init[1] or linkgit:git-clone[1]. Trying to change it after initialization will not work and will produce hard-to-diagnose issues. ++ +extensions.refStorage:: + Specify the ref storage mechanism to use. The acceptable values are `files` and + `reftable`. If not specified, `files` is assumed. It is an error to specify + this key unless `core.repositoryFormatVersion` is 1. ++ +Note that this setting should only be set by linkgit:git-init[1] or +linkgit:git-clone[1]. Trying to change it after initialization will not +work and will produce hard-to-diagnose issues.
Documentation/technical/repository-version.txt +7 -0
@@ -100,3 +100,10 @@ If set, by default "git config" reads from both "config" and multiple working directory mode, "config" file is shared while "config.worktree" is per-working directory (i.e., it's in GIT_COMMON_DIR/worktrees/<id>/config.worktree) + +==== `refStorage` + +Specifies the file format for the ref database. Values are `files` +(for the traditional packed + loose ref format) and `reftable` for the +binary reftable format. See https://github.com/google/reftable for +more information.
Makefile +1 -0
@@ -990,6 +990,7 @@ LIB_OBJS += reflog-walk.o LIB_OBJS += refs.o LIB_OBJS += refs/debug.o LIB_OBJS += refs/files-backend.o +LIB_OBJS += refs/reftable-backend.o LIB_OBJS += refs/iterator.o LIB_OBJS += refs/packed-backend.o LIB_OBJS += refs/ref-cache.o
builtin/clone.c +3 -2
@@ -1148,7 +1148,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }init_db (git_dir, real_git_dir, option_template,GIT_HASH_UNKNOWN ,NULL , -INIT_DB_QUIET ); +default_ref_storage (),INIT_DB_QUIET );if (real_git_dir) git_dir = real_git_dir;
@@ -1299,7 +1299,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)* Now that we know what algorithm the remote side is using, * let's set ours to the same thing. */ -initialize_repository_version (hash_algo,1 ); +initialize_repository_version (hash_algo,1 , +default_ref_storage ());repo_set_hash_algo (the_repository, hash_algo); mapped_refs =wanted_peer_refs (refs, &remote->fetch);
builtin/init-db.c +31 -8
@@ -167,12 +167,14 @@ static int needs_work_tree_config(const char *git_dir, const char *work_tree)return 1 ; } -void initialize_repository_version (int hash_algo,int reinit) +void initialize_repository_version (int hash_algo,int reinit, +const char *ref_storage_format) {char repo_version_string[10 ];int repo_version =GIT_REPO_VERSION ; -if (hash_algo !=GIT_HASH_SHA1 ) +if (hash_algo !=GIT_HASH_SHA1 || + !strcmp (ref_storage_format,"reftable" )) repo_version =GIT_REPO_VERSION_READ ;/* This forces creation of new config file */
@@ -226,6 +228,7 @@ static int create_default_files(const char *template_path, is_bare_repository_cfg = init_is_bare_repository || !work_tree;if (init_shared_repository !=-1 )set_shared_repository (init_shared_repository); + the_repository->ref_storage_format =xstrdup (fmt->ref_storage);/* * We would have created the above under user's umask -- under
@@ -235,6 +238,24 @@ static int create_default_files(const char *template_path,adjust_shared_perm (get_git_dir ()); } +/* +* Check to see if .git/HEAD exists; this must happen before +* initializing the ref db, because we want to see if there is an +* existing HEAD. +*/ + path =git_path_buf (&buf,"HEAD" ); + reinit = (!access (path,R_OK ) || +readlink (path, junk,sizeof (junk) -1 ) !=-1 ); + +/* +* refs/heads is a file when using reftable. We can't reinitialize with +* a reftable because it will overwrite HEAD +*/ +if (reinit && (!strcmp (fmt->ref_storage,"reftable" )) == +is_directory (git_path_buf (&buf,"refs/heads" ))) { +die ("cannot switch ref storage format." ); + } +/* * We need to create a "refs" dir in any case so that older * versions of git can tell that this is a repository.
@@ -249,9 +270,6 @@ static int create_default_files(const char *template_path,* Point the HEAD symref to the initial branch with if HEAD does * not yet exist. */ - path =git_path_buf (&buf,"HEAD" ); - reinit = (!access (path,R_OK ) - ||readlink (path, junk,sizeof (junk)-1 ) !=-1 );if (!reinit) {char *ref;
@@ -268,7 +286,7 @@ static int create_default_files(const char *template_path,free (ref); } -initialize_repository_version (fmt->hash_algo,0 ); +initialize_repository_version (fmt->hash_algo,0 , fmt->ref_storage);/* Check filemode trustability */ path =git_path_buf (&buf,"config" );
@@ -383,7 +401,7 @@ static void validate_hash_algorithm(struct repository_format *repo_fmt, int hashint init_db (const char *git_dir,const char *real_git_dir,const char *template_dir,int hash,const char *initial_branch, -unsigned int flags) +const char *ref_storage_format,unsigned int flags) {int reinit;int exist_ok = flags &INIT_DB_EXIST_OK ;
@@ -422,6 +440,7 @@ int init_db(const char *git_dir, const char *real_git_dir,* is an attempt to reinitialize new repository with an old tool. */ check_repository_format (&repo_fmt); + repo_fmt.ref_storage =xstrdup (ref_storage_format);validate_hash_algorithm (&repo_fmt, hash);
@@ -476,6 +495,9 @@ int init_db(const char *git_dir, const char *real_git_dir,git_config_set ("receive.denyNonFastforwards" ,"true" ); } +if (!strcmp (ref_storage_format,"reftable" )) +git_config_set ("extensions.refStorage" , ref_storage_format); +if (!(flags &INIT_DB_QUIET )) {int len =strlen (git_dir);
@@ -549,6 +571,7 @@ static const char *const init_db_usage[] = {int cmd_init_db (int argc,const char **argv,const char *prefix) {const char *git_dir; +const char *ref_storage_format =default_ref_storage ();const char *real_git_dir =NULL ;const char *work_tree;const char *template_dir =NULL ;
@@ -713,5 +736,5 @@ int cmd_init_db(int argc, const char **argv, const char *prefix) flags |=INIT_DB_EXIST_OK ;return init_db (git_dir, real_git_dir, template_dir, hash_algo, - initial_branch, flags); + initial_branch, ref_storage_format, flags); }
builtin/stash.c +7 -1
@@ -207,10 +207,16 @@ static int get_stash_info(struct stash_info *info, int argc, const char **argv)static int do_clear_stash (void ) {struct object_id obj; +int result;if (get_oid (ref_stash, &obj))return 0 ; -return delete_ref (NULL , ref_stash, &obj,0 ); + result =delete_ref (NULL , ref_stash, &obj,0 ); + +/* Ignore error; this is necessary for reftable, which keeps reflogs +* even when refs are deleted. */ +delete_reflog (ref_stash); +return result; }static int clear_stash (int argc,const char **argv,const char *prefix)
builtin/worktree.c +24 -3
@@ -13,6 +13,7 @@#include "utf8.h" #include "worktree.h" #include "quote.h" +#include "../refs/refs-internal.h" static const char *const worktree_usage[] = {N_ ("git worktree add [<options>] <path> [<commit-ish>]" ),
@@ -328,9 +329,29 @@ static int add_worktree(const char *path, const char *refname,* worktree. */ strbuf_reset (&sb); -strbuf_addf (&sb,"%s/HEAD" , sb_repo.buf); -write_file (sb.buf,"%s" ,oid_to_hex (null_oid ())); -strbuf_reset (&sb); +if (get_main_ref_store (the_repository)->be == &refs_be_reftable) { +/* XXX this is cut & paste from reftable_init_db. */ +strbuf_addf (&sb,"%s/HEAD" , sb_repo.buf); +write_file (sb.buf,"%s" ,"ref: refs/heads/.invalid\n" ); +strbuf_reset (&sb); + +strbuf_addf (&sb,"%s/refs" , sb_repo.buf); +safe_create_dir (sb.buf,1 ); +strbuf_reset (&sb); + +strbuf_addf (&sb,"%s/refs/heads" , sb_repo.buf); +write_file (sb.buf,"this repository uses the reftable format" ); +strbuf_reset (&sb); + +strbuf_addf (&sb,"%s/reftable" , sb_repo.buf); +safe_create_dir (sb.buf,1 ); +strbuf_reset (&sb); + }else { +strbuf_addf (&sb,"%s/HEAD" , sb_repo.buf); +write_file (sb.buf,"%s" ,oid_to_hex (null_oid ())); +strbuf_reset (&sb); + } +strbuf_addf (&sb,"%s/commondir" , sb_repo.buf);write_file (sb.buf,"../.." );
cache.h +5 -3
@@ -656,9 +656,10 @@ int path_inside_repo(const char *prefix, const char *path);#define INIT_DB_EXIST_OK 0x0002int init_db (const char *git_dir,const char *real_git_dir, -const char *template_dir,int hash_algo, -const char *initial_branch,unsigned int flags); -void initialize_repository_version (int hash_algo,int reinit); +const char *template_dir,int hash_algo,const char *initial_branch, +const char *ref_storage_format,unsigned int flags); +void initialize_repository_version (int hash_algo,int reinit, +const char *ref_storage_format);void sanitize_stdfds (void );int daemonize (void );
@@ -1076,6 +1077,7 @@ struct repository_format {int hash_algo;int sparse_index;char *work_tree; +char *ref_storage;struct string_list unknown_extensions;struct string_list v1_only_extensions; };
config.mak.uname +1 -1
@@ -712,7 +712,7 @@ vcxproj: # Make .vcxproj files and add them unset QUIET_GEN QUIET_BUILT_IN; \ perl contrib/buildsystems/generate -g Vcxproj - git add -f git.sln {*,*/lib,t/helper/*}/*.vcxproj + git add -f git.sln {*,*/lib,*/libreftable,t/helper/*}/*.vcxproj # Generate the LinkOrCopyBuiltins.targets and LinkOrCopyRemoteHttp.targets file (echo '<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">' && \
contrib/buildsystems/Generators/Vcxproj.pm +10 -1
@@ -77,7 +77,7 @@ sub createProject { my $libs_release = "\n "; my $libs_debug = "\n "; if (!$static_library) { - $libs_release = join(";", sort(grep /^(?!libgit\.lib|xdiff\/lib\.lib|vcs-svn\/lib\.lib)/, @{$$build_structure{"$prefix${name}_LIBS"}})); + $libs_release = join(";", sort(grep /^(?!libgit\.lib|xdiff\/lib\.lib|vcs-svn\/lib\.lib|reftable\/libreftable\.lib)/, @{$$build_structure{"$prefix${name}_LIBS"}})); $libs_debug = $libs_release; $libs_debug =~ s/zlib\.lib/zlibd\.lib/g; $libs_debug =~ s/libexpat\.lib/libexpatd\.lib/g;
@@ -232,6 +232,7 @@ sub createProject { EOM if (!$static_library || $target =~ 'vcs-svn' || $target =~ 'xdiff') { my $uuid_libgit = $$build_structure{"LIBS_libgit_GUID"}; + my $uuid_libreftable = $$build_structure{"LIBS_reftable/libreftable_GUID"}; my $uuid_xdiff_lib = $$build_structure{"LIBS_xdiff/lib_GUID"}; print F << "EOM";
@@ -241,6 +242,14 @@ sub createProject { <ReferenceOutputAssembly>false</ReferenceOutputAssembly> </ProjectReference> EOM + if (!($name =~ /xdiff|libreftable/)) { + print F << "EOM"; + <ProjectReference Include="$cdup\\reftable\\libreftable\\libreftable.vcxproj"> + <Project>$uuid_libreftable</Project> + <ReferenceOutputAssembly>false</ReferenceOutputAssembly> + </ProjectReference> +EOM + } if (!($name =~ 'xdiff')) { print F << "EOM"; <ProjectReference Include="$cdup\\xdiff\\lib\\xdiff_lib.vcxproj">
contrib/workdir/git-new-workdir +1 -1
@@ -79,7 +79,7 @@ trap cleanup $siglist # create the links to the original repo. explicitly exclude index, HEAD and # logs/HEAD from the list since they are purely related to the current working # directory, and should not be shared. -for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn +for x in config refs logs/refs objects info hooks packed-refs remotes rr-cache svn reftable do # create a containing directory if needed case $x in
refs.c +18 -8
@@ -19,10 +19,15 @@#include "repository.h" #include "sigchain.h" +const char *default_ref_storage (void ) +{ +return git_env_bool ("GIT_TEST_REFTABLE" ,0 ) ?"reftable" :"files" ; +} +/* * List of all available backends */ -static struct ref_storage_be *refs_backends = &refs_be_files; +static struct ref_storage_be *refs_backends = &refs_be_reftable;static struct ref_storage_be *find_ref_storage_backend (const char *name) {
@@ -1895,13 +1900,13 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,* Create, record, and return a ref_store instance for the specified * gitdir. */ -static struct ref_store *ref_store_init (const char *gitdir, +static struct ref_store *ref_store_init (const char *gitdir,const char *be_name,unsigned int flags) { -const char *be_name ="files" ; -struct ref_storage_be *be =find_ref_storage_backend (be_name); +struct ref_storage_be *be;struct ref_store *refs; + be =find_ref_storage_backend (be_name);if (!be)BUG ("reference backend %s is unknown" , be_name);
@@ -1917,7 +1922,11 @@ struct ref_store *get_main_ref_store(struct repository *r)if (!r->gitdir)BUG ("attempting to get main_ref_store outside of repository" ); - r->refs_private =ref_store_init (r->gitdir,REF_STORE_ALL_CAPS ); + r->refs_private =ref_store_init (r->gitdir, + r->ref_storage_format ? + r->ref_storage_format : +default_ref_storage (), +REF_STORE_ALL_CAPS ); r->refs_private =maybe_debug_wrap_ref_store (r->gitdir, r->refs_private);return r->refs_private; }
@@ -1973,7 +1982,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule) goto done;/* assume that add_submodule_odb() has been called */ - refs =ref_store_init (submodule_sb.buf, + refs =ref_store_init (submodule_sb.buf,default_ref_storage (),REF_STORE_READ |REF_STORE_ODB );register_ref_store_map (&submodule_ref_stores,"submodule" , refs, submodule);
@@ -1987,6 +1996,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)struct ref_store *get_worktree_ref_store (const struct worktree *wt) { +const char *format =default_ref_storage ();struct ref_store *refs;const char *id;
@@ -2000,9 +2010,9 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)if (wt->id) refs =ref_store_init (git_common_path ("worktrees/%s" , wt->id), -REF_STORE_ALL_CAPS ); + format,REF_STORE_ALL_CAPS );else - refs =ref_store_init (get_git_common_dir (), + refs =ref_store_init (get_git_common_dir (), format,REF_STORE_ALL_CAPS );if (refs)
refs.h +3 -0
@@ -11,6 +11,9 @@ struct string_list;struct string_list_item ;struct worktree ; +/* Returns the ref storage backend to use by default. */ +const char *default_ref_storage (void ); +/* * Resolve a reference, recursively following symbolic refererences. *
refs/refs-internal.h +1 -0
@@ -676,6 +676,7 @@ struct ref_storage_be { };extern struct ref_storage_be refs_be_files; +extern struct ref_storage_be refs_be_reftable;extern struct ref_storage_be refs_be_packed;/*
refs/reftable-backend.c new file +1691 -0
1,691 lines not on this page · raw message
repository.c +2 -0
@@ -180,6 +180,8 @@ int repo_init(struct repository *repo,if (worktree)repo_set_worktree (repo, worktree); + repo->ref_storage_format =xstrdup_or_null (format.ref_storage); +clear_repository_format (&format);return 0 ;
repository.h +3 -0
@@ -82,6 +82,9 @@ struct repository {*/ struct ref_store *refs_private; +/* The format to use for the ref database. */ +char *ref_storage_format; +/* * Contains path to often used file names. */
setup.c +8 -0
@@ -498,6 +498,9 @@ static enum extension_result handle_extension(const char *var,return error ("invalid value for 'extensions.objectformat'" ); data->hash_algo = format;return EXTENSION_OK ; + }else if (!strcmp (ext,"refstorage" )) { + data->ref_storage =xstrdup (value); +return EXTENSION_OK ; }return EXTENSION_UNKNOWN ; }
@@ -648,6 +651,7 @@ void clear_repository_format(struct repository_format *format)string_list_clear (&format->v1_only_extensions,0 );free (format->work_tree);free (format->partial_clone); +free (format->ref_storage);init_repository_format (format); }
@@ -1312,6 +1316,8 @@ const char *setup_git_directory_gently(int *nongit_ok) the_repository->repository_format_partial_clone = repo_fmt.partial_clone; repo_fmt.partial_clone =NULL ; + the_repository->ref_storage_format = +xstrdup_or_null (repo_fmt.ref_storage); } }/*
@@ -1399,6 +1405,8 @@ void check_repository_format(struct repository_format *fmt)repo_set_hash_algo (the_repository, fmt->hash_algo); the_repository->repository_format_partial_clone =xstrdup_or_null (fmt->partial_clone); +/* XXX why is repo->ref_storage_format set in multiple places?! */ + the_repository->ref_storage_format =xstrdup_or_null (fmt->ref_storage);clear_repository_format (&repo_fmt); }
t/t0031-reftable.sh new file +310 -0
310 lines not on this page · raw message
t/t1409-avoid-packing-refs.sh +6 -0
@@ -4,6 +4,12 @@ test_description='avoid rewriting packed-refs unnecessarily' . ./test-lib.sh +if test_have_prereq !REFFILES +then + skip_all='skipping pack-refs tests; need files backend' + test_done +fi + # Add an identifying mark to the packed-refs file header line. This # shouldn't upset readers, and it should be omitted if the file is # ever rewritten.
t/t1450-fsck.sh +6 -0
@@ -8,6 +8,12 @@ test_description='git fsck random collection of tests . ./test-lib.sh +if test_have_prereq !REFFILES +then + skip_all='skipping tests; incompatible with reftable' + test_done +fi + test_expect_success setup ' git config gc.auto 0 && git config i18n.commitencoding ISO-8859-1 &&
t/t3210-pack-refs.sh +6 -0
@@ -14,6 +14,12 @@ export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME . ./test-lib.sh +if test_have_prereq !REFFILES +then + skip_all='skipping pack-refs tests; requires files ref backend' + test_done +fi + test_expect_success 'enable reflogs' ' git config core.logallrefupdates true '
t/test-lib.sh +6 -1
@@ -1524,7 +1524,12 @@ parisc* | hppa*) ;; esac -test_set_prereq REFFILES +if test -n "$GIT_TEST_REFTABLE" +then + test_set_prereq !REFFILES +else + test_set_prereq REFFILES +fi ( COLUMNS=1 && test $COLUMNS = 1 ) && test_set_prereq COLUMNS_CAN_BE_1 test -z "$NO_PERL" && test_set_prereq PERL
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 21/28 git-prompt: prepare for reftable refs backend
In reply to Ævar Arnfjörð BjarmasonFrom: SZEDER Gábor <[email protected]>
In our git-prompt script we strive to use Bash builtins wherever possible, because fork()-ing subshells for command substitutions and fork()+exec()-ing Git commands are expensive on some platforms. We even read and parse '.git/HEAD' using Bash builtins to get the name of the current branch [1]. However, the upcoming reftable refs backend won't use '.git/HEAD' at all, but will write an invalid refname as placeholder for backwards compatibility instead, which will break our git-prompt script.
Update the git-prompt script to recognize the placeholder '.git/HEAD' written by the reftable backend (its content is specified in the reftable specs), and then fall back to use 'git symbolic-ref' to get the name of the current branch.
[1] 3a43c4b5bd (bash prompt: use bash builtins to find out current
branch, 2011-03-31)
--- contrib/completion/git-prompt.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
contrib/completion/git-prompt.sh +6 -1
@@ -478,10 +478,15 @@ __git_ps1 () if ! __git_eread "$g/HEAD" head; then return $exit fi - # is it a symbolic ref? b="${head#ref: }" if [ "$head" = "$b" ]; then detached=yes + elif [ "$b" = "refs/heads/.invalid" ]; then + # Reftable + b="$(git symbolic-ref HEAD 2>/dev/null)" || + detached=yes + fi + if [ "$detached" = yes ]; then b="$( case "${GIT_PS1_DESCRIBE_STYLE-}" in (contains)
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 22/28 Add "test-tool dump-reftable" command.
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
This command dumps individual tables or a stack of of tables.
--- Makefile | 1 + t/helper/test-reftable.c | 5 +++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t0031-reftable.sh | 6 ++++++ 5 files changed, 14 insertions(+)
Makefile +1 -0
@@ -2471,6 +2471,7 @@ REFTABLE_OBJS += reftable/writer.o REFTABLE_TEST_OBJS += reftable/basics_test.o REFTABLE_TEST_OBJS += reftable/block_test.o +REFTABLE_TEST_OBJS += reftable/dump.o REFTABLE_TEST_OBJS += reftable/merged_test.o REFTABLE_TEST_OBJS += reftable/pq_test.o REFTABLE_TEST_OBJS += reftable/record_test.o
t/helper/test-reftable.c +5 -0
@@ -14,3 +14,8 @@ int cmd__reftable(int argc, const char **argv)tree_test_main (argc, argv);return 0 ; } + +int cmd__dump_reftable (int argc,const char **argv) +{ +return reftable_dump_main (argc, (char *const *)argv); +}
t/helper/test-tool.c +1 -0
@@ -61,6 +61,7 @@ static struct test_cmd cmds[] = { { "read-midx", cmd__read_midx }, { "ref-store", cmd__ref_store }, { "reftable", cmd__reftable }, + { "dump-reftable", cmd__dump_reftable }, { "regex", cmd__regex }, { "repository", cmd__repository }, { "revision-walking", cmd__revision_walking },
t/helper/test-tool.h +1 -0
@@ -19,6 +19,7 @@ int cmd__dump_cache_tree(int argc, const char **argv);int cmd__dump_fsmonitor (int argc,const char **argv);int cmd__dump_split_index (int argc,const char **argv);int cmd__dump_untracked_cache (int argc,const char **argv); +int cmd__dump_reftable (int argc,const char **argv);int cmd__example_decorate (int argc,const char **argv);int cmd__fast_rebase (int argc,const char **argv);int cmd__genrandom (int argc,const char **argv);
t/t0031-reftable.sh +6 -0
@@ -288,6 +288,12 @@ test_expect_success 'FETCH_HEAD' ' test_cmp expect actual ' +test_expect_success 'dump reftable' ' + initialize && + hash_id=$(git config extensions.objectformat) && + test-tool dump-reftable $(test "${hash_id}" = "sha256" && echo "-6") -s .git/reftable +' + . "$TEST_DIRECTORY"/lib-httpd.sh start_httpd
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 23/28 t1301: document what needs to be done for reftable
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
--- t/t1301-shared-repo.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
t/t1301-shared-repo.sh +5 -3
@@ -22,9 +22,10 @@ test_expect_success 'shared = 0400 (faulty permission u-w)' ' ) ' +# TODO(hanwen): for REFTABLE should inspect group-readable of .git/reftable/ for u in 002 022 do - test_expect_success POSIXPERM "shared=1 does not clear bits preset by umask $u" ' + test_expect_success REFFILES,POSIXPERM "shared=1 does not clear bits preset by umask $u" ' mkdir sub && ( cd sub && umask $u &&
@@ -114,7 +115,8 @@ test_expect_success POSIXPERM 'info/refs respects umask in unshared repo' ' test_cmp expect actual ' -test_expect_success POSIXPERM 'git reflog expire honors core.sharedRepository' ' +# For reftable, the check on .git/reftable/ is sufficient. +test_expect_success REFFILES,POSIXPERM 'git reflog expire honors core.sharedRepository' ' umask 077 && git config core.sharedRepository group && git reflog expire --all &&
@@ -201,7 +203,7 @@ test_expect_success POSIXPERM 're-init respects core.sharedrepository (remote)' test_cmp expect actual ' -test_expect_success POSIXPERM 'template can set core.sharedrepository' ' +test_expect_success REFFILES,POSIXPERM 'template can set core.sharedrepository' ' rm -rf child.git && umask 0022 && git config core.sharedrepository 0666 &&
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 24/28 t1401,t2011: parameterize HEAD.lock for REFFILES
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
--- t/t1401-symbolic-ref.sh | 11 +++++++++-- t/t2011-checkout-invalid-head.sh | 11 +++++++++-- 2 files changed, 18 insertions(+), 4 deletions(-)
t/t1401-symbolic-ref.sh +9 -2
@@ -102,9 +102,16 @@ test_expect_success LONG_REF 'we can parse long symbolic ref' ' test_cmp expect actual ' +if test_have_prereq REFFILES +then + HEAD_LOCK=HEAD.lock +else + HEAD_LOCK=reftable/tables.list.lock +fi + test_expect_success 'symbolic-ref reports failure in exit code' ' - test_when_finished "rm -f .git/HEAD.lock" && - >.git/HEAD.lock && + test_when_finished "rm -f .git/$HEAD_LOCK" && + >.git/$HEAD_LOCK && test_must_fail git symbolic-ref HEAD refs/heads/whatever '
t/t2011-checkout-invalid-head.sh +9 -2
@@ -22,9 +22,16 @@ test_expect_success 'checkout main from invalid HEAD' ' git checkout main -- ' +if test_have_prereq REFFILES +then + HEAD_LOCK=HEAD.lock +else + HEAD_LOCK=reftable/tables.list.lock +fi + test_expect_success 'checkout notices failure to lock HEAD' ' - test_when_finished "rm -f .git/HEAD.lock" && - >.git/HEAD.lock && + test_when_finished "rm -f .git/$HEAD_LOCK" && + >.git/$HEAD_LOCK && test_must_fail git checkout -b other '
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 25/28 t1404: annotate test cases with REFFILES
In reply to Ævar Arnfjörð BjarmasonFrom: Han-Wen Nienhuys <[email protected]>
* Reftable for now lacks detailed error messages for directory/file conflicts. Skip message comparisons.
* Mark tests that muck with .git directly as REFFILES.
--- t/t1404-update-ref-errors.sh | 56 +++++++++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 14 deletions(-)
t/t1404-update-ref-errors.sh +42 -14
@@ -27,7 +27,9 @@ test_update_rejected () { fi && printf "create $prefix/%s $C\n" $create >input && test_must_fail git update-ref --stdin <input 2>output.err && - test_i18ngrep -F "$error" output.err && + if test_have_prereq REFFILES ; then + test_i18ngrep -F "$error" output.err + fi && git for-each-ref $prefix >actual && test_cmp unchanged actual }
@@ -101,7 +103,9 @@ df_test() { printf "%s\n" "delete $delname" "create $addname $D" fi >commands && test_must_fail git update-ref --stdin <commands 2>output.err && - test_cmp expected-err output.err && + if test_have_prereq REFFILES ; then + test_cmp expected-err output.err + fi && printf "%s\n" "$C $delref" >expected-refs && git for-each-ref --format="%(objectname) %(refname)" $prefix/r >actual-refs && test_cmp expected-refs actual-refs
@@ -336,7 +340,9 @@ test_expect_success 'missing old value blocks update' ' EOF printf "%s\n" "update $prefix/foo $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks update' '
@@ -347,7 +353,9 @@ test_expect_success 'incorrect old value blocks update' ' EOF printf "%s\n" "update $prefix/foo $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'existing old value blocks create' '
@@ -358,7 +366,9 @@ test_expect_success 'existing old value blocks create' ' EOF printf "%s\n" "create $prefix/foo $E" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks delete' '
@@ -369,7 +379,9 @@ test_expect_success 'incorrect old value blocks delete' ' EOF printf "%s\n" "delete $prefix/foo $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'missing old value blocks indirect update' '
@@ -380,7 +392,9 @@ test_expect_success 'missing old value blocks indirect update' ' EOF printf "%s\n" "update $prefix/symref $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks indirect update' '
@@ -392,7 +406,9 @@ test_expect_success 'incorrect old value blocks indirect update' ' EOF printf "%s\n" "update $prefix/symref $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'existing old value blocks indirect create' '
@@ -404,7 +420,9 @@ test_expect_success 'existing old value blocks indirect create' ' EOF printf "%s\n" "create $prefix/symref $E" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks indirect delete' '
@@ -416,7 +434,9 @@ test_expect_success 'incorrect old value blocks indirect delete' ' EOF printf "%s\n" "delete $prefix/symref $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'missing old value blocks indirect no-deref update' '
@@ -427,7 +447,9 @@ test_expect_success 'missing old value blocks indirect no-deref update' ' EOF printf "%s\n" "option no-deref" "update $prefix/symref $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks indirect no-deref update' '
@@ -439,7 +461,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref update' ' EOF printf "%s\n" "option no-deref" "update $prefix/symref $E $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'existing old value blocks indirect no-deref create' '
@@ -451,7 +475,9 @@ test_expect_success 'existing old value blocks indirect no-deref create' ' EOF printf "%s\n" "option no-deref" "create $prefix/symref $E" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success 'incorrect old value blocks indirect no-deref delete' '
@@ -463,7 +489,9 @@ test_expect_success 'incorrect old value blocks indirect no-deref delete' ' EOF printf "%s\n" "option no-deref" "delete $prefix/symref $D" | test_must_fail git update-ref --stdin 2>output.err && - test_cmp expected output.err + if test_have_prereq REFFILES ; then + test_cmp expected output.err + fi ' test_expect_success REFFILES 'non-empty directory blocks create' '
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 26/28 reftable: fixup for new base topic 1/3
In reply to Ævar Arnfjörð BjarmasonLifted from Han-Wen's [1], with the addition of the removal of the reftable_error_to_errno() which is now unused. I also changed the:
if err < 0
if err > 1
to just:
if err < 0
else if err
Which is clearer in this context, i.e. the reader doesn't need to squint to see that the two if's are related, but different only in "<" v.s. ">".
1. https://lore.kernel.org/git/[email protected]/
--- refs/reftable-backend.c | 38 ++++++++------------------------------ 1 file changed, 8 insertions(+), 30 deletions(-)
refs/reftable-backend.c +8 -30
@@ -108,7 +108,7 @@ static const char *bare_ref_name(const char *ref)static int git_reftable_read_raw_ref (struct ref_store *ref_store,const char *refname,struct object_id *oid,struct strbuf *referent, -unsigned int *type); +unsigned int *type,int *failure_errno);static void clear_reftable_log_record (struct reftable_log_record *log) {
@@ -424,6 +424,7 @@ static int fixup_symrefs(struct ref_store *ref_store,struct strbuf referent =STRBUF_INIT ;int i =0 ;int err =0 ; +int failure_errno;for (i =0 ; i < transaction->nr; i++) {struct ref_update *update = transaction->updates[i];
@@ -433,8 +434,8 @@ static int fixup_symrefs(struct ref_store *ref_store, &old_oid, &referent,/* mutate input, like files-backend.c */ - &update->type); -if (err <0 && errno ==ENOENT && + &update->type, &failure_errno); +if (err <0 && failure_errno ==ENOENT &&is_null_oid (&update->old_oid)) { err =0 ; }
@@ -1587,30 +1588,10 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,return err; } -static int reftable_error_to_errno (int err) -{ -switch (err) { -case REFTABLE_IO_ERROR : -return EIO ; -case REFTABLE_FORMAT_ERROR : -return EFAULT ; -case REFTABLE_NOT_EXIST_ERROR : -return ENOENT ; -case REFTABLE_LOCK_ERROR : -return EBUSY ; -case REFTABLE_API_ERROR : -return EINVAL ; -case REFTABLE_ZLIB_ERROR : -return EDOM ; -default : -return ERANGE ; - } -} -static int git_reftable_read_raw_ref (struct ref_store *ref_store,const char *refname,struct object_id *oid,struct strbuf *referent, -unsigned int *type) +unsigned int *type,int *failure_errno) {struct git_reftable_ref_store *refs = (struct git_reftable_ref_store *)ref_store;
@@ -1633,13 +1614,10 @@ static int git_reftable_read_raw_ref(struct ref_store *ref_store, } err =reftable_stack_read_ref (stack, refname, &ref); -if (err >0 ) { - errno =ENOENT ; - err =-1 ; - goto done; - }if (err <0 ) { - errno =reftable_error_to_errno (err); + goto done; + }else if (err) { + *failure_errno =ENOENT ; err =-1 ; goto done; }
Signature · 1 line
2.33.0.662.gbaddc25a55e
PATCH v4 27/28 reftable: fixup for new base topic 2/3
In reply to Ævar Arnfjörð BjarmasonSince my "refs API: remove OID argument to reflog_expire()" we don't have the "oid" as part of the reflog_expire() signature. Instead the reflog_expire() should pass the OID of the tip of the "locked" ref to the prepare_fn().
In files_reflog_expire() we do that by getting the OID from lock_ref_oid_basic(). I'm assuming (but am not familiar enough with reftable...) that by the time we get here we've got a locked ref already in some way, so let's just use refs_resolve_ref_unsafe_with_errno() to lookup the current OID of that presumably-locked ref.
--- refs/reftable-backend.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
refs/reftable-backend.c +12 -2
@@ -1463,7 +1463,7 @@ static int write_reflog_expiry_table(struct reftable_writer *writer, void *argv) static int git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname, - const struct object_id *oid, unsigned int flags, + unsigned int flags, reflog_expiry_prepare_fn prepare_fn, reflog_expiry_should_prune_fn should_prune_fn, reflog_expiry_cleanup_fn cleanup_fn,
@@ -1497,6 +1497,9 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,struct reftable_iterator it = {NULL };struct reftable_addition *add =NULL ;int err =0 ; +int ignore_errno; +struct object_id oid; +if (refs->err <0 ) {return refs->err; }
@@ -1515,7 +1518,14 @@ git_reftable_reflog_expire(struct ref_store *ref_store, const char *refname,if (err) { goto done; } -prepare_fn (refname, oid, policy_cb_data); +if (!refs_resolve_ref_unsafe_with_errno (ref_store, refname, +RESOLVE_REF_READING , &oid, +NULL , &ignore_errno)) { + err =-1 ; + goto done; + } +prepare_fn (refname, &oid, policy_cb_data); +while (1 ) {struct reftable_log_record log = {NULL };int err =reftable_iterator_next_log (&it, &log);
Signature · 1 line
2.33.0.662.gbaddc25a55e
Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Since my "refs API: remove OID argument to
reflog_expire()" we don't have the "oid" as part of thereflog_expire()signature. Instead thereflog_expire()should pass the OID of the tip of the "locked" ref to theprepare_fn().
Quoted 6 lines
In
files_reflog_expire()we do that by getting the OID fromlock_ref_oid_basic(). I'm assuming (but am not familiar enough with reftable...) that by the time we get here we've got a locked ref already in some way, so let's just userefs_resolve_ref_unsafe_with_errno()to lookup the current OID of that presumably-locked ref.
I quickly looked at the files code, but I don't understand why the OID needs to be passed-in (before your refactoring): in builtin/reflog.c (before), the current OID is read, with any protection. This means that its value can't be trusted.
After your refactoring, you lock the ref. I guess in the files backend this protects against non-atomic update of (ref, reflog) racing with a concurrent reflog expiry? In reftable, the (ref,reflog) update is atomic, so there is no need for locking to properly sequence operations.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3
Quoted 20 lines · Han-Wen Nienhuys
Quoted 6 lines · Ævar Arnfjörð Bjarmason
Since my "refs API: remove OID argument to
reflog_expire()" we don't have the "oid" as part of thereflog_expire()signature. Instead thereflog_expire()should pass the OID of the tip of the "locked" ref to theprepare_fn().Quoted 6 lines
In
files_reflog_expire()we do that by getting the OID fromlock_ref_oid_basic(). I'm assuming (but am not familiar enough with reftable...) that by the time we get here we've got a locked ref already in some way, so let's just userefs_resolve_ref_unsafe_with_errno()to lookup the current OID of that presumably-locked ref.I quickly looked at the files code, but I don't understand why the OID needs to be passed-in (before your refactoring): in
builtin/reflog.c(before), the current OID is read, with any protection. This means that its value can't be trusted.
Quoted 5 lines
After your refactoring, you lock the ref. I guess in the files backend this protects against non-atomic update of (ref, reflog) racing with a concurrent reflog expiry? In reftable, the (ref,reflog) update is atomic, so there is no need for locking to properly sequence operations.
Before my [1] we'd do:
1. Read the OID for the branch in builtin/reflog.c
2. Pass it to refs/files-backend.c, it would lock at that OID (or fail if it changed)
3. Pass the OID with the now-locked OID to the builtin/reflog.c code
After that [1] we do:
1. Lock the branch in refs/files-backend.c
2. Pass the OID with the now-locked OID to the builtin/reflog.c code (ditto #3 above)
Whatever reftable itself does with updates doesn't change that we need to do that 2nd step of passing the OID to builtin/reflog.c, as it makes use of it.
That code is a bit confusing, if you want to understand it I recommend reading it at the tip of my yet-unsubmitted avar/cleanup-refs-api-and-reflog-expire-post-no-eisdir, it makes the control flow a lot cleaner.
So as far as what we do here is concerned, we're stuck with the refs files backend inherently wanting to pass "I locked this for you, here's the OID".
I guess it could also pass "OK, now go ahead and expire" and pass no OID. We'd then in builtin/reflog.c lookup the current OID for the logic there, but just having the reftable backend appease the common API by looking up the OID and passing it seemed like the most straightforward thing to do.
I haven't tested this or thought it through, but I don't understand how reftable isn't going to race in reflog expiry then. Sure, the ref/reflog update itself is atomic, so it won't suffer from the needing-a-lock problem of two concurrent file backend writers doing the equivalent of:
echo $NEW_SHA1 >.git/refs/heads/some-branch
But we will need at least the optimistic locking of code like builtin/reflog.c wanting to do an expiry, and deciding whether to do that expiry based on a given state of the ref/reflog. I.e. we don't want:
1. Start reflog expiry
2. Code in builtin/reflog.c looks up the OID
3. Code in builtin/reflog.c decides whether expire the reflog
4. Concurrent with #4, another writer updates the ref/reflog pair
5. Code in builtin/reflog.c says "OK, expire it!"
6. Reftable queues a delete/prune of the reflog per #5.
This would be a sequente of updates to the ref/reflog, none of whom were racy as far as the reftable semantics itself are concerneb, but where we'd do the wrong thing because the writer thought we had A when we really had B. So we need the equivalent of an "git update-ref" with the "<oldvalue>".
Is there a better way to do that in this case that I'm missing?
1. https://lore.kernel.org/git/[email protected]/
2. https://github.com/avar/git/compare/avar/files-backend-remove-dead-errno-eisdir-6...avar:avar/cleanup-refs-api-and-reflog-expire-post-no-eisdir
Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3
Quoted 12 lines · Ævar Arnfjörð Bjarmason
But we will need at least the optimistic locking of code like
builtin/reflog.cwanting to do an expiry, and deciding whether to do that expiry based on a given state of the ref/reflog. I.e. we don't want:1. Start reflog expiry
2. Code inbuiltin/reflog.clooks up the OID
3. Code inbuiltin/reflog.cdecides whether expire the reflog
4. Concurrent with #4, another writer updates the ref/reflog pair
5. Code inbuiltin/reflog.csays "OK, expire it!"
6. Reftable queues a delete/prune of the reflog per #5.
Quoted 7 lines
This would be a sequente of updates to the ref/reflog, none of whom were racy as far as the reftable semantics itself are concerneb, but where we'd do the wrong thing because the writer thought we had A when we really had B. So we need the equivalent of an "git update-ref" with the "<oldvalue>".
Is there a better way to do that in this case that I'm missing?
I spent some more time looking at builtin/reflog.c, but I am still not 100% sure what the locking is used for.
From a quick glance, the OID goes into tip_commit, and tip_commit goes into a reachable list (?). The reachable list is then for something, but I can't really tell what.
In your example with 1.-6., it's still not clear to me what the undesired behavior is precisely. If the reflog is pruned in #6, is the worry that the update in #4 is pruned immediately after being effected?
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Re: [PATCH v4 27/28] reftable: fixup for new base topic 2/3
Quoted 25 lines · Han-Wen Nienhuys
Quoted 12 lines · Ævar Arnfjörð Bjarmason
But we will need at least the optimistic locking of code like
builtin/reflog.cwanting to do an expiry, and deciding whether to do that expiry based on a given state of the ref/reflog. I.e. we don't want:1. Start reflog expiry
2. Code inbuiltin/reflog.clooks up the OID
3. Code inbuiltin/reflog.cdecides whether expire the reflog
4. Concurrent with #4, another writer updates the ref/reflog pair
5. Code inbuiltin/reflog.csays "OK, expire it!"
6. Reftable queues a delete/prune of the reflog per #5.Quoted 7 lines
This would be a sequente of updates to the ref/reflog, none of whom were racy as far as the reftable semantics itself are concerneb, but where we'd do the wrong thing because the writer thought we had A when we really had B. So we need the equivalent of an "git update-ref" with the "<oldvalue>".
Is there a better way to do that in this case that I'm missing?
I spent some more time looking at
builtin/reflog.c, but I am still not 100% sure what the locking is used for.
Quoted 8 lines
From a quick glance, the OID goes into
tip_commit, andtip_commitgoes into a reachable list (?). The reachable list is then for something, but I can't really tell what.In your example with 1.-6., it's still not clear to me what the undesired behavior is precisely. If the reflog is pruned in #6, is the worry that the update in #4 is pruned immediately after being effected?
Yes, I think so. But I'm not sure. I skimmed the code quickly today, and when I wrote the referenced series didn't focus much on the nitty-gritty of the builtin/reflog.c behavior other than assuring myself that I was doing the exact same thing as before as far as its logic was concerned.
I.e. it always locked at a given OID. Before my in-flight "reflog
expire: don't lock reflogs using previously seen OI" it might not lock but get this error:
error: cannot lock ref '<refname>': ref '<refname>' is at <OID-A> but expected <OID-B>
But at least it wouldn't do anything, but the current code does require the passed-in OID. See the code that needs "unreachable_expire_kind" and "tip_commit".
Perhaps that whole thing can also be refactored somehow. If I change the "commit = lookup_commit(the_repository, oid);" in "reflog_expiry_prepare()" to just "commit = NULL" all tests pass, but that might just be missing test coverage in the face of same race...
PATCH v4 28/28 reftable: fixup for new base topic 3/3
In reply to Ævar Arnfjörð BjarmasonSince the "refs: add failure_errno to refs_read_raw_ref() signature" in the base topic we've preferred to use refs_resolve_ref_unsafe_with_errno() over the refs_resolve_ref_unsafe().
Let's make it explicitly clear that we're ignoring "errno" in these cases, and while we're at it change "if (x != NULL)" to just "if (x)" as refs_resolve_ref_unsafe*() returns a const char *, and explicitly checking 0 or NULL in if's is not in line with our coding guidelines.
--- refs/reftable-backend.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-)
refs/reftable-backend.c +17 -9
@@ -315,9 +315,11 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)break ;case REFTABLE_REF_SYMREF : {int out_flags =0 ; -const char *resolved =refs_resolve_ref_unsafe ( +int ignore_errno; +const char *resolved =refs_resolve_ref_unsafe_with_errno ( ri->ref_store, ri->ref.refname, -RESOLVE_REF_READING , &ri->oid, &out_flags); +RESOLVE_REF_READING , &ri->oid, &out_flags, + &ignore_errno); ri->base.flags = out_flags; if (resolved == NULL && !(ri->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&
@@ -543,8 +545,10 @@ static int reftable_check_old_oid(struct ref_store *refs, const char *refname, {struct object_id out_oid;int out_flags =0 ; -const char *resolved =refs_resolve_ref_unsafe ( - refs, refname,RESOLVE_REF_READING , &out_oid, &out_flags); +int ignore_errno; +const char *resolved =refs_resolve_ref_unsafe_with_errno ( + refs, refname,RESOLVE_REF_READING , &out_oid, &out_flags, + &ignore_errno);if (is_null_oid (want_oid) != (resolved ==NULL )) {return REFTABLE_LOCK_ERROR ; }
@@ -845,20 +849,24 @@ static int write_create_symref_table(struct reftable_writer *writer, void *arg)struct reftable_log_record log = {NULL };struct object_id new_oid;struct object_id old_oid; +int ignore_errno;fill_reftable_log_record (&log); log.refname = (char *)create->refname; log.update_index = ts; log.value.update.message = (char *)create->logmsg; -if (refs_resolve_ref_unsafe ( +if (refs_resolve_ref_unsafe_with_errno ( (struct ref_store *)create->refs, create->refname, -RESOLVE_REF_READING , &old_oid,NULL ) !=NULL ) { +RESOLVE_REF_READING , &old_oid,NULL , + &ignore_errno)) { log.value.update.old_hash = old_oid.hash; } -if (refs_resolve_ref_unsafe ((struct ref_store *)create->refs, - create->target,RESOLVE_REF_READING , - &new_oid,NULL ) !=NULL ) { +if (refs_resolve_ref_unsafe_with_errno ((struct ref_store *)create->refs, + create->target, +RESOLVE_REF_READING , + &new_oid,NULL , + &ignore_errno)) { log.value.update.new_hash = new_oid.hash; }
Signature · 1 line
2.33.0.662.gbaddc25a55e
Quoted 10 lines · Ævar Arnfjörð Bjarmason
This is a version of the reftable series queued on top of my just-re-rolled fixes to the refs APIs, which it can make use of. For the base topics see:
https://lore.kernel.org/git/[email protected]/ https://lore.kernel.org/git/[email protected]/For Han-Wen's v3 of this see:
https://lore.kernel.org/git/[email protected]/
Quoted 7 lines
I've got no desire to take over the reftable topic in its entirety, but think given the rationale in
https://lore.kernel.org/git/[email protected]/
(summarized in
https://lore.kernel.org/git/[email protected]/) that having the refs API fixes I noted above wait on the still-unstable reftable doesn't make sense.
Of course, you and Han-Wen are in much better position to judge the relative merit to decide which one should go first than I am, but I had an impression that the errno thing was even less stable, with API churn that deliberately broke the other topic in flight, which appeared to be just irresponsible.
Quoted 9 lines
I'll let Han-Wen deal with that squashing in a presumed future v5 of this, assuming of course that Junio's happy with the plan of basing hn/reftable on the refs API fixes above.
I'm not sure that the fix I have in 27/28 is the right one, perhaps we've already got information about what the tip OID of the refname is at that point in
git_reftable_reflog_expire()via some API I missed, but that fix works, and is clearly more correct than the outstanding segfault.
Quoted 11 lines · Junio C Hamano
Quoted 10 lines
For Han-Wen's v3 of this see:
https://lore.kernel.org/git/[email protected]/I've got no desire to take over the reftable topic in its entirety, but think given the rationale in
https://lore.kernel.org/git/[email protected]/
(summarized in
https://lore.kernel.org/git/[email protected]/) that having the refs API fixes I noted above wait on the still-unstable reftable doesn't make sense.
Quoted 5 lines
Of course, you and Han-Wen are in much better position to judge the relative merit to decide which one should go first than I am, but I had an impression that the errno thing was even less stable, with API churn that deliberately broke the other topic in flight, which appeared to be just irresponsible.
The bottom part of the errno series that I contributed has had ample scrutiny. It's a cleanup, and all-in-all much less experimental than the reftable work. However, because it changes a calling convention in the ref backend API, it causes difficulty with other topics (notably: reftable). I would be in favor of graduating the series upto "refs: make errno output explicit for read_raw_ref_fn" early to provide a stable basis for other patches.
Signature · 11 lines
Han-Wen Nienhuys - Google Munich I work 80%. Don't expect answers from me on Fridays. -- Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Quoted 7 lines · Han-Wen Nienhuys
The bottom part of the errno series that I contributed has had ample scrutiny. It's a cleanup, and all-in-all much less experimental than the reftable work. However, because it changes a calling convention in the ref backend API, it causes difficulty with other topics (notably: reftable). I would be in favor of graduating the series upto "refs: make errno output explicit for
read_raw_ref_fn" early to provide a stable basis for other patches.
Very glad to see that the two of you are in agreement of the order and the approach. Let me replace the topics that have been queued on 'seen' with the latest ones from Ævar, and we can go from there.
Thanks for a quick response.