reftable: fix reload with active iterators
cover reftable: fix reload with active iterators
Hi,
as reported by Peff in [1], the reftable backend fails in t5616 in a racy manner. The cause of this is that git-maintenance(1) processes leak into subsequent tests and perform concurrent compaction of the stack. The reftable backend is expected to handle this gracefully, but doesn't.
The issue can surface whenever reloading the reftable stack while an iterator is active. In case some of the tables got compacted, we will end up closing them and thus the iterator now tries to use those closed tables.
This patch series fixes that issue by starting to refcount the readers. Each iterator will bump the refcount, thus avoiding the problem. While at it, I also found a second issue where we segfault when reloading a table fails while reusing one of the table readers. In this scenario, we would end up releasing the reader of the stack itself, even though it would still be used by it.
This patch series addresses those issues by making the reftable reader refcounted. The first 6 patches do some simplifications of code which is close. The remaining 4 ones introduce refcounting and wire it up as required.
With this, the following command now passes:
make -C .. -j20 SANITIZE=address && GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress
The patch series builds on top of Junio's ps/reftable-drop-generic at 1a3c3870ee (reftable/generic: drop interface, 2024-08-14). It's only on seen right now, but would otherise cause a bunch of conflicts.
Thanks!
[1]: <[email protected]>
Patrick Steinhardt (10):
reftable/blocksource: drop malloc block source reftable/stack: inline stack_compact_range_stats() reftable/reader: rename reftable_new_reader() reftable/reader: inline init_reader() reftable/reader: inline reader_close() reftable/stack: fix broken refnames in write_n_ref_tables() reftable/reader: introduce refcounting
reftable/reader: keep readers alive during iteration reftable/stack: reorder swapping in the reloaded stack contents reftable/stack: fix segfault when reload with reused readers fails
reftable/block_test.c | 3 +- reftable/blocksource.c | 20 ----- reftable/blocksource.h | 2 - reftable/reader.c | 149 ++++++++++++++++--------------- reftable/reader.h | 5 +- reftable/readwrite_test.c | 85 +++++++++--------- reftable/reftable-reader.h | 19 ++-- reftable/stack.c | 90 +++++++++++-------- reftable/stack_test.c | 115 +++++++++++++++++++++++- t/helper/test-reftable.c | 4 +- t/unit-tests/t-reftable-merged.c | 10 +-- 11 files changed, 311 insertions(+), 191 deletions(-)
base-commit: 1a3c3870ee1a65b0579ccbac6b18e22b8c44c5b4
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 1/10 reftable/blocksource: drop malloc block source
The reftable blocksource provides a generic interface to read blocks via different sources, e.g. from disk or from memory. One of the block sources is the malloc block source, which can in theory read data from memory. We nowadays also have a strbuf block source though, which provides essentially the same functionality with better ergonomics.
Adapt the only remaining user of the malloc block source in our tests to use the strbuf block source, instead, and remove the now-unused malloc block source.
--- reftable/block_test.c | 3 ++- reftable/blocksource.c | 20 -------------------- reftable/blocksource.h | 2 -- 3 files changed, 2 insertions(+), 23 deletions(-)
reftable/block_test.c +2 -1
@@ -34,11 +34,12 @@ static void test_block_read_write(void)struct block_reader br = {0 };struct block_iter it =BLOCK_ITER_INIT ;int j =0 ; +struct strbuf block_data =STRBUF_INIT ;struct strbuf want =STRBUF_INIT ;REFTABLE_CALLOC_ARRAY (block.data, block_size); block.len = block_size; - block.source =malloc_block_source (); +block_source_from_strbuf (&block.source, &block_data);block_writer_init (&bw,BLOCK_TYPE_REF , block.data, block_size, header_off,hash_size (GIT_SHA1_FORMAT_ID ));
reftable/blocksource.c +0 -20
@@ -55,26 +55,6 @@ void block_source_from_strbuf(struct reftable_block_source *bs, bs->arg = buf; } -static void malloc_return_block (void *b,struct reftable_block *dest) -{ -if (dest->len) -memset (dest->data,0xff , dest->len); -reftable_free (dest->data); -} - -static struct reftable_block_source_vtable malloc_vtable = { - .return_block = &malloc_return_block, -}; - -static struct reftable_block_source malloc_block_source_instance = { - .ops = &malloc_vtable, -}; - -struct reftable_block_source malloc_block_source (void ) -{ -return malloc_block_source_instance; -} -struct file_block_source {uint64_t size;unsigned char *data;
reftable/blocksource.h +0 -2
@@ -17,6 +17,4 @@ struct reftable_block_source;void block_source_from_strbuf (struct reftable_block_source *bs,struct strbuf *buf); -struct reftable_block_source malloc_block_source (void ); -#endif
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 2/10 reftable/stack: inline `stack_compact_range_stats()`
In reply to Patrick SteinhardtThe only difference between stack_compact_range_stats() and stack_compact_range() is that the former updates stats on failure, whereas the latter doesn't. There are no callers anymore that do not want their stats updated though, making the indirection unnecessary.
Inline the stat updates into stack_compact_range().
--- reftable/stack.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)
reftable/stack.c +4 -12
@@ -1328,17 +1328,9 @@ static int stack_compact_range(struct reftable_stack *st,strbuf_release (&table_name);free_names (names); -return err; -} - -static int stack_compact_range_stats (struct reftable_stack *st, -size_t first,size_t last, -struct reftable_log_expiry_config *config, -unsigned int flags) -{ -int err =stack_compact_range (st, first, last, config, flags);if (err ==REFTABLE_LOCK_ERROR ) st->stats.failures++; +return err; }
@@ -1346,7 +1338,7 @@ int reftable_stack_compact_all(struct reftable_stack *st, struct reftable_log_expiry_config *config) {size_t last = st->merged->readers_len ? st->merged->readers_len -1 :0 ; -return stack_compact_range_stats (st,0 , last, config,0 ); +return stack_compact_range (st,0 , last, config,0 ); }static int segment_size (struct segment *s)
@@ -1452,8 +1444,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) st->opts.auto_compaction_factor);reftable_free (sizes);if (segment_size (&seg) >0 ) -return stack_compact_range_stats (st, seg.start, seg.end -1 , -NULL ,STACK_COMPACT_RANGE_BEST_EFFORT ); +return stack_compact_range (st, seg.start, seg.end -1 , +NULL ,STACK_COMPACT_RANGE_BEST_EFFORT );return 0 ; }
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Re: [PATCH 02/10] reftable/stack: inline `stack_compact_range_stats()`
Quoted 7 lines · Patrick Steinhardt
The only difference between
stack_compact_range_stats()andstack_compact_range()is that the former updates stats on failure, whereas the latter doesn't. There are no callers anymore that do not want their stats updated though, making the indirection unnecessary.Inline the stat updates into
stack_compact_range().
This also ensures that we don't have users who end up using stack_compact_range() because they didn't know about stack_compact_range_stats(). Which was the case before, I think, I remember one of the compaction function calling stack_compact_range() directly. I see it is no longer the case though. Nice.
[snip]
PATCH 3/10 reftable/reader: rename `reftable_new_reader()`
In reply to Patrick SteinhardtRename the reftable_new_reader() function to reftable_reader_new() to match our coding guidelines.
--- reftable/reader.c | 4 ++-- reftable/reader.h | 2 +- reftable/readwrite_test.c | 6 +++--- reftable/reftable-reader.h | 4 ++-- reftable/stack.c | 4 ++-- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 6 +++--- 7 files changed, 14 insertions(+), 14 deletions(-)
reftable/reader.c +2 -2
@@ -637,7 +637,7 @@ void reader_close(struct reftable_reader *r)FREE_AND_NULL (r->name); } -int reftable_new_reader (struct reftable_reader **p, +int reftable_reader_new (struct reftable_reader **p,struct reftable_block_source *src,char const *name) {struct reftable_reader *rd =reftable_calloc (1 ,sizeof (*rd));
@@ -786,7 +786,7 @@ int reftable_reader_print_blocks(const char *tablename)if (err <0 ) goto done; - err =reftable_new_reader (&r, &src, tablename); + err =reftable_reader_new (&r, &src, tablename);if (err <0 ) goto done;
reftable/reader.h +1 -1
@@ -52,7 +52,7 @@ struct reftable_reader {struct reftable_reader_offsets log_offsets; }; -int init_reader (struct reftable_reader *r,struct reftable_block_source *source, +int reader_init (struct reftable_reader *r,struct reftable_block_source *source,const char *name);void reader_close (struct reftable_reader *r);const char *reader_name (struct reftable_reader *r);
reftable/readwrite_test.c +3 -3
@@ -646,7 +646,7 @@ static void test_write_empty_table(void)block_source_from_strbuf (&source, &buf); - err =reftable_new_reader (&rd, &source,"filename" ); + err =reftable_reader_new (&rd, &source,"filename" );EXPECT_ERR (err);reftable_reader_init_ref_iterator (rd, &it);
@@ -850,7 +850,7 @@ static void test_write_multiple_indices(void)EXPECT (stats->log_stats.index_offset >0 );block_source_from_strbuf (&source, &writer_buf); - err =reftable_new_reader (&reader, &source,"filename" ); + err =reftable_reader_new (&reader, &source,"filename" );EXPECT_ERR (err);/*
@@ -907,7 +907,7 @@ static void test_write_multi_level_index(void)EXPECT (stats->ref_stats.max_index_level ==2 );block_source_from_strbuf (&source, &writer_buf); - err =reftable_new_reader (&reader, &source,"filename" ); + err =reftable_reader_new (&reader, &source,"filename" );EXPECT_ERR (err);/*
reftable/reftable-reader.h +2 -2
@@ -23,14 +23,14 @@/* The reader struct is a handle to an open reftable file. */ struct reftable_reader ; -/* reftable_new_reader opens a reftable for reading. If successful, +/* reftable_reader_new opens a reftable for reading. If successful, * returns 0 code and sets pp. The name is used for creating a * stack. Typically, it is the basename of the file. The block source * `src` is owned by the reader, and is closed on calling * reftable_reader_destroy(). On error, the block source `src` is * closed as well. */ -int reftable_new_reader (struct reftable_reader **pp, +int reftable_reader_new (struct reftable_reader **pp,struct reftable_block_source *src,const char *name);/* Initialize a reftable iterator for reading refs. */
reftable/stack.c +2 -2
@@ -258,7 +258,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (err <0 ) goto done; - err =reftable_new_reader (&rd, &src, name); + err =reftable_reader_new (&rd, &src, name);if (err <0 ) goto done; }
@@ -1532,7 +1532,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,if (err <0 ) goto done; - err =reftable_new_reader (&rd, &src, name); + err =reftable_reader_new (&rd, &src, name);if (err <0 ) goto done;
t/helper/test-reftable.c +1 -1
@@ -139,7 +139,7 @@ static int dump_reftable(const char *tablename)if (err <0 ) goto done; - err =reftable_new_reader (&r, &src, tablename); + err =reftable_reader_new (&r, &src, tablename);if (err <0 ) goto done;
t/unit-tests/t-reftable-merged.c +3 -3
@@ -102,7 +102,7 @@ merged_table_from_records(struct reftable_ref_record **refs,write_test_table (&buf[i], refs[i], sizes[i]);block_source_from_strbuf (&(*source)[i], &buf[i]); - err =reftable_new_reader (&(*readers)[i], &(*source)[i], + err =reftable_reader_new (&(*readers)[i], &(*source)[i],"name" );check (!err); }
@@ -277,7 +277,7 @@ merged_table_from_log_records(struct reftable_log_record **logs,write_test_log_table (&buf[i], logs[i], sizes[i], i +1 );block_source_from_strbuf (&(*source)[i], &buf[i]); - err =reftable_new_reader (&(*readers)[i], &(*source)[i], + err =reftable_reader_new (&(*readers)[i], &(*source)[i],"name" );check (!err); }
@@ -426,7 +426,7 @@ static void t_default_write_opts(void)block_source_from_strbuf (&source, &buf); - err =reftable_new_reader (&rd, &source,"filename" ); + err =reftable_reader_new (&rd, &source,"filename" );check (!err); hash_id =reftable_reader_hash_id (rd);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Re: [PATCH 03/10] reftable/reader: rename `reftable_new_reader()`
Quoted 5 lines · Patrick Steinhardt
Rename the
reftable_new_reader()function toreftable_reader_new()to match our coding guidelines.Signed-off-by Patrick Steinhardt <[email protected]>---
[snip]
reftable/reader.h struct reftable_reader { · line 56
Quoted 3 lines
reftable/reader.h +1
-1-int init_reader (struct reftable_reader *r,struct reftable_block_source *source, +int reader_init (struct reftable_reader *r,struct reftable_block_source *source,const char *name);
Here we also rename init_reader() to reader_init(), but do not update its call sites resulting in build errors. Since we remove this in the next patch anyway, let's drop this change.
-Justin
Re: [PATCH 03/10] reftable/reader: rename `reftable_new_reader()`
Quoted 6 lines · Justin Tobler
Quoted 5 lines · Patrick Steinhardt
Rename the
reftable_new_reader()function toreftable_reader_new()to match our coding guidelines.Signed-off-by Patrick Steinhardt <[email protected]>---
Quoted 8 lines
[snip]
reftable/reader.h struct reftable_reader { · line 56
Quoted 3 lines
reftable/reader.h +1
-1-int init_reader (struct reftable_reader *r,struct reftable_block_source *source, +int reader_init (struct reftable_reader *r,struct reftable_block_source *source,const char *name);Here we also rename
init_reader()toreader_init(), but do not update its call sites resulting in build errors. Since we remove this in the next patch anyway, let's drop this change.
True. The actual definition of the function is also left unmodified. Let me locally edit the hunk out. As you pointed out, the next step does mention the "new" name in order to remove it, so there needs a cascading adjustment there, but the fallout is fairly small (see the attached range-diff).
Thanks.
1: 348438a69e = 1: 348438a69e reftable/blocksource: drop malloc block source 2: ed8bb0e345 = 2: ed8bb0e345 reftable/stack: inline `stack_compact_range_stats()` 3: 80cf24d54f ! 3: 6b5466771c reftable/reader: rename `reftable_new_reader()`
@@ reftable/reader.c: int reftable_reader_print_blocks(const char *tablename)
goto done;
- ## reftable/reader.h ##
-@@ reftable/reader.h: struct reftable_reader {
- struct reftable_reader_offsets log_offsets;
- };
-
--int init_reader(struct reftable_reader *r, struct reftable_block_source *source, -+int reader_init(struct reftable_reader *r, struct reftable_block_source *source,
- const char *name);
- void reader_close(struct reftable_reader *r);
- const char *reader_name(struct reftable_reader *r);
-
## reftable/readwrite_test.c ##
@@ reftable/readwrite_test.c: static void test_write_empty_table(void)
4: ce93c128f5 ! 4: bc1eeb32ef reftable/reader: inline init_reader()
@@ reftable/reader.h: struct reftable_reader {
struct reftable_reader_offsets log_offsets;
};
--int reader_init(struct reftable_reader *r, struct reftable_block_source *source,
+-int init_reader(struct reftable_reader *r, struct reftable_block_source *source,
- const char *name);
void reader_close(struct reftable_reader *r);
const char *reader_name(struct reftable_reader *r);
5: 079a3d365e = 5: 3e244b036a reftable/reader: inline `reader_close()`
6: 7e951256c6 = 6: 41c537e44b reftable/stack: fix broken refnames in `write_n_ref_tables()`
7: 8e99330cff = 7: b7596c4c32 reftable/reader: introduce refcounting
8: 2383c196a6 = 8: 9f28628ecc reftable/reader: keep readers alive during iteration
9: 8af7c4485f = 9: 51df76889c reftable/stack: reorder swapping in the reloaded stack contents
10: 1ccdac05ab = 10: 32bbcc46b9 reftable/stack: fix segfault when reload with reused readers failsRe: [PATCH 03/10] reftable/reader: rename `reftable_new_reader()`
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Justin Tobler
Quoted 5 lines · Patrick Steinhardt
Rename the
reftable_new_reader()function toreftable_reader_new()to match our coding guidelines.Signed-off-by Patrick Steinhardt <[email protected]>---
Quoted 8 lines
Quoted 8 lines
[snip]
reftable/reader.h struct reftable_reader { · line 56
Quoted 3 lines
reftable/reader.h +1
-1-int init_reader (struct reftable_reader *r,struct reftable_block_source *source, +int reader_init (struct reftable_reader *r,struct reftable_block_source *source,const char *name);Here we also rename
init_reader()toreader_init(), but do not update its call sites resulting in build errors. Since we remove this in the next patch anyway, let's drop this change.
Well spotted! I originally wanted to fix that function's name, too. But then I realized it's not even needed in the first place.
Quoted 5 lines
True. The actual definition of the function is also left unmodified. Let me locally edit the hunk out. As you pointed out, the next step does mention the "new" name in order to remove it, so there needs a cascading adjustment there, but the fallout is fairly small (see the attached range-diff).
Thanks. The range diff looks as expected to me.
Patrick
PATCH 4/10 reftable/reader: inline `init_reader()`
In reply to Patrick SteinhardtMost users use an allocated version of the reftable_reader, except for some tests. We are about to convert the reader to become refcounted though, and providing the ability to keep a reader on the stack makes this conversion harder than necessary.
Update the tests to use reftable_reader_new() instead to prepare for this change.
--- reftable/reader.c | 122 +++++++++++++++++++------------------- reftable/reader.h | 2 - reftable/readwrite_test.c | 73 ++++++++++++----------- 3 files changed, 98 insertions(+), 99 deletions(-)
reftable/reader.c +61 -61
@@ -162,58 +162,6 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,return err; } -int init_reader (struct reftable_reader *r,struct reftable_block_source *source, -const char *name) -{ -struct reftable_block footer = {NULL }; -struct reftable_block header = {NULL }; -int err =0 ; -uint64_t file_size =block_source_size (source); - -/* Need +1 to read type of first block. */ -uint32_t read_size =header_size (2 ) +1 ;/* read v2 because it's larger. */ -memset (r,0 ,sizeof (struct reftable_reader )); - -if (read_size > file_size) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - - err =block_source_read_block (source, &header,0 , read_size); -if (err != read_size) { - err =REFTABLE_IO_ERROR ; - goto done; - } - -if (memcmp (header.data,"REFT" ,4 )) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - r->version = header.data[4 ]; -if (r->version !=1 && r->version !=2 ) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - - r->size = file_size -footer_size (r->version); - r->source = *source; - r->name =xstrdup (name); - r->hash_id =0 ; - - err =block_source_read_block (source, &footer, r->size, -footer_size (r->version)); -if (err !=footer_size (r->version)) { - err =REFTABLE_IO_ERROR ; - goto done; - } - - err =parse_footer (r, footer.data, header.data); -done: -reftable_block_done (&footer); -reftable_block_done (&header); -return err; -} -struct table_iter {struct reftable_reader *r;uint8_t typ;
@@ -637,16 +585,68 @@ void reader_close(struct reftable_reader *r)FREE_AND_NULL (r->name); } -int reftable_reader_new (struct reftable_reader **p, -struct reftable_block_source *src,char const *name) +int reftable_reader_new (struct reftable_reader **out, +struct reftable_block_source *source,char const *name) { -struct reftable_reader *rd =reftable_calloc (1 ,sizeof (*rd)); -int err =init_reader (rd, src, name); -if (err ==0 ) { - *p = rd; - }else { -block_source_close (src); -reftable_free (rd); +struct reftable_block footer = {0 }; +struct reftable_block header = {0 }; +struct reftable_reader *r; +uint64_t file_size =block_source_size (source); +uint32_t read_size; +int err; + +REFTABLE_CALLOC_ARRAY (r,1 ); + +/* +* We need one extra byte to read the type of first block. We also +* pretend to always be reading v2 of the format because it is larger. +*/ + read_size =header_size (2 ) +1 ; +if (read_size > file_size) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + + err =block_source_read_block (source, &header,0 , read_size); +if (err != read_size) { + err =REFTABLE_IO_ERROR ; + goto done; + } + +if (memcmp (header.data,"REFT" ,4 )) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + r->version = header.data[4 ]; +if (r->version !=1 && r->version !=2 ) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + + r->size = file_size -footer_size (r->version); + r->source = *source; + r->name =xstrdup (name); + r->hash_id =0 ; + + err =block_source_read_block (source, &footer, r->size, +footer_size (r->version)); +if (err !=footer_size (r->version)) { + err =REFTABLE_IO_ERROR ; + goto done; + } + + err =parse_footer (r, footer.data, header.data); +if (err) + goto done; + + *out = r; + +done: +reftable_block_done (&footer); +reftable_block_done (&header); +if (err) { +reftable_free (r); +block_source_close (source); }return err; }
reftable/reader.h +0 -2
@@ -52,8 +52,6 @@ struct reftable_reader {struct reftable_reader_offsets log_offsets; }; -int reader_init (struct reftable_reader *r,struct reftable_block_source *source, -const char *name);void reader_close (struct reftable_reader *r);const char *reader_name (struct reftable_reader *r);
reftable/readwrite_test.c +37 -36
240 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 5/10 reftable/reader: inline `reader_close()`
In reply to Patrick SteinhardtSame as with the preceding commit, we also provide a reader_close() function that allows the caller to close a reader without freeing it. This is unnecessary now that all users will have an allocated version of the reader.
Inline it into reftable_reader_free().
--- reftable/reader.c | 9 ++------- reftable/reader.h | 1 - reftable/stack.c | 5 +---- 3 files changed, 3 insertions(+), 12 deletions(-)
reftable/reader.c +2 -7
@@ -579,12 +579,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r,reader_init_iter (r, it,BLOCK_TYPE_LOG ); } -void reader_close (struct reftable_reader *r) -{ -block_source_close (&r->source); -FREE_AND_NULL (r->name); -} -int reftable_reader_new (struct reftable_reader **out,struct reftable_block_source *source,char const *name) {
@@ -655,7 +649,8 @@ void reftable_reader_free(struct reftable_reader *r) {if (!r)return ; -reader_close (r); +block_source_close (&r->source); +FREE_AND_NULL (r->name);reftable_free (r); }
reftable/reader.h +0 -1
@@ -52,7 +52,6 @@ struct reftable_reader {struct reftable_reader_offsets log_offsets; }; -void reader_close (struct reftable_reader *r);const char *reader_name (struct reftable_reader *r); void reader_init_iter(struct reftable_reader *r,
reftable/stack.c +1 -4
@@ -290,7 +290,6 @@ static int reftable_stack_reload_once(struct reftable_stack *st,const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reader_close (cur[i]);reftable_reader_free (cur[i]);/* On Windows, can only unlink after closing. */
@@ -299,10 +298,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, } done: -for (i =0 ; i < new_readers_len; i++) { -reader_close (new_readers[i]); +for (i =0 ; i < new_readers_len; i++)reftable_reader_free (new_readers[i]); - }reftable_free (new_readers);reftable_free (cur);strbuf_release (&table_path);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 6/10 reftable/stack: fix broken refnames in `write_n_ref_tables()`
In reply to Patrick SteinhardtThe write_n_ref_tables() helper function writes N references in separate tables. We never reset the computed name of those references though, leading us to end up with unexpected names.
Fix this by resetting the buffer.
--- reftable/stack_test.c | 1 + 1 file changed, 1 insertion(+)
reftable/stack_test.c +1 -0
@@ -125,6 +125,7 @@ static void write_n_ref_tables(struct reftable_stack *st, .value_type = REFTABLE_REF_VAL1, }; +strbuf_reset (&buf);strbuf_addf (&buf,"refs/heads/branch-%04u" , (unsigned ) i); ref.refname = buf.buf;set_test_hash (ref.value.val1, i);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 7/10 reftable/reader: introduce refcounting
In reply to Patrick SteinhardtIt was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads.
Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults.
One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one:
+ git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL
=================================================================
==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)
==657994==The signal is caused by a READ memory access.
#0 0x55f23e52ddf9 in get_var_int reftable/record.c:29
#1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170
#2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194
#3 0x55f23e54e72e in block_iter_next reftable/block.c:398
#4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240
#5 0x55f23e5573dc in table_iter_next reftable/reader.c:355
#6 0x55f23e5573dc in table_iter_next reftable/reader.c:339
#7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69
#8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123
#9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172
#10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175
#11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464
#12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13
#13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452
#14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623
#15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659
#16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133
#17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432
#18 0x55f23dba7764 in run_builtin git.c:484
#19 0x55f23dba7764 in handle_builtin git.c:741
#20 0x55f23dbab61e in run_argv git.c:805
#21 0x55f23dbab61e in cmd_main git.c:1000
#22 0x55f23dba4781 in main common-main.c:64
#23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360
#25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't.
The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling reftable_stack_reload(), which is wrong. We can only close them once we know that there are no iterators using them anymore.
Prepare for a fix by converting the reftable readers to be refcounted.
--- reftable/reader.c | 16 ++++++++++++++-- reftable/reader.h | 2 ++ reftable/readwrite_test.c | 18 +++++++++--------- reftable/reftable-reader.h | 15 ++++++++++++--- reftable/stack.c | 8 ++++---- reftable/stack_test.c | 6 ++---- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 4 ++-- 8 files changed, 46 insertions(+), 25 deletions(-)
reftable/reader.c +14 -2
@@ -621,6 +621,7 @@ int reftable_reader_new(struct reftable_reader **out, r->source = *source; r->name =xstrdup (name); r->hash_id =0 ; + r->refcount =1 ; err =block_source_read_block (source, &footer, r->size,footer_size (r->version));
@@ -645,10 +646,21 @@ int reftable_reader_new(struct reftable_reader **out,return err; } -void reftable_reader_free (struct reftable_reader *r) +void reftable_reader_incref (struct reftable_reader *r) +{ +if (!r->refcount) +BUG ("cannot increment ref counter of dead reader" ); + r->refcount++; +} + +void reftable_reader_decref (struct reftable_reader *r) {if (!r)return ; +if (!r->refcount) +BUG ("cannot decrement ref counter of dead reader" ); +if (--r->refcount) +return ;block_source_close (&r->source);FREE_AND_NULL (r->name);reftable_free (r);
@@ -812,7 +824,7 @@ int reftable_reader_print_blocks(const char *tablename) } done: -reftable_reader_free (r); +reftable_reader_decref (r);table_iter_close (&ti);return err; }
reftable/reader.h +2 -0
@@ -50,6 +50,8 @@ struct reftable_reader {struct reftable_reader_offsets ref_offsets;struct reftable_reader_offsets obj_offsets;struct reftable_reader_offsets log_offsets; + +uint64_t refcount; };const char *reader_name (struct reftable_reader *r);
reftable/readwrite_test.c +9 -9
@@ -279,7 +279,7 @@ static void test_log_write_read(void)/* cleanup. */ strbuf_release (&buf);free_names (names); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_log_zlib_corruption (void )
@@ -341,7 +341,7 @@ static void test_log_zlib_corruption(void)reftable_iterator_destroy (&it);/* cleanup. */ -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf); }
@@ -383,7 +383,7 @@ static void test_table_read_write_sequential(void)EXPECT (j ==N );reftable_iterator_destroy (&it); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf);free_names (names); }
@@ -431,7 +431,7 @@ static void test_table_read_api(void) }reftable_iterator_destroy (&it);reftable_free (names); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf); }
@@ -498,7 +498,7 @@ static void test_table_read_write_seek(int index, int hash_id)reftable_free (names[i]); }reftable_free (names); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_table_read_write_seek_linear (void )
@@ -611,7 +611,7 @@ static void test_table_refs_for(int indexed)strbuf_release (&buf);free_names (want_names);reftable_iterator_destroy (&it); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_table_refs_for_no_index (void )
@@ -657,7 +657,7 @@ static void test_write_empty_table(void)EXPECT (err >0 );reftable_iterator_destroy (&it); -reftable_reader_free (rd); +reftable_reader_decref (rd);strbuf_release (&buf); }
@@ -863,7 +863,7 @@ static void test_write_multiple_indices(void)reftable_iterator_destroy (&it);reftable_writer_free (writer); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&writer_buf);strbuf_release (&buf); }
@@ -919,7 +919,7 @@ static void test_write_multi_level_index(void)reftable_iterator_destroy (&it);reftable_writer_free (writer); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&writer_buf);strbuf_release (&buf); }
reftable/reftable-reader.h +12 -3
@@ -33,6 +33,18 @@ struct reftable_reader;int reftable_reader_new (struct reftable_reader **pp,struct reftable_block_source *src,const char *name); +/* +* Manage the reference count of the reftable reader. A newly initialized +* reader starts with a refcount of 1 and will be deleted once the refcount has +* reached 0. +* +* This is required because readers may have longer lifetimes than the stack +* they belong to. The stack may for example be reloaded while the old tables +* are still being accessed by an iterator. +*/ +void reftable_reader_incref (struct reftable_reader *reader); +void reftable_reader_decref (struct reftable_reader *reader); +/* Initialize a reftable iterator for reading refs. */ void reftable_reader_init_ref_iterator (struct reftable_reader *r,struct reftable_iterator *it);
@@ -44,9 +56,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r,/* returns the hash ID used in this table. */ uint32_t reftable_reader_hash_id (struct reftable_reader *r); -/* closes and deallocates a reader. */ -void reftable_reader_free (struct reftable_reader *); -/* return an iterator for the refs pointing to `oid`. */ int reftable_reader_refs_for (struct reftable_reader *r,struct reftable_iterator *it,uint8_t *oid);
reftable/stack.c +4 -4
@@ -186,7 +186,7 @@ void reftable_stack_destroy(struct reftable_stack *st)if (names && !has_name (names, name)) {stack_filename (&filename, st, name); } -reftable_reader_free (st->readers[i]); +reftable_reader_decref (st->readers[i]);if (filename.len) {/* On Windows, can only unlink after closing. */
@@ -290,7 +290,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st,const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reftable_reader_free (cur[i]); +reftable_reader_decref (cur[i]);/* On Windows, can only unlink after closing. */ unlink (table_path.buf);
@@ -299,7 +299,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, done:for (i =0 ; i < new_readers_len; i++) -reftable_reader_free (new_readers[i]); +reftable_reader_decref (new_readers[i]);reftable_free (new_readers);reftable_free (cur);strbuf_release (&table_path);
@@ -1534,7 +1534,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max, goto done; update_idx =reftable_reader_max_update_index (rd); -reftable_reader_free (rd); +reftable_reader_decref (rd);if (update_idx <= max) {unlink (table_path.buf);
reftable/stack_test.c +2 -4
@@ -1036,10 +1036,8 @@ static void test_reftable_stack_compaction_concurrent(void)static void unclean_stack_close (struct reftable_stack *st) {/* break abstraction boundary to simulate unclean shutdown. */ -int i =0 ; -for (; i < st->readers_len; i++) { -reftable_reader_free (st->readers[i]); - } +for (size_t i =0 ; i < st->readers_len; i++) +reftable_reader_decref (st->readers[i]); st->readers_len =0 ;FREE_AND_NULL (st->readers); }
t/helper/test-reftable.c +1 -1
@@ -152,7 +152,7 @@ static int dump_reftable(const char *tablename) done:reftable_merged_table_free (mt); -reftable_reader_free (r); +reftable_reader_decref (r);return err; }
t/unit-tests/t-reftable-merged.c +2 -2
@@ -115,7 +115,7 @@ merged_table_from_records(struct reftable_ref_record **refs,static void readers_destroy (struct reftable_reader **readers,const size_t n) {for (size_t i =0 ; i < n; i++) -reftable_reader_free (readers[i]); +reftable_reader_decref (readers[i]);reftable_free (readers); }
@@ -437,7 +437,7 @@ static void t_default_write_opts(void) err =reftable_merged_table_new (&merged, &rd,1 ,GIT_SHA1_FORMAT_ID );check (!err); -reftable_reader_free (rd); +reftable_reader_decref (rd);reftable_merged_table_free (merged);strbuf_release (&buf); }
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Re: [PATCH 07/10] reftable/reader: introduce refcounting
Quoted 39 lines · Patrick Steinhardt
It was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads.
Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults.
One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one:
+ git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL
=================================================================
==657994==ERROR: AddressSanitizer: SEGV on unknown address0x7fa0f0ec6089(pc0x55f23e52ddf9bp0x7ffe7bfa1700sp0x7ffe7bfa1700T0)
==657994==The signal is caused by a READ memory access.#0 0x55f23e52ddf9 in get_var_int reftable/record.c:29 #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170 #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194 #3 0x55f23e54e72e in block_iter_next reftable/block.c:398 #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240 #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355 #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339 #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69 #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123 #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172 #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175 #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464 #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13 #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452 #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623 #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659 #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133 #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432
Quoted 9 lines
#18 0x55f23dba7764 in run_builtin git.c:484
#19 0x55f23dba7764 in handle_builtin git.c:741
#20 0x55f23dbab61e in run_argv git.c:805
#21 0x55f23dbab61e in cmd_main git.c:1000
#22 0x55f23dba4781 in main common-main.c:64
#23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360
#25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)The stacktrace is for iterating over refs, what I don't understand is where in this flow do we actually reload the stack.
Quoted 4 lines
While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't.
Quoted 8 lines
The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling
reftable_stack_reload(), which is wrong. We can only close them once we know that there are no iterators using them anymore.Prepare for a fix by converting the reftable readers to be refcounted.
Okay so my understanding is that refcounted refers to a reference count which keeps tracks of the stacks which are referring to the reader. The name is also used in struct blame_origin in blame.{c,h}. Makes a lot more sense now :)
Quoted 18 lines
Reported-by Jeff King <[email protected]>Signed-off-by Patrick Steinhardt <[email protected]>--- reftable/reader.c | 16 ++++++++++++++-- reftable/reader.h | 2 ++ reftable/readwrite_test.c | 18 +++++++++--------- reftable/reftable-reader.h | 15 ++++++++++++--- reftable/stack.c | 8 ++++---- reftable/stack_test.c | 6 ++---- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 4 ++-- 8 files changed, 46 insertions(+), 25 deletions(-)reftable/reader.c +0
-0
reftable/reader.c int reftable_reader_new(struct reftable_reader **out, · line 625
Quoted 5 lines
reftable/reader.c +1
-0@@ -621,6 +621,7 @@ int reftable_reader_new(struct reftable_reader **out, r->source = *source; r->name =xstrdup (name); r->hash_id =0 ; + r->refcount =1 ;
So when the reader is initialized by someone, this sets the counter to one.
Quoted 27 lines
reftable/reader.c +12
-1err =block_source_read_block (source, &footer, r->size,footer_size (r->version));@@ -645,10 +646,21 @@ int reftable_reader_new(struct reftable_reader **out,return err; } -void reftable_reader_free (struct reftable_reader *r) +void reftable_reader_incref (struct reftable_reader *r) +{ +if (!r->refcount) +BUG ("cannot increment ref counter of dead reader" ); + r->refcount++; +} + +void reftable_reader_decref (struct reftable_reader *r) {if (!r)return ; +if (!r->refcount) +BUG ("cannot decrement ref counter of dead reader" ); +if (--r->refcount) +return ;block_source_close (&r->source);FREE_AND_NULL (r->name);reftable_free (r);
reftable/reader.c int reftable_reader_print_blocks(const char *tablename) · line 830
Quoted 8 lines
reftable/reader.c +1
-1@@ -812,7 +824,7 @@ int reftable_reader_print_blocks(const char *tablename) } done: -reftable_reader_free (r); +reftable_reader_decref (r);table_iter_close (&ti);return err; }
We remove the _free() function and instead introduce _incref() and _decref() to increase and decrease the counter. The latter takes over the functionality of _free() when counter hits 0.
reftable/reader.h struct reftable_reader { · line 54
Quoted 10 lines
reftable/reader.h +2
-0@@ -50,6 +50,8 @@ struct reftable_reader {struct reftable_reader_offsets ref_offsets;struct reftable_reader_offsets obj_offsets;struct reftable_reader_offsets log_offsets; + +uint64_t refcount;
Wonder if there is a chance that we decrement refcount from 0 and hence cause a wraparound.
[snip]
I have some questions regarding my own understanding, but the patch looks good to me.
Re: [PATCH 07/10] reftable/reader: introduce refcounting
Quoted 51 lines · karthik nayak
Quoted 39 lines · Patrick Steinhardt
It was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads.
Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults.
One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one:
+ git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL
=================================================================
==657994==ERROR: AddressSanitizer: SEGV on unknown address0x7fa0f0ec6089(pc0x55f23e52ddf9bp0x7ffe7bfa1700sp0x7ffe7bfa1700T0)
==657994==The signal is caused by a READ memory access.#0 0x55f23e52ddf9 in get_var_int reftable/record.c:29 #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170 #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194 #3 0x55f23e54e72e in block_iter_next reftable/block.c:398 #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240 #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355 #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339 #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69 #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123 #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172 #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175 #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464 #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13 #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452 #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623 #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659 #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133 #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432Quoted 9 lines
#18 0x55f23dba7764 in run_builtin git.c:484 #19 0x55f23dba7764 in handle_builtin git.c:741 #20 0x55f23dbab61e in run_argv git.c:805 #21 0x55f23dbab61e in cmd_main git.c:1000 #22 0x55f23dba4781 in main common-main.c:64 #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360 #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)
Quoted 2 lines
The stacktrace is for iterating over refs, what I don't understand is where in this flow do we actually reload the stack.
Basically, whenever you call into the reftable backend we check whether we need to reload the stack. So, when creating a reftable iterator, reading a single ref, writing refs and so on. So in the above code flow we had a ref iterator, but during iteration we ended up reading other refs, as well.
Quoted 10 lines
Quoted 10 lines
While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't.
The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling
reftable_stack_reload(), which is wrong. We can only close them once we know that there are no iterators using them anymore.
Quoted 7 lines
Quoted 2 lines
Prepare for a fix by converting the reftable readers to be refcounted.
Okay so my understanding is that
refcountedrefers to a reference count which keeps tracks of the stacks which are referring to the reader. The name is also used instruct blame_originin blame.{c,h}. Makes a lot more sense now :)
Yup.
Quoted 5 lines
Quoted 5 lines
reftable/reader.h +0
-0@@ -50,6 +50,8 @@ struct reftable_reader {
Quoted 8 lines
reftable/reader.h struct reftable_reader { · line 54
Quoted 5 lines
reftable/reader.h +2
-0struct reftable_reader_offsets ref_offsets;struct reftable_reader_offsets obj_offsets;struct reftable_reader_offsets log_offsets; + +uint64_t refcount;Wonder if there is a chance that we decrement refcount from 0 and hence cause a wraparound.
This should never happen in practice. And if it does, we would hit a BUG():
void reftable_reader_decref(struct reftable_reader *r)
{
if (!r)
return;
if (!r->refcount)
BUG("cannot decrement ref counter of dead reader");
if (--r->refcount)
return;
block_source_close(&r->source);
FREE_AND_NULL(r->name);
reftable_free(r);
}If the refcount is at zero already, we hit the bug.
Patrick
Re: [PATCH 07/10] reftable/reader: introduce refcounting
Quoted 52 lines · Patrick Steinhardt
Quoted 51 lines · karthik nayak
Quoted 48 lines · Patrick Steinhardt
It was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads.
Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults.
One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one:
+ git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL
=================================================================
==657994==ERROR: AddressSanitizer: SEGV on unknown address0x7fa0f0ec6089(pc0x55f23e52ddf9bp0x7ffe7bfa1700sp0x7ffe7bfa1700T0)
==657994==The signal is caused by a READ memory access.#0 0x55f23e52ddf9 in get_var_int reftable/record.c:29 #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170 #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194 #3 0x55f23e54e72e in block_iter_next reftable/block.c:398 #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240 #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355 #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339 #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69 #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123 #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172 #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175 #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464 #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13 #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452 #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623 #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659 #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133 #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432 #18 0x55f23dba7764 in run_builtin git.c:484 #19 0x55f23dba7764 in handle_builtin git.c:741 #20 0x55f23dbab61e in run_argv git.c:805 #21 0x55f23dbab61e in cmd_main git.c:1000 #22 0x55f23dba4781 in main common-main.c:64 #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360 #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)
Quoted 9 lines
Quoted 2 lines
The stacktrace is for iterating over refs, what I don't understand is where in this flow do we actually reload the stack.
Basically, whenever you call into the reftable backend we check whether we need to reload the stack. So, when creating a reftable iterator, reading a single ref, writing refs and so on. So in the above code flow we had a ref iterator, but during iteration we ended up reading other refs, as well.
Yes, makes sense. Thanks for clarifying.
Quoted 50 lines
Quoted 10 lines
Quoted 10 lines
While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't.
The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling
reftable_stack_reload(), which is wrong. We can only close them once we know that there are no iterators using them anymore.Quoted 7 lines
Quoted 2 lines
Prepare for a fix by converting the reftable readers to be refcounted.
Okay so my understanding is that
refcountedrefers to a reference count which keeps tracks of the stacks which are referring to the reader. The name is also used instruct blame_originin blame.{c,h}. Makes a lot more sense now :)Yup.
Quoted 5 lines
Quoted 5 lines
reftable/reader.h +0
-0@@ -50,6 +50,8 @@ struct reftable_reader {Quoted 8 lines
reftable/reader.h struct reftable_reader { · line 54
Quoted 5 lines
reftable/reader.h +2
-0struct reftable_reader_offsets ref_offsets;struct reftable_reader_offsets obj_offsets;struct reftable_reader_offsets log_offsets; + +uint64_t refcount;Wonder if there is a chance that we decrement refcount from 0 and hence cause a wraparound.
This should never happen in practice. And if it does, we would hit a
BUG():void reftable_reader_decref(struct reftable_reader *r) { if (!r) return; if (!r->refcount) BUG("cannot decrement ref counter of dead reader"); if (--r->refcount) return; block_source_close(&r->source); FREE_AND_NULL(r->name); reftable_free(r); }
Quoted 3 lines
If the refcount is at zero already, we hit the bug.
Patrick
Yup, this seems correct. Thanks.
PATCH 8/10 reftable/reader: keep readers alive during iteration
In reply to Patrick SteinhardtThe lifetime of a table iterator may surive the lifetime of a reader when the stack gets reloaded. Keep the reader from being released by increasing its refcount while the iterator is still being used.
--- reftable/reader.c | 2 ++ reftable/stack_test.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+)
reftable/reader.c +2 -0
@@ -175,6 +175,7 @@ static int table_iter_init(struct table_iter *ti, struct reftable_reader *r) {struct block_iter bi =BLOCK_ITER_INIT ;memset (ti,0 ,sizeof (*ti)); +reftable_reader_incref (r); ti->r = r; ti->bi = bi;return 0 ;
@@ -262,6 +263,7 @@ static void table_iter_close(struct table_iter *ti) {table_iter_block_done (ti);block_iter_close (&ti->bi); +reftable_reader_decref (ti->r); }static int table_iter_next_block (struct table_iter *ti)
reftable/stack_test.c +49 -0
61 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Re: [PATCH 08/10] reftable/reader: keep readers alive during iteration
Quoted 1 line · Patrick Steinhardt
The lifetime of a table iterator may surive the lifetime of a reader
s/surive/survive
Quoted 50 lines
when the stack gets reloaded. Keep the reader from being released by increasing its refcount while the iterator is still being used.
Signed-off-by Patrick Steinhardt <[email protected]>--- reftable/reader.c | 2 ++ reftable/stack_test.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+)reftable/reader.c +2
-0@@ -175,6 +175,7 @@ static int table_iter_init(struct table_iter *ti, struct reftable_reader *r) {struct block_iter bi =BLOCK_ITER_INIT ;memset (ti,0 ,sizeof (*ti)); +reftable_reader_incref (r); ti->r = r; ti->bi = bi;return 0 ;@@ -262,6 +263,7 @@ static void table_iter_close(struct table_iter *ti) {table_iter_block_done (ti);block_iter_close (&ti->bi); +reftable_reader_decref (ti->r); }static int table_iter_next_block (struct table_iter *ti)reftable/stack_test.c +13
-0@@ -1076,6 +1076,54 @@ static void test_reftable_stack_compaction_concurrent_clean(void)clear_dir (dir); } +static void test_reftable_stack_read_across_reload (void ) +{ +struct reftable_write_options opts = {0 }; +struct reftable_stack *st1 =NULL , *st2 =NULL ; +struct reftable_ref_record rec = {0 }; +struct reftable_iterator it = {0 }; +char *dir =get_tmp_dir (__LINE__); +int err; + +/* Create a first stack and set up an iterator for it. */ + err =reftable_new_stack (&st1, dir, &opts); +EXPECT_ERR (err); +write_n_ref_tables (st1,2 );
reftable/stack_test.c static void test_reftable_stack_compaction_concurrent_clean(void) · line 1101
Quoted 10 lines
reftable/stack_test.c +10
-0+EXPECT (st1->merged->readers_len ==2 ); +reftable_stack_init_ref_iterator (st1, &it); + err =reftable_iterator_seek_ref (&it,"" ); +EXPECT_ERR (err); + +/* Set up a second stack for the same directory and compact it. */ + err =reftable_new_stack (&st2, dir, &opts); +EXPECT_ERR (err); +EXPECT (st2->merged->readers_len ==2 ); + err =reftable_stack_compact_all (st2,NULL );
Shouldn't we verify that EXPECT(st2->merged->readers_len == 1); here?
reftable/stack_test.c static void test_reftable_stack_compaction_concurrent_clean(void) · line 1110
Quoted 9 lines
reftable/stack_test.c +9
-0+EXPECT_ERR (err); + +/* +* Verify that we can continue to use the old iterator even after we +* have reloaded its stack. +*/ + err =reftable_stack_reload (st1); +EXPECT_ERR (err); +EXPECT (st2->merged->readers_len ==1 );
Oh we do it here, I would've expected it above the st1 reload. And probably expected EXPECT(st1->merged->readers_len == 2); here to confirm that we still have the readers.
Quoted 29 lines
reftable/stack_test.c +17
-1+ err =reftable_iterator_next_ref (&it, &rec); +EXPECT_ERR (err); +EXPECT (!strcmp (rec.refname,"refs/heads/branch-0000" )); + err =reftable_iterator_next_ref (&it, &rec); +EXPECT_ERR (err); +EXPECT (!strcmp (rec.refname,"refs/heads/branch-0001" )); + err =reftable_iterator_next_ref (&it, &rec); +EXPECT (err >0 ); + +reftable_ref_record_release (&rec); +reftable_iterator_destroy (&it); +reftable_stack_destroy (st1); +reftable_stack_destroy (st2); +clear_dir (dir); +} +int stack_test_main (int argc,const char *argv[]) {RUN_TEST (test_empty_add);@@ -1098,6 +1146,7 @@ int stack_test_main(int argc, const char *argv[])RUN_TEST (test_reftable_stack_auto_compaction_fails_gracefully);RUN_TEST (test_reftable_stack_update_index_check);RUN_TEST (test_reftable_stack_uptodate); +RUN_TEST (test_reftable_stack_read_across_reload);RUN_TEST (test_suggest_compaction_segment);RUN_TEST (test_suggest_compaction_segment_nothing);return 0 ; --2.46.0.164.g477ce5ccd6.dirty
Re: [PATCH 08/10] reftable/reader: keep readers alive during iteration
Quoted 26 lines · karthik nayak
Quoted 25 lines · Patrick Steinhardt
reftable/stack_test.c +17
-0@@ -1076,6 +1076,54 @@ static void test_reftable_stack_compaction_concurrent_clean(void)clear_dir (dir); } +static void test_reftable_stack_read_across_reload (void ) +{ +struct reftable_write_options opts = {0 }; +struct reftable_stack *st1 =NULL , *st2 =NULL ; +struct reftable_ref_record rec = {0 }; +struct reftable_iterator it = {0 }; +char *dir =get_tmp_dir (__LINE__); +int err; + +/* Create a first stack and set up an iterator for it. */ + err =reftable_new_stack (&st1, dir, &opts); +EXPECT_ERR (err); +write_n_ref_tables (st1,2 ); +EXPECT (st1->merged->readers_len ==2 ); +reftable_stack_init_ref_iterator (st1, &it); + err =reftable_iterator_seek_ref (&it,"" ); +EXPECT_ERR (err);
Quoted 8 lines
reftable/stack_test.c static void test_reftable_stack_compaction_concurrent_clean(void) · line 1101
Quoted 6 lines
reftable/stack_test.c +6
-0+ +/* Set up a second stack for the same directory and compact it. */ + err =reftable_new_stack (&st2, dir, &opts); +EXPECT_ERR (err); +EXPECT (st2->merged->readers_len ==2 ); + err =reftable_stack_compact_all (st2,NULL );Shouldn't we verify that
EXPECT(st2->merged->readers_len == 1);here?
Yes, we should.
Quoted 5 lines
Quoted 5 lines
reftable/stack_test.c +5
-0+EXPECT_ERR (err); + +/* +* Verify that we can continue to use the old iterator even after we +* have reloaded its stack.
Quoted 8 lines
reftable/stack_test.c static void test_reftable_stack_compaction_concurrent_clean(void) · line 1110
Quoted 4 lines
reftable/stack_test.c +4
-0+*/ + err =reftable_stack_reload (st1); +EXPECT_ERR (err); +EXPECT (st2->merged->readers_len ==1 );Oh we do it here, I would've expected it above the
st1reload. And probably expectedEXPECT(st1->merged->readers_len == 2);here to confirm that we still have the readers.
And yes, this was a typo, it should've been st1 indeed. Good eyes, thanks!
Patrick
PATCH 9/10 reftable/stack: reorder swapping in the reloaded stack contents
In reply to Patrick SteinhardtThe code flow of how we swap in the reloaded stack contents is somewhat convoluted because we switch back and forth between swapping in different parts of the stack.
Reorder the code to simplify it. We now first close and unlink the old tables which do not get reused before we update the stack to point to the new stack.
--- reftable/stack.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-)
reftable/stack.c +19 -15
@@ -273,30 +273,34 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (err <0 ) goto done; - st->readers_len = new_readers_len; -if (st->merged) -reftable_merged_table_free (st->merged); -if (st->readers) { -reftable_free (st->readers); - } - st->readers = new_readers; - new_readers =NULL ; - new_readers_len =0 ; - - new_merged->suppress_deletions =1 ; - st->merged = new_merged; +/* +* Close the old, non-reused readers and proactively try to unlink +* them. This is done for systems like Windows, where the underlying +* file of such an open reader wouldn't have been possible to be +* unlinked by the compacting process. +*/ for (i =0 ; i < cur_len; i++) {if (cur[i]) {const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reftable_reader_decref (cur[i]); - -/* On Windows, can only unlink after closing. */ unlink (table_path.buf); } } +/* Update the stack to point to the new tables. */ +if (st->merged) +reftable_merged_table_free (st->merged); + new_merged->suppress_deletions =1 ; + st->merged = new_merged; + +if (st->readers) +reftable_free (st->readers); + st->readers = new_readers; + st->readers_len = new_readers_len; + new_readers =NULL ; + new_readers_len =0 ; + done:for (i =0 ; i < new_readers_len; i++)reftable_reader_decref (new_readers[i]);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH 10/10 reftable/stack: fix segfault when reload with reused readers fails
In reply to Patrick SteinhardtIt is expected that reloading the stack fails with concurrent writers, e.g. because a table that we just wanted to read just got compacted. In case we decided to reuse readers this will cause a segfault though because we unconditionally release all new readers, including the reused ones. As those are still referenced by the current stack, the result is that we will eventually try to dereference those already-freed readers.
Fix this bug by incrementing the refcount of reused readers temporarily.
--- reftable/stack.c | 23 +++++++++++++++++ reftable/stack_test.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+)
reftable/stack.c +23 -0
@@ -226,6 +226,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, {size_t cur_len = !st->merged ?0 : st->merged->readers_len;struct reftable_reader **cur =stack_copy_readers (st, cur_len); +struct reftable_reader **reused =NULL ; +size_t reused_len =0 , reused_alloc =0 ;size_t names_len =names_length (names);struct reftable_reader **new_readers =reftable_calloc (names_len,sizeof (*new_readers));
@@ -245,6 +247,18 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (cur[i] &&0 ==strcmp (cur[i]->name, name)) { rd = cur[i]; cur[i] =NULL ; + +/* +* When reloading the stack fails, we end up +* releasing all new readers. This also +* includes the reused readers, even though +* they are still in used by the old stack. We +* thus need to keep them alive here, which we +* do by bumping their refcount. +*/ +REFTABLE_ALLOC_GROW (reused, reused_len +1 , reused_alloc); + reused[reused_len++] = rd; +reftable_reader_incref (rd);break ; } }
@@ -301,10 +315,19 @@ static int reftable_stack_reload_once(struct reftable_stack *st, new_readers =NULL ; new_readers_len =0 ; +/* +* Decrement the refcount of reused readers again. This only needs to +* happen on the successful case, because on the unsuccessful one we +* decrement their refcount via `new_readers`. +*/ +for (i =0 ; i < reused_len; i++) +reftable_reader_decref (reused[i]); + done:for (i =0 ; i < new_readers_len; i++)reftable_reader_decref (new_readers[i]);reftable_free (new_readers); +reftable_free (reused);reftable_free (cur);strbuf_release (&table_path);return err;
reftable/stack_test.c +59 -0
77 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Quoted 7 lines · Patrick Steinhardt
Hi,
as reported by Peff in [1], the reftable backend fails in t5616 in a racy manner. The cause of this is that git-maintenance(1) processes leak into subsequent tests and perform concurrent compaction of the stack. The reftable backend is expected to handle this gracefully, but doesn't.
I've not looked into the recent changes made to git-maintenance(1) but my understanding is that, now, git-maintenance(1) can detach itself without relying on git-gc(1)'s detach.
Quoted 5 lines
The issue can surface whenever reloading the reftable stack while an iterator is active. In case some of the tables got compacted, we will end up closing them and thus the iterator now tries to use those closed tables.
Make sense till here.
Quoted 1 line
This patch series fixes that issue by starting to refcount the readers.
Not sure what you mean by 'refcount' here though.
Quoted 11 lines
Each iterator will bump the refcount, thus avoiding the problem. While at it, I also found a second issue where we segfault when reloading a table fails while reusing one of the table readers. In this scenario, we would end up releasing the reader of the stack itself, even though it would still be used by it.
This patch series addresses those issues by making the reftable reader refcounted. The first 6 patches do some simplifications of code which is close. The remaining 4 ones introduce refcounting and wire it up as required.
Perhaps it becomes more clearer as I read through the patches.
[snip]
Quoted 6 lines · Patrick Steinhardt
This patch series fixes that issue by starting to refcount the readers. Each iterator will bump the refcount, thus avoiding the problem. While at it, I also found a second issue where we segfault when reloading a table fails while reusing one of the table readers. In this scenario, we would end up releasing the reader of the stack itself, even though it would still be used by it.
I gave a fairly cursory look over this, as I'm not all that familiar with the reftable code. But everything looked pretty sensible to me.
I wondered how we might test this. It looks like you checked the --stress output (which I confirmed seems fixed for me), along with a synthetic test directly calling various reftable functions. Both seem good to me.
I had hoped to be able to have a non-racy external test, where running actual Git commands showed the segfault in a deterministic way. But I couldn't come up with one. One trick we've used for "pausing" a reading process is to run "cat-file --batch" from a fifo. You can ask it to do some ref lookups, then while it waits for more input, update the reftable, and then ask it to do more.
But I don't think that's sufficient here, as the race happens while we are actually iterating. You'd really need some for_each_ref() callback that blocks in some externally controllable way. Possibly you could do something clever with partial-clone lazy fetches (where you stall the fetch and then do ref updates in the middle), but that is getting pretty convoluted.
So I think the tests you included seem like a good place to stop.
I did have a little trouble applying this for testing. I wanted to do it on 'next', which has the maintenance changes to cause the race. But merging it there ended up with a lot of conflicts with other reftable topics (especially the tests). I was able to resolve them all, but you might be able to make Junio's life easier by coordinating the topics a bit.
-Peff
Quoted 11 lines · Jeff King
Quoted 6 lines · Patrick Steinhardt
This patch series fixes that issue by starting to refcount the readers. Each iterator will bump the refcount, thus avoiding the problem. While at it, I also found a second issue where we segfault when reloading a table fails while reusing one of the table readers. In this scenario, we would end up releasing the reader of the stack itself, even though it would still be used by it.
I gave a fairly cursory look over this, as I'm not all that familiar with the reftable code. But everything looked pretty sensible to me.
Thanks for your review, appreciated!
Quoted 12 lines
I wondered how we might test this. It looks like you checked the --stress output (which I confirmed seems fixed for me), along with a synthetic test directly calling various reftable functions. Both seem good to me.
I had hoped to be able to have a non-racy external test, where running actual Git commands showed the segfault in a deterministic way. But I couldn't come up with one. One trick we've used for "pausing" a reading process is to run "cat-file --batch" from a fifo. You can ask it to do some ref lookups, then while it waits for more input, update the reftable, and then ask it to do more.
Quoted 8 lines
But I don't think that's sufficient here, as the race happens while we are actually iterating. You'd really need some
for_each_ref()callback that blocks in some externally controllable way. Possibly you could do something clever with partial-clone lazy fetches (where you stall the fetch and then do ref updates in the middle), but that is getting pretty convoluted.So I think the tests you included seem like a good place to stop.
Yeah. I think demonstrating the issues on the API level is okayish. It would of course have been nice to also do it via our shell scripts, but as you say, it feels rather fragile.
Quoted 6 lines
I did have a little trouble applying this for testing. I wanted to do it on 'next', which has the maintenance changes to cause the race. But merging it there ended up with a lot of conflicts with other reftable topics (especially the tests). I was able to resolve them all, but you might be able to make Junio's life easier by coordinating the topics a bit.
I can certainly do that. I know that there are conflicts both with the patch series dropping the generic tables and with the patch series that move the reftable unit tests into our own codebase. I did address the former by basing it on top of that series, but didn't yet address the latter.
I'm okay with waiting a bit until most of the conflicting topics land. I guess most of them should be close to landing anyway. Alternatively, I can also pull all of them in as dependencies.
Patrick
Quoted 9 lines · Patrick Steinhardt
I can certainly do that. I know that there are conflicts both with the patch series dropping the generic tables and with the patch series that move the reftable unit tests into our own codebase. I did address the former by basing it on top of that series, but didn't yet address the latter.
I'm okay with waiting a bit until most of the conflicting topics land. I guess most of them should be close to landing anyway. Alternatively, I can also pull all of them in as dependencies.
As long as the resolution you see in 'seen' looks acceptable, I'd rather prefer you do nothing, than a rebase that _reduces_ the number of conflicts (but does not completely eliminate them), when I already have a working resolution recorded in my rerere database.
Of course, updates to polish the substance of the topic are very much appreciated, and I'll redo the resolution in such a case if needed. That's one of the things the maintainers do.
Thanks.
Quoted 12 lines · Junio C Hamano
Quoted 9 lines · Patrick Steinhardt
I can certainly do that. I know that there are conflicts both with the patch series dropping the generic tables and with the patch series that move the reftable unit tests into our own codebase. I did address the former by basing it on top of that series, but didn't yet address the latter.
I'm okay with waiting a bit until most of the conflicting topics land. I guess most of them should be close to landing anyway. Alternatively, I can also pull all of them in as dependencies.
Quoted 8 lines
As long as the resolution you see in 'seen' looks acceptable, I'd rather prefer you do nothing, than a rebase that _reduces_ the number of conflicts (but does not completely eliminate them), when I already have a working resolution recorded in my rerere database.
Of course, updates to polish the substance of the topic are very much appreciated, and I'll redo the resolution in such a case if needed. That's one of the things the maintainers do.
Yup, the merge in seen looks good to me. Thanks!
Patrick
Quoted 6 lines · Jeff King
But I don't think that's sufficient here, as the race happens while we are actually iterating. You'd really need some
for_each_ref()callback that blocks in some externally controllable way. Possibly you could do something clever with partial-clone lazy fetches (where you stall the fetch and then do ref updates in the middle), but that is getting pretty convoluted.
;-)
Quoted 1 line
So I think the tests you included seem like a good place to stop.
Yeah, I agree with this assessment.
Quoted 6 lines
I did have a little trouble applying this for testing. I wanted to do it on 'next', which has the maintenance changes to cause the race. But merging it there ended up with a lot of conflicts with other reftable topics (especially the tests). I was able to resolve them all, but you might be able to make Junio's life easier by coordinating the topics a bit.
I haven't tried merging the topic to 'next', jumping over other topics in flight in 'seen'. Thanks for a heads up, but I am hoping that my rerere database is fairly complete for this topic by now.
Thanks.
v2 cover reftable: fix reload with active iterators
In reply to Patrick SteinhardtHi,
this is the second version of my patch series that fixes issues in the reftable libary caused by having concurrent readers and writers.
Changes compared to v1:
- Remove a spurious change that renamed init_reader() to reader_init(). That function went away in the subsequent commit anyway.
- Fix a typo in a commit message.
- Fix some assertions in one of the added tests.
Thanks!
Patrick
Patrick Steinhardt (10):
reftable/blocksource: drop malloc block source reftable/stack: inline stack_compact_range_stats() reftable/reader: rename reftable_new_reader() reftable/reader: inline init_reader() reftable/reader: inline reader_close() reftable/stack: fix broken refnames in write_n_ref_tables() reftable/reader: introduce refcounting
reftable/reader: keep readers alive during iteration reftable/stack: reorder swapping in the reloaded stack contents reftable/stack: fix segfault when reload with reused readers fails
reftable/block_test.c | 3 +- reftable/blocksource.c | 20 ----- reftable/blocksource.h | 2 - reftable/reader.c | 149 ++++++++++++++++--------------- reftable/reader.h | 5 +- reftable/readwrite_test.c | 85 +++++++++--------- reftable/reftable-reader.h | 19 ++-- reftable/stack.c | 90 +++++++++++-------- reftable/stack_test.c | 116 +++++++++++++++++++++++- t/helper/test-reftable.c | 4 +- t/unit-tests/t-reftable-merged.c | 10 +-- 11 files changed, 312 insertions(+), 191 deletions(-)
Range-diff against v1:
1: fee3d3523eb = 1: fee3d3523eb reftable/blocksource: drop malloc block source 2: 3c0cf2bf46f = 2: 3c0cf2bf46f reftable/stack: inline `stack_compact_range_stats()` 3: e658b372f04 ! 3: b4cf97bf758 reftable/reader: rename `reftable_new_reader()`
@@ reftable/reader.c: int reftable_reader_print_blocks(const char *tablename)
goto done;
- ## reftable/reader.h ##
-@@ reftable/reader.h: struct reftable_reader {
- struct reftable_reader_offsets log_offsets;
- };
-
--int init_reader(struct reftable_reader *r, struct reftable_block_source *source, -+int reader_init(struct reftable_reader *r, struct reftable_block_source *source,
- const char *name);
- void reader_close(struct reftable_reader *r);
- const char *reader_name(struct reftable_reader *r);
-
## reftable/readwrite_test.c ##
@@ reftable/readwrite_test.c: static void test_write_empty_table(void)
4: f628b7dafb9 ! 4: 3b667097501 reftable/reader: inline init_reader()
@@ reftable/reader.h: struct reftable_reader {
struct reftable_reader_offsets log_offsets;
};
--int reader_init(struct reftable_reader *r, struct reftable_block_source *source,
+-int init_reader(struct reftable_reader *r, struct reftable_block_source *source,
- const char *name);
void reader_close(struct reftable_reader *r);
const char *reader_name(struct reftable_reader *r);
5: 4a9fe150427 = 5: b129d8a8687 reftable/reader: inline `reader_close()`
6: 4965402e7bf = 6: e3b28709b5f reftable/stack: fix broken refnames in `write_n_ref_tables()`
7: fc0ed68d467 = 7: 6535d1ca9de reftable/reader: introduce refcounting
8: 02682056288 ! 8: 8d08c3bc515 reftable/reader: keep readers alive during iteration
@@ Metadata
## Commit message ##
reftable/reader: keep readers alive during iteration
- The lifetime of a table iterator may surive the lifetime of a reader
+ The lifetime of a table iterator may survive the lifetime of a reader
when the stack gets reloaded. Keep the reader from being released by
increasing its refcount while the iterator is still being used.
@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_cle
+ EXPECT(st2->merged->readers_len == 2);
+ err = reftable_stack_compact_all(st2, NULL);
+ EXPECT_ERR(err);
++ EXPECT(st2->merged->readers_len == 1);
+
+ /*
+ * Verify that we can continue to use the old iterator even after we
@@ reftable/stack_test.c: static void test_reftable_stack_compaction_concurrent_cle
+ */
+ err = reftable_stack_reload(st1);
+ EXPECT_ERR(err);
-+ EXPECT(st2->merged->readers_len == 1);
++ EXPECT(st1->merged->readers_len == 1);
+ err = reftable_iterator_next_ref(&it, &rec);
+ EXPECT_ERR(err);
+ EXPECT(!strcmp(rec.refname, "refs/heads/branch-0000"));
9: d98316fbf4c = 9: 5aee91de25e reftable/stack: reorder swapping in the reloaded stack contents
10: b777818ea99 = 10: 4a8d45cc9b4 reftable/stack: fix segfault when reload with reused readers failsSignature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 1/10 reftable/blocksource: drop malloc block source
The reftable blocksource provides a generic interface to read blocks via different sources, e.g. from disk or from memory. One of the block sources is the malloc block source, which can in theory read data from memory. We nowadays also have a strbuf block source though, which provides essentially the same functionality with better ergonomics.
Adapt the only remaining user of the malloc block source in our tests to use the strbuf block source, instead, and remove the now-unused malloc block source.
--- reftable/block_test.c | 3 ++- reftable/blocksource.c | 20 -------------------- reftable/blocksource.h | 2 -- 3 files changed, 2 insertions(+), 23 deletions(-)
reftable/block_test.c +2 -1
@@ -34,11 +34,12 @@ static void test_block_read_write(void)struct block_reader br = {0 };struct block_iter it =BLOCK_ITER_INIT ;int j =0 ; +struct strbuf block_data =STRBUF_INIT ;struct strbuf want =STRBUF_INIT ;REFTABLE_CALLOC_ARRAY (block.data, block_size); block.len = block_size; - block.source =malloc_block_source (); +block_source_from_strbuf (&block.source, &block_data);block_writer_init (&bw,BLOCK_TYPE_REF , block.data, block_size, header_off,hash_size (GIT_SHA1_FORMAT_ID ));
reftable/blocksource.c +0 -20
@@ -55,26 +55,6 @@ void block_source_from_strbuf(struct reftable_block_source *bs, bs->arg = buf; } -static void malloc_return_block (void *b,struct reftable_block *dest) -{ -if (dest->len) -memset (dest->data,0xff , dest->len); -reftable_free (dest->data); -} - -static struct reftable_block_source_vtable malloc_vtable = { - .return_block = &malloc_return_block, -}; - -static struct reftable_block_source malloc_block_source_instance = { - .ops = &malloc_vtable, -}; - -struct reftable_block_source malloc_block_source (void ) -{ -return malloc_block_source_instance; -} -struct file_block_source {uint64_t size;unsigned char *data;
reftable/blocksource.h +0 -2
@@ -17,6 +17,4 @@ struct reftable_block_source;void block_source_from_strbuf (struct reftable_block_source *bs,struct strbuf *buf); -struct reftable_block_source malloc_block_source (void ); -#endif
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 2/10 reftable/stack: inline `stack_compact_range_stats()`
In reply to Patrick SteinhardtThe only difference between stack_compact_range_stats() and stack_compact_range() is that the former updates stats on failure, whereas the latter doesn't. There are no callers anymore that do not want their stats updated though, making the indirection unnecessary.
Inline the stat updates into stack_compact_range().
--- reftable/stack.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)
reftable/stack.c +4 -12
@@ -1328,17 +1328,9 @@ static int stack_compact_range(struct reftable_stack *st,strbuf_release (&table_name);free_names (names); -return err; -} - -static int stack_compact_range_stats (struct reftable_stack *st, -size_t first,size_t last, -struct reftable_log_expiry_config *config, -unsigned int flags) -{ -int err =stack_compact_range (st, first, last, config, flags);if (err ==REFTABLE_LOCK_ERROR ) st->stats.failures++; +return err; }
@@ -1346,7 +1338,7 @@ int reftable_stack_compact_all(struct reftable_stack *st, struct reftable_log_expiry_config *config) {size_t last = st->merged->readers_len ? st->merged->readers_len -1 :0 ; -return stack_compact_range_stats (st,0 , last, config,0 ); +return stack_compact_range (st,0 , last, config,0 ); }static int segment_size (struct segment *s)
@@ -1452,8 +1444,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) st->opts.auto_compaction_factor);reftable_free (sizes);if (segment_size (&seg) >0 ) -return stack_compact_range_stats (st, seg.start, seg.end -1 , -NULL ,STACK_COMPACT_RANGE_BEST_EFFORT ); +return stack_compact_range (st, seg.start, seg.end -1 , +NULL ,STACK_COMPACT_RANGE_BEST_EFFORT );return 0 ; }
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 3/10 reftable/reader: rename `reftable_new_reader()`
In reply to Patrick SteinhardtRename the reftable_new_reader() function to reftable_reader_new() to match our coding guidelines.
--- reftable/reader.c | 4 ++-- reftable/readwrite_test.c | 6 +++--- reftable/reftable-reader.h | 4 ++-- reftable/stack.c | 4 ++-- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 6 +++--- 6 files changed, 13 insertions(+), 13 deletions(-)
reftable/reader.c +2 -2
@@ -637,7 +637,7 @@ void reader_close(struct reftable_reader *r)FREE_AND_NULL (r->name); } -int reftable_new_reader (struct reftable_reader **p, +int reftable_reader_new (struct reftable_reader **p,struct reftable_block_source *src,char const *name) {struct reftable_reader *rd =reftable_calloc (1 ,sizeof (*rd));
@@ -786,7 +786,7 @@ int reftable_reader_print_blocks(const char *tablename)if (err <0 ) goto done; - err =reftable_new_reader (&r, &src, tablename); + err =reftable_reader_new (&r, &src, tablename);if (err <0 ) goto done;
reftable/readwrite_test.c +3 -3
@@ -646,7 +646,7 @@ static void test_write_empty_table(void)block_source_from_strbuf (&source, &buf); - err =reftable_new_reader (&rd, &source,"filename" ); + err =reftable_reader_new (&rd, &source,"filename" );EXPECT_ERR (err);reftable_reader_init_ref_iterator (rd, &it);
@@ -850,7 +850,7 @@ static void test_write_multiple_indices(void)EXPECT (stats->log_stats.index_offset >0 );block_source_from_strbuf (&source, &writer_buf); - err =reftable_new_reader (&reader, &source,"filename" ); + err =reftable_reader_new (&reader, &source,"filename" );EXPECT_ERR (err);/*
@@ -907,7 +907,7 @@ static void test_write_multi_level_index(void)EXPECT (stats->ref_stats.max_index_level ==2 );block_source_from_strbuf (&source, &writer_buf); - err =reftable_new_reader (&reader, &source,"filename" ); + err =reftable_reader_new (&reader, &source,"filename" );EXPECT_ERR (err);/*
reftable/reftable-reader.h +2 -2
@@ -23,14 +23,14 @@/* The reader struct is a handle to an open reftable file. */ struct reftable_reader ; -/* reftable_new_reader opens a reftable for reading. If successful, +/* reftable_reader_new opens a reftable for reading. If successful, * returns 0 code and sets pp. The name is used for creating a * stack. Typically, it is the basename of the file. The block source * `src` is owned by the reader, and is closed on calling * reftable_reader_destroy(). On error, the block source `src` is * closed as well. */ -int reftable_new_reader (struct reftable_reader **pp, +int reftable_reader_new (struct reftable_reader **pp,struct reftable_block_source *src,const char *name);/* Initialize a reftable iterator for reading refs. */
reftable/stack.c +2 -2
@@ -258,7 +258,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (err <0 ) goto done; - err =reftable_new_reader (&rd, &src, name); + err =reftable_reader_new (&rd, &src, name);if (err <0 ) goto done; }
@@ -1532,7 +1532,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max,if (err <0 ) goto done; - err =reftable_new_reader (&rd, &src, name); + err =reftable_reader_new (&rd, &src, name);if (err <0 ) goto done;
t/helper/test-reftable.c +1 -1
@@ -139,7 +139,7 @@ static int dump_reftable(const char *tablename)if (err <0 ) goto done; - err =reftable_new_reader (&r, &src, tablename); + err =reftable_reader_new (&r, &src, tablename);if (err <0 ) goto done;
t/unit-tests/t-reftable-merged.c +3 -3
@@ -102,7 +102,7 @@ merged_table_from_records(struct reftable_ref_record **refs,write_test_table (&buf[i], refs[i], sizes[i]);block_source_from_strbuf (&(*source)[i], &buf[i]); - err =reftable_new_reader (&(*readers)[i], &(*source)[i], + err =reftable_reader_new (&(*readers)[i], &(*source)[i],"name" );check (!err); }
@@ -277,7 +277,7 @@ merged_table_from_log_records(struct reftable_log_record **logs,write_test_log_table (&buf[i], logs[i], sizes[i], i +1 );block_source_from_strbuf (&(*source)[i], &buf[i]); - err =reftable_new_reader (&(*readers)[i], &(*source)[i], + err =reftable_reader_new (&(*readers)[i], &(*source)[i],"name" );check (!err); }
@@ -426,7 +426,7 @@ static void t_default_write_opts(void)block_source_from_strbuf (&source, &buf); - err =reftable_new_reader (&rd, &source,"filename" ); + err =reftable_reader_new (&rd, &source,"filename" );check (!err); hash_id =reftable_reader_hash_id (rd);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 4/10 reftable/reader: inline `init_reader()`
In reply to Patrick SteinhardtMost users use an allocated version of the reftable_reader, except for some tests. We are about to convert the reader to become refcounted though, and providing the ability to keep a reader on the stack makes this conversion harder than necessary.
Update the tests to use reftable_reader_new() instead to prepare for this change.
--- reftable/reader.c | 122 +++++++++++++++++++------------------- reftable/reader.h | 2 - reftable/readwrite_test.c | 73 ++++++++++++----------- 3 files changed, 98 insertions(+), 99 deletions(-)
reftable/reader.c +61 -61
@@ -162,58 +162,6 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,return err; } -int init_reader (struct reftable_reader *r,struct reftable_block_source *source, -const char *name) -{ -struct reftable_block footer = {NULL }; -struct reftable_block header = {NULL }; -int err =0 ; -uint64_t file_size =block_source_size (source); - -/* Need +1 to read type of first block. */ -uint32_t read_size =header_size (2 ) +1 ;/* read v2 because it's larger. */ -memset (r,0 ,sizeof (struct reftable_reader )); - -if (read_size > file_size) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - - err =block_source_read_block (source, &header,0 , read_size); -if (err != read_size) { - err =REFTABLE_IO_ERROR ; - goto done; - } - -if (memcmp (header.data,"REFT" ,4 )) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - r->version = header.data[4 ]; -if (r->version !=1 && r->version !=2 ) { - err =REFTABLE_FORMAT_ERROR ; - goto done; - } - - r->size = file_size -footer_size (r->version); - r->source = *source; - r->name =xstrdup (name); - r->hash_id =0 ; - - err =block_source_read_block (source, &footer, r->size, -footer_size (r->version)); -if (err !=footer_size (r->version)) { - err =REFTABLE_IO_ERROR ; - goto done; - } - - err =parse_footer (r, footer.data, header.data); -done: -reftable_block_done (&footer); -reftable_block_done (&header); -return err; -} -struct table_iter {struct reftable_reader *r;uint8_t typ;
@@ -637,16 +585,68 @@ void reader_close(struct reftable_reader *r)FREE_AND_NULL (r->name); } -int reftable_reader_new (struct reftable_reader **p, -struct reftable_block_source *src,char const *name) +int reftable_reader_new (struct reftable_reader **out, +struct reftable_block_source *source,char const *name) { -struct reftable_reader *rd =reftable_calloc (1 ,sizeof (*rd)); -int err =init_reader (rd, src, name); -if (err ==0 ) { - *p = rd; - }else { -block_source_close (src); -reftable_free (rd); +struct reftable_block footer = {0 }; +struct reftable_block header = {0 }; +struct reftable_reader *r; +uint64_t file_size =block_source_size (source); +uint32_t read_size; +int err; + +REFTABLE_CALLOC_ARRAY (r,1 ); + +/* +* We need one extra byte to read the type of first block. We also +* pretend to always be reading v2 of the format because it is larger. +*/ + read_size =header_size (2 ) +1 ; +if (read_size > file_size) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + + err =block_source_read_block (source, &header,0 , read_size); +if (err != read_size) { + err =REFTABLE_IO_ERROR ; + goto done; + } + +if (memcmp (header.data,"REFT" ,4 )) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + r->version = header.data[4 ]; +if (r->version !=1 && r->version !=2 ) { + err =REFTABLE_FORMAT_ERROR ; + goto done; + } + + r->size = file_size -footer_size (r->version); + r->source = *source; + r->name =xstrdup (name); + r->hash_id =0 ; + + err =block_source_read_block (source, &footer, r->size, +footer_size (r->version)); +if (err !=footer_size (r->version)) { + err =REFTABLE_IO_ERROR ; + goto done; + } + + err =parse_footer (r, footer.data, header.data); +if (err) + goto done; + + *out = r; + +done: +reftable_block_done (&footer); +reftable_block_done (&header); +if (err) { +reftable_free (r); +block_source_close (source); }return err; }
reftable/reader.h +0 -2
@@ -52,8 +52,6 @@ struct reftable_reader {struct reftable_reader_offsets log_offsets; }; -int init_reader (struct reftable_reader *r,struct reftable_block_source *source, -const char *name);void reader_close (struct reftable_reader *r);const char *reader_name (struct reftable_reader *r);
reftable/readwrite_test.c +37 -36
240 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 5/10 reftable/reader: inline `reader_close()`
In reply to Patrick SteinhardtSame as with the preceding commit, we also provide a reader_close() function that allows the caller to close a reader without freeing it. This is unnecessary now that all users will have an allocated version of the reader.
Inline it into reftable_reader_free().
--- reftable/reader.c | 9 ++------- reftable/reader.h | 1 - reftable/stack.c | 5 +---- 3 files changed, 3 insertions(+), 12 deletions(-)
reftable/reader.c +2 -7
@@ -579,12 +579,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r,reader_init_iter (r, it,BLOCK_TYPE_LOG ); } -void reader_close (struct reftable_reader *r) -{ -block_source_close (&r->source); -FREE_AND_NULL (r->name); -} -int reftable_reader_new (struct reftable_reader **out,struct reftable_block_source *source,char const *name) {
@@ -655,7 +649,8 @@ void reftable_reader_free(struct reftable_reader *r) {if (!r)return ; -reader_close (r); +block_source_close (&r->source); +FREE_AND_NULL (r->name);reftable_free (r); }
reftable/reader.h +0 -1
@@ -52,7 +52,6 @@ struct reftable_reader {struct reftable_reader_offsets log_offsets; }; -void reader_close (struct reftable_reader *r);const char *reader_name (struct reftable_reader *r); void reader_init_iter(struct reftable_reader *r,
reftable/stack.c +1 -4
@@ -290,7 +290,6 @@ static int reftable_stack_reload_once(struct reftable_stack *st,const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reader_close (cur[i]);reftable_reader_free (cur[i]);/* On Windows, can only unlink after closing. */
@@ -299,10 +298,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, } done: -for (i =0 ; i < new_readers_len; i++) { -reader_close (new_readers[i]); +for (i =0 ; i < new_readers_len; i++)reftable_reader_free (new_readers[i]); - }reftable_free (new_readers);reftable_free (cur);strbuf_release (&table_path);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 6/10 reftable/stack: fix broken refnames in `write_n_ref_tables()`
In reply to Patrick SteinhardtThe write_n_ref_tables() helper function writes N references in separate tables. We never reset the computed name of those references though, leading us to end up with unexpected names.
Fix this by resetting the buffer.
--- reftable/stack_test.c | 1 + 1 file changed, 1 insertion(+)
reftable/stack_test.c +1 -0
@@ -125,6 +125,7 @@ static void write_n_ref_tables(struct reftable_stack *st, .value_type = REFTABLE_REF_VAL1, }; +strbuf_reset (&buf);strbuf_addf (&buf,"refs/heads/branch-%04u" , (unsigned ) i); ref.refname = buf.buf;set_test_hash (ref.value.val1, i);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 7/10 reftable/reader: introduce refcounting
In reply to Patrick SteinhardtIt was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads.
Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults.
One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one:
+ git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL
=================================================================
==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)
==657994==The signal is caused by a READ memory access.
#0 0x55f23e52ddf9 in get_var_int reftable/record.c:29
#1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170
#2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194
#3 0x55f23e54e72e in block_iter_next reftable/block.c:398
#4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240
#5 0x55f23e5573dc in table_iter_next reftable/reader.c:355
#6 0x55f23e5573dc in table_iter_next reftable/reader.c:339
#7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69
#8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123
#9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172
#10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175
#11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464
#12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13
#13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452
#14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623
#15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659
#16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133
#17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432
#18 0x55f23dba7764 in run_builtin git.c:484
#19 0x55f23dba7764 in handle_builtin git.c:741
#20 0x55f23dbab61e in run_argv git.c:805
#21 0x55f23dbab61e in cmd_main git.c:1000
#22 0x55f23dba4781 in main common-main.c:64
#23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360
#25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't.
The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling reftable_stack_reload(), which is wrong. We can only close them once we know that there are no iterators using them anymore.
Prepare for a fix by converting the reftable readers to be refcounted.
--- reftable/reader.c | 16 ++++++++++++++-- reftable/reader.h | 2 ++ reftable/readwrite_test.c | 18 +++++++++--------- reftable/reftable-reader.h | 15 ++++++++++++--- reftable/stack.c | 8 ++++---- reftable/stack_test.c | 6 ++---- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 4 ++-- 8 files changed, 46 insertions(+), 25 deletions(-)
reftable/reader.c +14 -2
@@ -621,6 +621,7 @@ int reftable_reader_new(struct reftable_reader **out, r->source = *source; r->name =xstrdup (name); r->hash_id =0 ; + r->refcount =1 ; err =block_source_read_block (source, &footer, r->size,footer_size (r->version));
@@ -645,10 +646,21 @@ int reftable_reader_new(struct reftable_reader **out,return err; } -void reftable_reader_free (struct reftable_reader *r) +void reftable_reader_incref (struct reftable_reader *r) +{ +if (!r->refcount) +BUG ("cannot increment ref counter of dead reader" ); + r->refcount++; +} + +void reftable_reader_decref (struct reftable_reader *r) {if (!r)return ; +if (!r->refcount) +BUG ("cannot decrement ref counter of dead reader" ); +if (--r->refcount) +return ;block_source_close (&r->source);FREE_AND_NULL (r->name);reftable_free (r);
@@ -812,7 +824,7 @@ int reftable_reader_print_blocks(const char *tablename) } done: -reftable_reader_free (r); +reftable_reader_decref (r);table_iter_close (&ti);return err; }
reftable/reader.h +2 -0
@@ -50,6 +50,8 @@ struct reftable_reader {struct reftable_reader_offsets ref_offsets;struct reftable_reader_offsets obj_offsets;struct reftable_reader_offsets log_offsets; + +uint64_t refcount; };const char *reader_name (struct reftable_reader *r);
reftable/readwrite_test.c +9 -9
@@ -279,7 +279,7 @@ static void test_log_write_read(void)/* cleanup. */ strbuf_release (&buf);free_names (names); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_log_zlib_corruption (void )
@@ -341,7 +341,7 @@ static void test_log_zlib_corruption(void)reftable_iterator_destroy (&it);/* cleanup. */ -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf); }
@@ -383,7 +383,7 @@ static void test_table_read_write_sequential(void)EXPECT (j ==N );reftable_iterator_destroy (&it); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf);free_names (names); }
@@ -431,7 +431,7 @@ static void test_table_read_api(void) }reftable_iterator_destroy (&it);reftable_free (names); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&buf); }
@@ -498,7 +498,7 @@ static void test_table_read_write_seek(int index, int hash_id)reftable_free (names[i]); }reftable_free (names); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_table_read_write_seek_linear (void )
@@ -611,7 +611,7 @@ static void test_table_refs_for(int indexed)strbuf_release (&buf);free_names (want_names);reftable_iterator_destroy (&it); -reftable_reader_free (reader); +reftable_reader_decref (reader); }static void test_table_refs_for_no_index (void )
@@ -657,7 +657,7 @@ static void test_write_empty_table(void)EXPECT (err >0 );reftable_iterator_destroy (&it); -reftable_reader_free (rd); +reftable_reader_decref (rd);strbuf_release (&buf); }
@@ -863,7 +863,7 @@ static void test_write_multiple_indices(void)reftable_iterator_destroy (&it);reftable_writer_free (writer); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&writer_buf);strbuf_release (&buf); }
@@ -919,7 +919,7 @@ static void test_write_multi_level_index(void)reftable_iterator_destroy (&it);reftable_writer_free (writer); -reftable_reader_free (reader); +reftable_reader_decref (reader);strbuf_release (&writer_buf);strbuf_release (&buf); }
reftable/reftable-reader.h +12 -3
@@ -33,6 +33,18 @@ struct reftable_reader;int reftable_reader_new (struct reftable_reader **pp,struct reftable_block_source *src,const char *name); +/* +* Manage the reference count of the reftable reader. A newly initialized +* reader starts with a refcount of 1 and will be deleted once the refcount has +* reached 0. +* +* This is required because readers may have longer lifetimes than the stack +* they belong to. The stack may for example be reloaded while the old tables +* are still being accessed by an iterator. +*/ +void reftable_reader_incref (struct reftable_reader *reader); +void reftable_reader_decref (struct reftable_reader *reader); +/* Initialize a reftable iterator for reading refs. */ void reftable_reader_init_ref_iterator (struct reftable_reader *r,struct reftable_iterator *it);
@@ -44,9 +56,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r,/* returns the hash ID used in this table. */ uint32_t reftable_reader_hash_id (struct reftable_reader *r); -/* closes and deallocates a reader. */ -void reftable_reader_free (struct reftable_reader *); -/* return an iterator for the refs pointing to `oid`. */ int reftable_reader_refs_for (struct reftable_reader *r,struct reftable_iterator *it,uint8_t *oid);
reftable/stack.c +4 -4
@@ -186,7 +186,7 @@ void reftable_stack_destroy(struct reftable_stack *st)if (names && !has_name (names, name)) {stack_filename (&filename, st, name); } -reftable_reader_free (st->readers[i]); +reftable_reader_decref (st->readers[i]);if (filename.len) {/* On Windows, can only unlink after closing. */
@@ -290,7 +290,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st,const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reftable_reader_free (cur[i]); +reftable_reader_decref (cur[i]);/* On Windows, can only unlink after closing. */ unlink (table_path.buf);
@@ -299,7 +299,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, done:for (i =0 ; i < new_readers_len; i++) -reftable_reader_free (new_readers[i]); +reftable_reader_decref (new_readers[i]);reftable_free (new_readers);reftable_free (cur);strbuf_release (&table_path);
@@ -1534,7 +1534,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max, goto done; update_idx =reftable_reader_max_update_index (rd); -reftable_reader_free (rd); +reftable_reader_decref (rd);if (update_idx <= max) {unlink (table_path.buf);
reftable/stack_test.c +2 -4
@@ -1036,10 +1036,8 @@ static void test_reftable_stack_compaction_concurrent(void)static void unclean_stack_close (struct reftable_stack *st) {/* break abstraction boundary to simulate unclean shutdown. */ -int i =0 ; -for (; i < st->readers_len; i++) { -reftable_reader_free (st->readers[i]); - } +for (size_t i =0 ; i < st->readers_len; i++) +reftable_reader_decref (st->readers[i]); st->readers_len =0 ;FREE_AND_NULL (st->readers); }
t/helper/test-reftable.c +1 -1
@@ -152,7 +152,7 @@ static int dump_reftable(const char *tablename) done:reftable_merged_table_free (mt); -reftable_reader_free (r); +reftable_reader_decref (r);return err; }
t/unit-tests/t-reftable-merged.c +2 -2
@@ -115,7 +115,7 @@ merged_table_from_records(struct reftable_ref_record **refs,static void readers_destroy (struct reftable_reader **readers,const size_t n) {for (size_t i =0 ; i < n; i++) -reftable_reader_free (readers[i]); +reftable_reader_decref (readers[i]);reftable_free (readers); }
@@ -437,7 +437,7 @@ static void t_default_write_opts(void) err =reftable_merged_table_new (&merged, &rd,1 ,GIT_SHA1_FORMAT_ID );check (!err); -reftable_reader_free (rd); +reftable_reader_decref (rd);reftable_merged_table_free (merged);strbuf_release (&buf); }
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 8/10 reftable/reader: keep readers alive during iteration
In reply to Patrick SteinhardtThe lifetime of a table iterator may survive the lifetime of a reader when the stack gets reloaded. Keep the reader from being released by increasing its refcount while the iterator is still being used.
--- reftable/reader.c | 2 ++ reftable/stack_test.c | 50 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
reftable/reader.c +2 -0
@@ -175,6 +175,7 @@ static int table_iter_init(struct table_iter *ti, struct reftable_reader *r) {struct block_iter bi =BLOCK_ITER_INIT ;memset (ti,0 ,sizeof (*ti)); +reftable_reader_incref (r); ti->r = r; ti->bi = bi;return 0 ;
@@ -262,6 +263,7 @@ static void table_iter_close(struct table_iter *ti) {table_iter_block_done (ti);block_iter_close (&ti->bi); +reftable_reader_decref (ti->r); }static int table_iter_next_block (struct table_iter *ti)
reftable/stack_test.c +50 -0
62 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 9/10 reftable/stack: reorder swapping in the reloaded stack contents
In reply to Patrick SteinhardtThe code flow of how we swap in the reloaded stack contents is somewhat convoluted because we switch back and forth between swapping in different parts of the stack.
Reorder the code to simplify it. We now first close and unlink the old tables which do not get reused before we update the stack to point to the new stack.
--- reftable/stack.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-)
reftable/stack.c +19 -15
@@ -273,30 +273,34 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (err <0 ) goto done; - st->readers_len = new_readers_len; -if (st->merged) -reftable_merged_table_free (st->merged); -if (st->readers) { -reftable_free (st->readers); - } - st->readers = new_readers; - new_readers =NULL ; - new_readers_len =0 ; - - new_merged->suppress_deletions =1 ; - st->merged = new_merged; +/* +* Close the old, non-reused readers and proactively try to unlink +* them. This is done for systems like Windows, where the underlying +* file of such an open reader wouldn't have been possible to be +* unlinked by the compacting process. +*/ for (i =0 ; i < cur_len; i++) {if (cur[i]) {const char *name =reader_name (cur[i]);stack_filename (&table_path, st, name); -reftable_reader_decref (cur[i]); - -/* On Windows, can only unlink after closing. */ unlink (table_path.buf); } } +/* Update the stack to point to the new tables. */ +if (st->merged) +reftable_merged_table_free (st->merged); + new_merged->suppress_deletions =1 ; + st->merged = new_merged; + +if (st->readers) +reftable_free (st->readers); + st->readers = new_readers; + st->readers_len = new_readers_len; + new_readers =NULL ; + new_readers_len =0 ; + done:for (i =0 ; i < new_readers_len; i++)reftable_reader_decref (new_readers[i]);
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
PATCH v2 10/10 reftable/stack: fix segfault when reload with reused readers fails
In reply to Patrick SteinhardtIt is expected that reloading the stack fails with concurrent writers, e.g. because a table that we just wanted to read just got compacted. In case we decided to reuse readers this will cause a segfault though because we unconditionally release all new readers, including the reused ones. As those are still referenced by the current stack, the result is that we will eventually try to dereference those already-freed readers.
Fix this bug by incrementing the refcount of reused readers temporarily.
--- reftable/stack.c | 23 +++++++++++++++++ reftable/stack_test.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+)
reftable/stack.c +23 -0
@@ -226,6 +226,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, {size_t cur_len = !st->merged ?0 : st->merged->readers_len;struct reftable_reader **cur =stack_copy_readers (st, cur_len); +struct reftable_reader **reused =NULL ; +size_t reused_len =0 , reused_alloc =0 ;size_t names_len =names_length (names);struct reftable_reader **new_readers =reftable_calloc (names_len,sizeof (*new_readers));
@@ -245,6 +247,18 @@ static int reftable_stack_reload_once(struct reftable_stack *st,if (cur[i] &&0 ==strcmp (cur[i]->name, name)) { rd = cur[i]; cur[i] =NULL ; + +/* +* When reloading the stack fails, we end up +* releasing all new readers. This also +* includes the reused readers, even though +* they are still in used by the old stack. We +* thus need to keep them alive here, which we +* do by bumping their refcount. +*/ +REFTABLE_ALLOC_GROW (reused, reused_len +1 , reused_alloc); + reused[reused_len++] = rd; +reftable_reader_incref (rd);break ; } }
@@ -301,10 +315,19 @@ static int reftable_stack_reload_once(struct reftable_stack *st, new_readers =NULL ; new_readers_len =0 ; +/* +* Decrement the refcount of reused readers again. This only needs to +* happen on the successful case, because on the unsuccessful one we +* decrement their refcount via `new_readers`. +*/ +for (i =0 ; i < reused_len; i++) +reftable_reader_decref (reused[i]); + done:for (i =0 ; i < new_readers_len; i++)reftable_reader_decref (new_readers[i]);reftable_free (new_readers); +reftable_free (reused);reftable_free (cur);strbuf_release (&table_path);return err;
reftable/stack_test.c +59 -0
77 lines not on this page · raw message
Signature · 1 line
2.46.0.164.g477ce5ccd6.dirty
Quoted 9 lines · Patrick Steinhardt
Changes compared to v1:
- Remove a spurious change that renamed
init_reader()toreader_init(). That function went away in the subsequent commit anyway.- Fix a typo in a commit message.
- Fix some assertions in one of the added tests.
All changes relative to v1 looked sensible to me.
Will replace.