could not freshen shared index ..../sharedindex.0000000000000000000000000000000000000000'
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'
(There's a repo called dev_full, and I've got a worktree where I'm working on my 3rd attempt to make it work with gcc8).
That file doesn't actually exist but there are a bunch of sharedindex.XXX files in there with more convincing looking names.
2.20.1.611.gfbb209baf1
Quoted 5 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'
Thanks for the report!
Have you tried to run the test suite with GIT_TEST_SPLIT_INDEX set to "true" on your machine/environment?
Also does the rebase fail or otherwise misbehave?
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'
There are only two places in the code that could print this. The one in read_index_from() can't happen unless is_null_oid() is broken (very very unlikely).
The other one is in write_locked_index() which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
-- 8< --
read-cache.c +1 -0
@@ -3165,6 +3165,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,fill_fsmonitor_bitmap (istate);if (!si || alternate_index_output || + (si &&is_null_oid (&si->base_oid)) || (istate->cache_changed & ~EXTMASK )) {if (si)oidclr (&si->base_oid);
-- 8< --
Quoted 7 lines
(There's a repo called
dev_full, and I've got a worktree where I'm working on my 3rd attempt to make it work with gcc8).That file doesn't actually exist but there are a bunch of sharedindex.XXX files in there with more convincing looking names.
2.20.1.611.gfbb209baf1
--
Duy
Quoted 7 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'
Quoted 7 lines
There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
They've gone away!
Quoted 8 lines
-- 8< --
read-cache.c +0
-0
read-cache.c int write_locked_index(struct index_state *istate, struct lock_file *lock,
Quoted 8 lines
read-cache.c +1
-1@@ -3165,6 +3165,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,fill_fsmonitor_bitmap (istate);if (!si || alternate_index_output || + (si &&is_null_oid (&si->base_oid)) || (istate->cache_changed & ~EXTMASK )) {if (si)oidclr (&si->base_oid); -- 8< --
Luke
Quoted 13 lines · Luke Diamand
Quoted 11 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).
Quoted 5 lines
Quoted 3 lines
The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?They've gone away!
Great! Since you seem able to reproduce this (and can build git!) could you try bisect to pin point the commit that causes this problem? That would help a lot. I think you could start maybe from v2.19.0
Signature · 1 line
Duy
Quoted 19 lines · Duy Nguyen
Quoted 17 lines · Luke Diamand
Quoted 14 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
Quoted 5 lines
Quoted 1 line
They've gone away!
Great! Since you seem able to reproduce this (and can build git!) could you try bisect to pin point the commit that causes this problem? That would help a lot. I think you could start maybe from v2.19.0
The first bad commit was d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2.
$ git show d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2 commit d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2 (refs/bisect/bad) Merge: 6b0f1d9c47 7db118303a Author: Junio C Hamano <[email protected]> Date: Wed May 23 14:38:22 2018 +0900
Merge branch 'en/unpack-trees-split-index-fix'
The split-index feature had a long-standing and dormant bug in certain use of the in-core merge machinery, which has been fixed.
* en/unpack-trees-split-index-fix:
unpack_trees: fix breakage when o->src_index != o->dst_index
The test I'm doing is just:
1. git checkout some_tag
2. git rebase -i HEAD~5
3. Swap the top and bottom commit
4. repeat
I just chose "5" as my first wild guess, other numbers are also available.
With "5" I get 3 lots of:
warning: could not freshen shared index
'.git/sharedindex.0000000000000000000000000000000000000000'
As far as I can tell the actual rebasing is fine.
Quoted 21 lines · Luke Diamand
Quoted 19 lines · Duy Nguyen
Quoted 17 lines · Luke Diamand
Quoted 14 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
They've gone away!
Great! Since you seem able to reproduce this (and can build git!) could you try bisect to pin point the commit that causes this problem? That would help a lot. I think you could start maybe from v2.19.0
The first bad commit was
d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2.
Thanks! I could reproduce it and I think I know what the problem is now. Patches coming soon.
Quoted 29 lines
$ git show d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2 commit d658196f3c4b2478ebdff638e2c3e4fb2f9cbba2 (refs/bisect/bad) Merge: 6b0f1d9c47 7db118303a Author: Junio C Hamano <[email protected]> Date: Wed May 23 14:38:22 2018 +0900Merge branch 'en/unpack-trees-split-index-fix'
The split-index feature had a long-standing and dormant bug in certain use of the in-core merge machinery, which has been fixed.
* en/unpack-trees-split-index-fix:
unpack_trees: fix breakage wheno->src_index!=o->dst_indexThe test I'm doing is just:
1. git checkout
some_tag
2. git rebase -i HEAD~5
3. Swap the top and bottom commit
4. repeatI just chose "5" as my first wild guess, other numbers are also available.
With "5" I get 3 lots of:
warning: could not freshen shared index
'.git/sharedindex.0000000000000000000000000000000000000000'As far as I can tell the actual rebasing is fine.
Signature · 1 line
Duy
PATCH 1/1 unpack-trees.c: fix writing "link" index ext with null base oid
In reply to Luke DiamandSince commit 7db118303a (unpack_trees: fix breakage when o->src_index != o->dst_index - 2018-04-23) and changes in merge code to use separate index_state for source and destination, when doing a merge with split index activated, we may run into this line in unpack_trees():
o->result.split_index = init_split_index(&o->result);
This is by itself not wrong. But this split index information is not fully populated (and it's only so when move_cache_to_base_index() is called, aka force splitting the index, or loading index_state from a file). Both "base_oid" and "base" in this case remain null.
So when writing the main index down, we link to this index with null oid (default value after init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh:
warning: could not freshen shared index '.../sharedindex.0{40}'
This patch makes sure we will not refresh null base_oid (because the file is never there). It also makes sure not to write "link" extension with null base_oid in the first place (no point having it at all). Read code already has protection against null base_oid.
There is also another side fix in remove_split_index() that causes a crash when doing "git update-index --no-split-index" when base_oid in the index file is null. In this case we will not load istate->split_index->base but we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.
---
I considered adding a test, but since the problem is a warning, not sure how to catch that. And a test would not be able to verify all changes in this patch anyway.
read-cache.c | 5 +++-- split-index.c | 34 ++++++++++++++++++---------------- 2 files changed, 21 insertions(+), 18 deletions(-)
read-cache.c +3 -2
@@ -2894,7 +2894,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,return -1 ; } -if (!strip_extensions && istate->split_index) { +if (!strip_extensions && istate->split_index && + !is_null_oid (&istate->split_index->base_oid)) {struct strbuf sb =STRBUF_INIT ; err = write_link_extension(&sb, istate) < 0 ||
@@ -3189,7 +3190,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret =write_split_index (istate, lock, flags);/* Freshen the shared index only if the split-index was written */ -if (!ret && !new_shared_index) { +if (!ret && !new_shared_index && !is_null_oid (&si->base_oid)) {const char *shared_index =git_path ("sharedindex.%s" ,oid_to_hex (&si->base_oid));freshen_shared_index (shared_index,1 );
split-index.c +18 -16
@@ -440,24 +440,26 @@ void add_split_index(struct index_state *istate)void remove_split_index (struct index_state *istate) {if (istate->split_index) { -/* -* When removing the split index, we need to move -* ownership of the mem_pool associated with the -* base index to the main index. There may be cache entries -* allocated from the base's memory pool that are shared with -* the_index.cache[]. -*/ -mem_pool_combine (istate->ce_mem_pool, istate->split_index->base->ce_mem_pool); +if (istate->split_index->base) { +/* +* When removing the split index, we need to move +* ownership of the mem_pool associated with the +* base index to the main index. There may be cache entries +* allocated from the base's memory pool that are shared with +* the_index.cache[]. +*/ +mem_pool_combine (istate->ce_mem_pool, + istate->split_index->base->ce_mem_pool); -/* -* The split index no longer owns the mem_pool backing -* its cache array. As we are discarding this index, -* mark the index as having no cache entries, so it -* will not attempt to clean up the cache entries or -* validate them. -*/ -if (istate->split_index->base) +/* +* The split index no longer owns the mem_pool backing +* its cache array. As we are discarding this index, +* mark the index as having no cache entries, so it +* will not attempt to clean up the cache entries or +* validate them. +*/ istate->split_index->base->cache_nr =0 ; + }/* * We can discard the split index because its
Signature · 1 line
2.20.1.682.gd5861c6d90
Re: [PATCH] unpack-trees.c: fix writing "link" index ext with null base oid
Quoted 114 lines · Nguyễn Thái Ngọc Duy
Since commit
7db118303a(unpack_trees: fix breakage wheno->src_index!=o->dst_index- 2018-04-23) and changes in merge code to use separateindex_statefor source and destination, when doing a merge with split index activated, we may run into this line inunpack_trees():o->result.split_index = init_split_index(&o->result);This is by itself not wrong. But this split index information is not fully populated (and it's only so when
move_cache_to_base_index()is called, aka force splitting the index, or loadingindex_statefrom a file). Both "base_oid" and "base" in this case remain null.So when writing the main index down, we link to this index with null oid (default value after
init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh:warning: could not freshen shared index '.../sharedindex.0{40}'
This patch makes sure we will not refresh null
base_oid(because the file is never there). It also makes sure not to write "link" extension with nullbase_oidin the first place (no point having it at all). Read code already has protection against nullbase_oid.There is also another side fix in
remove_split_index()that causes a crash when doing "git update-index --no-split-index" whenbase_oidin the index file is null. In this case we will not loadistate->split_index->basebut we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.Reported-by Luke Diamand <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>---
I considered adding a test, but since the problem is a warning, not sure how to catch that. And a test would not be able to verify all changes in this patch anyway.read-cache.c | 5 +++-- split-index.c | 34 ++++++++++++++++++---------------- 2 files changed, 21 insertions(+), 18 deletions(-)read-cache.c +3
-2@@ -2894,7 +2894,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,return -1 ; } -if (!strip_extensions && istate->split_index) { +if (!strip_extensions && istate->split_index && + !is_null_oid (&istate->split_index->base_oid)) {struct strbuf sb =STRBUF_INIT ; err = write_link_extension(&sb, istate) < 0 ||@@ -3189,7 +3190,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret =write_split_index (istate, lock, flags);/* Freshen the shared index only if the split-index was written */ -if (!ret && !new_shared_index) { +if (!ret && !new_shared_index && !is_null_oid (&si->base_oid)) {const char *shared_index =git_path ("sharedindex.%s" ,oid_to_hex (&si->base_oid));freshen_shared_index (shared_index,1 );split-index.c +18
-17@@ -440,24 +440,26 @@ void add_split_index(struct index_state *istate)void remove_split_index (struct index_state *istate) {if (istate->split_index) { -/* -* When removing the split index, we need to move -* ownership of the mem_pool associated with the -* base index to the main index. There may be cache entries -* allocated from the base's memory pool that are shared with -* the_index.cache[]. -*/ -mem_pool_combine (istate->ce_mem_pool, istate->split_index->base->ce_mem_pool); +if (istate->split_index->base) { +/* +* When removing the split index, we need to move +* ownership of the mem_pool associated with the +* base index to the main index. There may be cache entries +* allocated from the base's memory pool that are shared with +* the_index.cache[]. +*/ +mem_pool_combine (istate->ce_mem_pool, + istate->split_index->base->ce_mem_pool); -/* -* The split index no longer owns the mem_pool backing -* its cache array. As we are discarding this index, -* mark the index as having no cache entries, so it -* will not attempt to clean up the cache entries or -* validate them. -*/ -if (istate->split_index->base) +/* +* The split index no longer owns the mem_pool backing +* its cache array. As we are discarding this index, +* mark the index as having no cache entries, so it +* will not attempt to clean up the cache entries or +* validate them. +*/ istate->split_index->base->cache_nr =0 ; + }/* * We can discard the split index because its --
Quoted 1 line
2.20.1.682.gd5861c6d90
Ack, I no longer see those messages.
Quoted 1 line
Re: [PATCH] unpack-trees.c: fix writing "link" index ext with null base oid
In reply to Nguyễn Thái Ngọc DuyQuoted 32 lines · Nguyễn Thái Ngọc Duy
Since commit
7db118303a(unpack_trees: fix breakage wheno->src_index!=o->dst_index- 2018-04-23) and changes in merge code to use separateindex_statefor source and destination, when doing a merge with split index activated, we may run into this line inunpack_trees():o->result.split_index = init_split_index(&o->result);This is by itself not wrong. But this split index information is not fully populated (and it's only so when
move_cache_to_base_index()is called, aka force splitting the index, or loadingindex_statefrom a file). Both "base_oid" and "base" in this case remain null.So when writing the main index down, we link to this index with null oid (default value after
init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh:warning: could not freshen shared index '.../sharedindex.0{40}'
This patch makes sure we will not refresh null
base_oid(because the file is never there). It also makes sure not to write "link" extension with nullbase_oidin the first place (no point having it at all). Read code already has protection against nullbase_oid.There is also another side fix in
remove_split_index()that causes a crash when doing "git update-index --no-split-index" whenbase_oidin the index file is null. In this case we will not loadistate->split_index->basebut we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.
Quoted 2 lines
Reported-by Luke Diamand <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>
Thanks for digging this down and fixing it. When I saw this split index bug bisect to me that my heart sank a little; there's so much of that code I don't understand. Nice to see you've already come along and fixed it all up. :-)
Quoted 82 lines
---
I considered adding a test, but since the problem is a warning, not sure how to catch that. And a test would not be able to verify all changes in this patch anyway.read-cache.c | 5 +++-- split-index.c | 34 ++++++++++++++++++---------------- 2 files changed, 21 insertions(+), 18 deletions(-)read-cache.c +3
-2@@ -2894,7 +2894,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,return -1 ; } -if (!strip_extensions && istate->split_index) { +if (!strip_extensions && istate->split_index && + !is_null_oid (&istate->split_index->base_oid)) {struct strbuf sb =STRBUF_INIT ; err = write_link_extension(&sb, istate) < 0 ||@@ -3189,7 +3190,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret =write_split_index (istate, lock, flags);/* Freshen the shared index only if the split-index was written */ -if (!ret && !new_shared_index) { +if (!ret && !new_shared_index && !is_null_oid (&si->base_oid)) {const char *shared_index =git_path ("sharedindex.%s" ,oid_to_hex (&si->base_oid));freshen_shared_index (shared_index,1 );split-index.c +18
-17@@ -440,24 +440,26 @@ void add_split_index(struct index_state *istate)void remove_split_index (struct index_state *istate) {if (istate->split_index) { -/* -* When removing the split index, we need to move -* ownership of the mem_pool associated with the -* base index to the main index. There may be cache entries -* allocated from the base's memory pool that are shared with -* the_index.cache[]. -*/ -mem_pool_combine (istate->ce_mem_pool, istate->split_index->base->ce_mem_pool); +if (istate->split_index->base) { +/* +* When removing the split index, we need to move +* ownership of the mem_pool associated with the +* base index to the main index. There may be cache entries +* allocated from the base's memory pool that are shared with +* the_index.cache[]. +*/ +mem_pool_combine (istate->ce_mem_pool, + istate->split_index->base->ce_mem_pool); -/* -* The split index no longer owns the mem_pool backing -* its cache array. As we are discarding this index, -* mark the index as having no cache entries, so it -* will not attempt to clean up the cache entries or -* validate them. -*/ -if (istate->split_index->base) +/* +* The split index no longer owns the mem_pool backing +* its cache array. As we are discarding this index, +* mark the index as having no cache entries, so it +* will not attempt to clean up the cache entries or +* validate them. +*/ istate->split_index->base->cache_nr =0 ; + }/* * We can discard the split index because its -- 2.20.1.682.gd5861c6d90
Re: [PATCH] unpack-trees.c: fix writing "link" index ext with null base oid
Quoted 8 lines · Elijah Newren
Quoted 8 lines
There is also another side fix in
remove_split_index()that causes a crash when doing "git update-index --no-split-index" whenbase_oidin the index file is null. In this case we will not loadistate->split_index->basebut we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.
Quoted 7 lines
Quoted 2 lines
Reported-by Luke Diamand <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>Thanks for digging this down and fixing it. When I saw this split index bug bisect to me that my heart sank a little; there's so much of that code I don't understand. Nice to see you've already come along and fixed it all up. :-)
Thanks, all.
Re: [PATCH] unpack-trees.c: fix writing "link" index ext with null base oid
In reply to Nguyễn Thái Ngọc DuyOn Sat, Feb 09 2019, Nguyễn Thái Ngọc Duy wrote:
I ran into this bug in the past but wasn't able to reproduce it long enough to track it down. Thanks for the fix.
It would be nice to have a test for it if it's easily narrowed down, tricky edge cases we fix are worth testing for...
Quoted 3 lines
I considered adding a test, but since the problem is a warning, not sure how to catch that. And a test would not be able to verify all changes in this patch anyway.
You'd write the test like this:
git [...] 2>stderr &&
test_i18ngrep "could not.*000000000" stderrOr something like that. We have a lot of tests like this already, some can be found with:
git grep -B1 test_i18ngrep.*stderr
So something being a warning shouldn't stop us from testing for it.
PATCH v2 1/1 read-cache.c: fix writing "link" index ext with null base oid
In reply to Nguyễn Thái Ngọc DuySince commit 7db118303a (unpack_trees: fix breakage when o->src_index != o->dst_index - 2018-04-23) and changes in merge code to use separate index_state for source and destination, when doing a merge with split index activated, we may run into this line in unpack_trees():
o->result.split_index = init_split_index(&o->result);
This is by itself not wrong. But this split index information is not fully populated (and it's only so when move_cache_to_base_index() is called, aka force splitting the index, or loading index_state from a file). Both "base_oid" and "base" in this case remain null.
So when writing the main index down, we link to this index with null oid (default value after init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh:
warning: could not freshen shared index '.../sharedindex.0{40}'
This patch makes sure we will not refresh null base_oid (because the file is never there). It also makes sure not to write "link" extension with null base_oid in the first place (no point having it at all). Read code already has protection against null base_oid.
There is also another side fix in remove_split_index() that causes a crash when doing "git update-index --no-split-index" when base_oid in the index file is null. In this case we will not load istate->split_index->base but we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.
---
v2 added a new test
read-cache.c | 5 +++-- split-index.c | 34 ++++++++++++++++++---------------- t/t1700-split-index.sh | 18 ++++++++++++++++++ 3 files changed, 39 insertions(+), 18 deletions(-)
read-cache.c +3 -2
@@ -2520,7 +2520,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,return err;/* Write extension data here */ -if (!strip_extensions && istate->split_index) { +if (!strip_extensions && istate->split_index && + !is_null_oid (&istate->split_index->base_oid)) {struct strbuf sb =STRBUF_INIT ; err = write_link_extension(&sb, istate) < 0 ||
@@ -2794,7 +2795,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock, ret =write_split_index (istate, lock, flags);/* Freshen the shared index only if the split-index was written */ -if (!ret && !new_shared_index) { +if (!ret && !new_shared_index && !is_null_oid (&si->base_oid)) {const char *shared_index =git_path ("sharedindex.%s" ,oid_to_hex (&si->base_oid));freshen_shared_index (shared_index,1 );
split-index.c +18 -16
@@ -440,24 +440,26 @@ void add_split_index(struct index_state *istate)void remove_split_index (struct index_state *istate) {if (istate->split_index) { -/* -* When removing the split index, we need to move -* ownership of the mem_pool associated with the -* base index to the main index. There may be cache entries -* allocated from the base's memory pool that are shared with -* the_index.cache[]. -*/ -mem_pool_combine (istate->ce_mem_pool, istate->split_index->base->ce_mem_pool); +if (istate->split_index->base) { +/* +* When removing the split index, we need to move +* ownership of the mem_pool associated with the +* base index to the main index. There may be cache entries +* allocated from the base's memory pool that are shared with +* the_index.cache[]. +*/ +mem_pool_combine (istate->ce_mem_pool, + istate->split_index->base->ce_mem_pool); -/* -* The split index no longer owns the mem_pool backing -* its cache array. As we are discarding this index, -* mark the index as having no cache entries, so it -* will not attempt to clean up the cache entries or -* validate them. -*/ -if (istate->split_index->base) +/* +* The split index no longer owns the mem_pool backing +* its cache array. As we are discarding this index, +* mark the index as having no cache entries, so it +* will not attempt to clean up the cache entries or +* validate them. +*/ istate->split_index->base->cache_nr =0 ; + }/* * We can discard the split index because its
t/t1700-split-index.sh +18 -0
@@ -447,4 +447,22 @@ test_expect_success 'writing split index with null sha1 does not write cache tre test_line_count = 0 cache-tree.out ' +test_expect_success 'do not refresh null base index' ' + test_create_repo merge && + ( + cd merge && + test_commit initial && + git checkout -b side-branch && + test_commit extra && + git checkout master && + git update-index --split-index && + test_commit more && + # must not write a new shareindex, or we wont catch the problem + git -c splitIndex.maxPercentChange=100 merge --no-edit side-branch 2>err && + # i.e. do not expect warnings like + # could not freshen shared index .../shareindex.00000... + test_must_be_empty err + ) +' + test_done
Signature · 1 line
2.21.0.rc0.328.g0e39304f8d
Re: [PATCH v2] read-cache.c: fix writing "link" index ext with null base oid
Quoted 31 lines · Nguyễn Thái Ngọc Duy
Since commit
7db118303a(unpack_trees: fix breakage wheno->src_index!=o->dst_index- 2018-04-23) and changes in merge code to use separateindex_statefor source and destination, when doing a merge with split index activated, we may run into this line inunpack_trees():o->result.split_index = init_split_index(&o->result);This is by itself not wrong. But this split index information is not fully populated (and it's only so when
move_cache_to_base_index()is called, aka force splitting the index, or loadingindex_statefrom a file). Both "base_oid" and "base" in this case remain null.So when writing the main index down, we link to this index with null oid (default value after
init_split_index()), which also means "no split index" internally. This triggers an incorrect base index refresh:warning: could not freshen shared index '.../sharedindex.0{40}'
This patch makes sure we will not refresh null
base_oid(because the file is never there). It also makes sure not to write "link" extension with nullbase_oidin the first place (no point having it at all). Read code already has protection against nullbase_oid.There is also another side fix in
remove_split_index()that causes a crash when doing "git update-index --no-split-index" whenbase_oidin the index file is null. In this case we will not loadistate->split_index->basebut we dereference it anyway and are rewarded with a segfault. This should not happen anymore, but it's still wrong to dereference a potential NULL pointer, especially when we do check for NULL pointer in the next code.
Quoted 4 lines
Reported-by Luke Diamand <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>---
v2 added a new test
This round looks good to me. Passes all tests with/without GIT_TEST_SPLIT_INDEX=true
Quoted 10 lines
read-cache.c | 5 +++-- split-index.c | 34 ++++++++++++++++++---------------- t/t1700-split-index.sh | 18 ++++++++++++++++++ 3 files changed, 39 insertions(+), 18 deletions(-)read-cache.c +0
-0
read-cache.c static int do_write_index(struct index_state *istate, struct tempfile *tempfile, · line 2527
Quoted 9 lines
read-cache.c +2
-1@@ -2520,7 +2520,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,return err;/* Write extension data here */ -if (!strip_extensions && istate->split_index) { +if (!strip_extensions && istate->split_index && + !is_null_oid (&istate->split_index->base_oid)) {struct strbuf sb =STRBUF_INIT ; err = write_link_extension(&sb, istate) < 0 ||
However, it looks like you based this on a pre-2.20.0 version of git. This conflicts with read-cache.c earlier than 3b1d9e045e ("eoie:
add End of Index Entry (EOIE) extension", 2018-10-10).
I fixed that manually, and pushed it out to github.com/avar/git.git duy-read-cache-null-split-index-fix, that's what I've tested on top of current "master".
Quoted 6 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'
Quoted 7 lines
There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
... meaning, if it hides the symptom, we'd know the codepath that causes a NULL si->base_oid to appear here is the culprit? Or do you mean that it is expected si->base_oid sometimes is NULL and we should have been pretending as if si were NULL (i.e. split_index is not being used)?
I take it as the latter (i.e. "helps" to narrow down the bug hunting field, not "helps" by fixing the bug).
Thanks.
Quoted 26 lines
-- 8< --
read-cache.c +1
-1@@ -3165,6 +3165,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,fill_fsmonitor_bitmap (istate);if (!si || alternate_index_output || + (si &&is_null_oid (&si->base_oid)) || (istate->cache_changed & ~EXTMASK )) {if (si)oidclr (&si->base_oid); -- 8< --Quoted 9 lines
(There's a repo called
dev_full, and I've got a worktree where I'm working on my 3rd attempt to make it work with gcc8).That file doesn't actually exist but there are a bunch of sharedindex.XXX files in there with more convincing looking names.
2.20.1.611.gfbb209baf1
--
Duy
Quoted 17 lines · Junio C Hamano
Quoted 6 lines · Duy Nguyen
Quoted 4 lines · Luke Diamand
I've recently started seeing a lot of this message when doing a rebase:
warning: could not freshen shared index
'/home/ldiamand/git/dev_full/.git/worktrees/gcc8-take-2/sharedindex.0000000000000000000000000000000000000000'Quoted 7 lines
There are only two places in the code that could print this. The one in
read_index_from()can't happen unlessis_null_oid()is broken (very very unlikely).The other one is in
write_locked_index()which could happen in theory but I don't understand how it got there. If you could build git, could you try this patch and see if it helps?
Quoted 5 lines
... meaning, if it hides the symptom, we'd know the codepath that causes a NULL
si->base_oidto appear here is the culprit? Or do you mean that it is expectedsi->base_oidsometimes is NULL and we should have been pretending as if si were NULL (i.e.split_indexis not being used)?
I didn't go that far when I suggested my patch. Looking more at the code, I think we may have written the "link" extension with null oid. Which could trigger this case and is definitely wrong.
Quoted 2 lines
I take it as the latter (i.e. "helps" to narrow down the bug hunting field, not "helps" by fixing the bug).
Yeah definitely not the fix (how can I write "I don't know why" in the commit message for the fix). At least now I know I'm on the right track.
Szeder recently fixed some racy bug in split-index.c, which seems fit in this rebase scenario since we'll be updating the index very often in a short period of time. Perhaps that uncovers some case that we don't handle well. I haven't been able to connect the dots though.
Signature · 1 line
Duy