multi-pack-index: fix --object-dir from outside repo
PATCH 1/1 multi-pack-index: fix --object-dir from outside repo
If using --object-dir to point into a repo, 'write' will segfault trying to access the object-dir via the repo it found, but that's not fully initialized. Fix it to use the object_dir properly.
--- midx.c | 10 +++++----- t/t5319-multi-pack-index.sh | 8 ++++++++ 2 files changed, 13 insertions(+), 5 deletions(-)
midx.c +5 -5
@@ -882,7 +882,7 @@ static void write_midx_reverse_index(char *midx_name, unsigned char *midx_hash,strbuf_release (&buf); } -static void clear_midx_files_ext (struct repository *r,const char *ext, +static void clear_midx_files_ext (const char *object_dir,const char *ext,unsigned char *keep_hash);static int midx_checksum_valid (struct multi_pack_index *m)
@@ -1086,7 +1086,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *if (flags &MIDX_WRITE_REV_INDEX )write_midx_reverse_index (midx_name, midx_hash, &ctx); -clear_midx_files_ext (the_repository,".rev" , midx_hash); +clear_midx_files_ext (object_dir,".rev" , midx_hash);commit_lock_file (&lk);
@@ -1135,7 +1135,7 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len,die_errno (_ ("failed to remove %s" ), full_path); } -static void clear_midx_files_ext (struct repository *r,const char *ext, +static void clear_midx_files_ext (const char *object_dir,const char *ext,unsigned char *keep_hash) {struct clear_midx_data data;
@@ -1146,7 +1146,7 @@ static void clear_midx_files_ext(struct repository *r, const char *ext, hash_to_hex(keep_hash), ext); data.ext = ext; -for_each_file_in_pack_dir (r->objects->odb->path, +for_each_file_in_pack_dir (object_dir, clear_midx_file_ext, &data);
@@ -1165,7 +1165,7 @@ void clear_midx_file(struct repository *r)if (remove_path (midx))die (_ ("failed to clear multi-pack-index at %s" ), midx); -clear_midx_files_ext (r,".rev" ,NULL ); +clear_midx_files_ext (r->objects->odb->path,".rev" ,NULL );free (midx); }
t/t5319-multi-pack-index.sh +8 -0
@@ -201,6 +201,14 @@ test_expect_success 'write midx with twelve packs' ' compare_results_with_midx "twelve packs" +test_expect_success 'multi-pack-index with --object-dir need not be in repo' ' + p="$(pwd)" && + rm -f $objdir/multi-pack-index && + cd / && + git multi-pack-index --object-dir="$p/$objdir" write && + cd "$p" +' + test_expect_success 'warn on improper hash version' ' git init --object-format=sha1 sha1 && (
Signature · 1 line
2.31.1
Quoted 4 lines · Johannes Berg
If using --object-dir to point into a repo, 'write' will segfault trying to access the object-dir via the repo it found, but that's not fully initialized. Fix it to use the
object_dirproperly.
Thanks for finding this! It's difficult to cover all of the cases, but I'm glad you found this and added a test.
t/t5319-multi-pack-index.sh test_expect_success 'write midx with twelve packs' ' · line 209
Quoted 6 lines
t/t5319-multi-pack-index.sh +6
-0+test_expect_success 'multi-pack-index with --object-dir need not be in repo' ' + p="$(pwd)" && + rm -f $objdir/multi-pack-index && + cd / && + git multi-pack-index --object-dir="$p/$objdir" write && + cd "$p"
Why are you using "cd /" here? Even if you mean to use "cd", please do so within a sub-shell.
Could you instead init a new repo within the current directory and point the object-dir to that location?
It could look something like this, (warning: I did not test this)
git init other && test_commit -C other first && git multi-pack-index --object-dir=other/.git/objects write
And is the only post-condition you are checking that we do not crash? Or is there a specific result you are looking for? For instance, we can double check that the MIDX was written:
test_path_is_file other/.git/objects/pack/multi-pack-index
but also you seem to be touching areas that delete files. Could we 'touch' some of those and then see them get deleted?
Thanks,
-Stolee
Hi Derrick,
Quoted 9 lines
t/t5319-multi-pack-index.sh test_expect_success 'write midx with twelve packs' ' · line 209
Quoted 6 lines
t/t5319-multi-pack-index.sh +6
-0+test_expect_success 'multi-pack-index with --object-dir need not be in repo' ' + p="$(pwd)" && + rm -f $objdir/multi-pack-index && + cd / && + git multi-pack-index --object-dir="$p/$objdir" write && + cd "$p"Why are you using "cd /" here?
I just needed to go outside the current test git directory, the tests are running in a way that the current working directory is already the git tree I'm operating in.
Quoted 2 lines
Even if you mean to use "cd",
please do so within a sub-shell.
I thought about it, but clearly all the tests are run in a sub-shell, so it didn't seem necessary? But happy to change, I don't really care either way.
Could you instead init a new repo within the current directory and point the object-dir to that location?
I guess I could, but all the other stuff in here is already making a new repo in the current working dir, and already initializing it with objects, etc.
Doing it all over again seemed like a waste of time?
It could look something like this, (warning: I did not test this)
git init other && test_commit -C other first && git multi-pack-index --object-dir=other/.git/objects write
Sure.
Actually, this won't work to test for the crash, I'd have to do something like
git init othertest_commit -C other first &&
(
mkdir non-git &&
cd non-git &&
git multi-pack-index --object-dir=../other/.git/objects write )
or so.
And is the only post-condition you are checking that we do not crash?
Yes, I was assuming that it'd actually work at that point - maybe not the best assumption, it could (erroneously) exit with a 0 exit status but have done nothing.
Quoted 1 line
Or is there a specific result you are looking for? For
instance, we can double check that the MIDX was written:
test_path_is_file other/.git/objects/pack/multi-pack-index
So I guess that would be a good idea.
but also you seem to be touching areas that delete files. Could we 'touch' some of those and then see them get deleted?
Ah, well, that's the underlying issue but I'm not sure we even ever get to that code? Then again, yes, the *.rev files should get removed, I'll see - not even sure I know how to get them to be generated in the first place, is that even supported already?
johannes
Quoted 10 lines · Johannes Berg
I just needed to go outside the current test git directory, the tests are running in a way that the current working directory is already the git tree I'm operating in.
Quoted 2 lines
Even if you mean to use "cd",
please do so within a sub-shell.I thought about it, but clearly all the tests are run in a sub-shell, so it didn't seem necessary? But happy to change, I don't really care either way.
Please learn to care before you write your next test, then ;-)
These tests are not run in a sub-shell; they are eval'ed, so that the assignment they make to variables can persist and affect the next test piece.
Thanks.
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Johannes Berg
I just needed to go outside the current test git directory, the tests are running in a way that the current working directory is already the git tree I'm operating in.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Even if you mean to use "cd",
please do so within a sub-shell.I thought about it, but clearly all the tests are run in a sub-shell, so it didn't seem necessary? But happy to change, I don't really care either way.
Please learn to care before you write your next test, then ;-)
Hey now, I'm fixing your segfaults ;-)
Quoted 3 lines
These tests are not run in a sub-shell; they are eval'ed, so that the assignment they make to variables can persist and affect the next test piece.
Makes sense. FWIW, the test *did* restore the CWD so things worked, and subshells are actually ugly (need to import test-lib-functions.sh again if you want to use those), but I'll make it work somehow.
More importantly, how do you feel about the "cd /"?
The tests are always run in a place where there's a parent git folder (even if it's git itself), so you cannot reproduce the segfault in a test without the "cd /", though I guess "cd /tmp" would also work or something, but "cd /" felt pretty safe, hopefully not many people have "/.git" on their system.
johannes
Quoted 16 lines · Johannes Berg
Quoted 6 lines · Junio C Hamano
Quoted 4 lines · Johannes Berg
I just needed to go outside the current test git directory, the tests are running in a way that the current working directory is already the git tree I'm operating in.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Even if you mean to use "cd",
please do so within a sub-shell.I thought about it, but clearly all the tests are run in a sub-shell, so it didn't seem necessary? But happy to change, I don't really care either way.
Please learn to care before you write your next test, then ;-)
Quoted 7 lines
Hey now, I'm fixing your segfaults ;-)
Quoted 3 lines
These tests are not run in a sub-shell; they are eval'ed, so that the assignment they make to variables can persist and affect the next test piece.
Makes sense. FWIW, the test *did* restore the CWD so things worked,
This assumes that your test completes to run the second "cd".
Quoted 3 lines
and
subshells are actually ugly (need to importtest-lib-functions.shagain if you want to use those), but I'll make it work somehow.
We just add subshells this way:
test_expect_success 'test name' '
prep_step && ( # now in a subshell cd wherever && do things # don't need to cd again ) && continue test
'
Quoted 7 lines
More importantly, how do you feel about the "cd /"?
The tests are always run in a place where there's a parent git folder (even if it's git itself), so you cannot reproduce the segfault in a test without the "cd /", though I guess "cd /tmp" would also work or something, but "cd /" felt pretty safe, hopefully not many people have "/.git" on their system.
Don't leave the directory your test is set up to run in.
Git has a very large test suite full of examples to use for inspiration. If you do not see a pattern used within the test suite, then there is probably good reason to avoid that pattern.
Thanks,
-Stolee
Quoted 12 lines · Derrick Stolee
We just add subshells this way:
test_expect_success'test name' 'prep_step && ( # now in a subshell cd wherever && do things # don't need to cd again ) && continue test
Quoted 1 line
'
Sure. I know how to do subshells :)
My point was that inside the subshell you cannot do test_path_is_file and similar, because the subshell didn't import the libs.
Quoted 9 lines
Quoted 7 lines
More importantly, how do you feel about the "cd /"?
The tests are always run in a place where there's a parent git folder (even if it's git itself), so you cannot reproduce the segfault in a test without the "cd /", though I guess "cd /tmp" would also work or something, but "cd /" felt pretty safe, hopefully not many people have "/.git" on their system.
Don't leave the directory your test is set up to run in.
I was specifically asking Junio ;-)
But realistically, if this is the requirement you want to impose, then you _cannot_ test for the segfault within git's test suite. Your loss.
johannes
Quoted 15 lines · Johannes Berg
Quoted 12 lines · Derrick Stolee
We just add subshells this way:
test_expect_success'test name' 'prep_step && ( # now in a subshell cd wherever && do things # don't need to cd again ) && continue testQuoted 1 line
'
Quoted 4 lines
Sure. I know how to do subshells :)
My point was that inside the subshell you cannot do
test_path_is_fileand similar, because the subshell didn't import the libs.
Everything the call to prep_step, and anything that came before that call, did to the environment, like setting shell functions and variables, is visible inside the ( ... subshell ... ).
Quoted 4 lines
I was specifically asking Junio ;-)
But realistically, if this is the requirement you want to impose, then you _cannot_ test for the segfault within git's test suite. Your loss.
With that "cannot", I think you are assuming too much.
Because Git is a fairly long-lived project, we've had our share of cases where we needed to test for bugs that happen only when outside a repository.
And we have facility for just that, it's called "nongit" test helper that comes from test-lib-functions.sh, which is dot-included already so your subshells get it for free.
t5300-pack-object.sh, for example, wants to make sure that the "git index-pack --stdin" command invoked in a directory that is not a repository, but "git index-pack <packfile>" works outside a repository, and has two tests that uses the nongit helper.
Thanks.
Quoted 3 lines · Johannes Berg
Makes sense. FWIW, the test *did* restore the CWD so things worked, and subshells are actually ugly (need to import
test-lib-functions.shagain if you want to use those), but I'll make it work somehow.
You do not need to dot-include test-lib-functions or anything ugly or special. The variables (not only the exported ones but regular shell variables) and shell functions that is visible immediately before you enter the opening "(" are all visible in the subshell.
The only notable difference you need to keep in mind when using subshell is that you cannot affect variables and environment in general of the calling shell. In this case, you are taking advantage of it---no matter where you chdir to, the main test procedure that spawned the subshell will not be affected even if your tests fail inside a subshell. But it also disallows you from doing certain things that rely on the ability to modify shell variables, like setting up test_when_finished clean-up routine.
Quoted 1 line
More importantly, how do you feel about the "cd /"?
Please don't. If somebody had a repository in /.git and the program you are testing is buggy, you'd risk destroying it. In general, it is not a good idea to step outside the test directory you are given, especially if you are *not* limiting yourself to read-only operation.
Quoted 3 lines
The tests are always run in a place where there's a parent git folder (even if it's git itself), so you cannot reproduce the segfault in a test without the "cd /"
There is a "nongit" test helper in the test suite. Would that work for your case?
Thanks.
Quoted 4 lines · Johannes Berg
If using --object-dir to point into a repo, 'write' will segfault trying to access the object-dir via the repo it found, but that's not fully initialized. Fix it to use the
object_dirproperly.
Thanks for CC'ing me; I have definitely been wondering about the intended behavior of --object-dir on the list recently [1].
I think your patch message could use some clarifying, though. Invoking
cd $REPO/..
git multi-pack-index write --object-dir=$REPO/.git/objectshas... different behavior depending on which side of the "write" argument you put `--object-dir". On the left-hand side (i.e., "--object-dir=... write", you get something like:
cd $REPO/..
git multi-pack-index --object-dir=$REPO/.git/objects write
zsh: segmentation fault git.compile multi-pack-index ...because the_repository->objects->odb isn't initialized (so reading path in clear_midx_files_ext crashes). But in the opposite order (i.e., "write --object-dir=...") you get:
BUG: environment.c:280: git environment hasn't been setup
zsh: abort git.compile multi-pack-index writebecause we catch that case much earlier in get_object_directory(). Why? Because cmd_multi_pack_index() fills in the value of object_dir with get_object_directory() if it isn't filled in already, but seeing "write" causes us to stop parsing and dispatch to the sub-command cmd_multi_pack_index_write().
I discussed this a little in [1] also (see the part about using RUN_SETUP instead). There are definitely different ways to handle that; you could equally imagine only dying if we were both outside of a Git repository and didn't point at one via --object-dir.
But that's separate from another issue which is fixed by your patch which is that we don't respect the value of --object-dir when cleaning up MIDX .rev files via clear_midx_files_ext().
Your fix there (to use the path of an object_dir instead of a repository struct) makes sense (since we don't ever fill in a repository struct corresponding to the --object-dir parameter from the MIDX code).
But I think that's a separate issue than the RUN_SETUP thing I mentioned earlier, so I would probably consider breaking this into two patches, the first which addresses the RUN_SETUP thing, and the second which is this fix.
midx.c static int write_midx_internal(const char *object_dir, struct multi_pack_index * · line 1089
Quoted 7 lines
static int
midx_checksum_valid(struct multi_pack_index *m)midx.c +1
-1@@ -1086,7 +1086,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *if (flags &MIDX_WRITE_REV_INDEX )write_midx_reverse_index (midx_name, midx_hash, &ctx); -clear_midx_files_ext (the_repository,".rev" , midx_hash); +clear_midx_files_ext (object_dir,".rev" , midx_hash);
We can rely on this value always being non-NULL, so this is good.
midx.c static void clear_midx_files_ext(struct repository *r, const char *ext, · line 1151
Quoted 13 lines
midx.c +1
-1-static void clear_midx_files_ext (struct repository *r,const char *ext, +static void clear_midx_files_ext (const char *object_dir,const char *ext,unsigned char *keep_hash) {midx.c +1
-1struct clear_midx_data data;@@ -1146,7 +1146,7 @@ static void clear_midx_files_ext(struct repository *r, const char *ext, hash_to_hex(keep_hash), ext); data.ext = ext; -for_each_file_in_pack_dir (r->objects->odb->path, +for_each_file_in_pack_dir (object_dir, clear_midx_file_ext, &data);
And here's the most important part of the change, which is obviously correct. But note to other reviewers that this has nothing to do with the RUN_SETUP issue I mentioned earlier, since for_each_file_in_pack_dir() doesn't care about that.
t/t5319-multi-pack-index.sh test_expect_success 'write midx with twelve packs' ' · line 211
Quoted 8 lines
t/t5319-multi-pack-index.sh +8
-0+test_expect_success 'multi-pack-index with --object-dir need not be in repo' ' + p="$(pwd)" && + rm -f $objdir/multi-pack-index && + cd / && + git multi-pack-index --object-dir="$p/$objdir" write && + cd "$p" +' +
I agree with Stolee that there should be a new repo created within the current working directory, that way you can "cd .." and be both outside of the repo you just created, but not outside of the test environment.
But let's make sure that we're not deleting any files that we should be leaving alone. So it might be good to do something like:
git init repo &&
test_when_finished "rm -fr repo" &&
(
cd repo &&
test_commit base &&
git repack -d &&
) &&
rev="$objdir/pack/multi-pack-index-$(midx_checksum $objdir).rev" &&
touch $rev &&
git multi-pack-index write --object-dir=repo/.git/objects &&
test_path_is_file repo/.git/objects/pack/multi-pack-index &&
test_path_is_file repo/.git/objects/multi-pack-index &&
test_path_is_file $objdir/pack/multi-pack-index &&
test_path_is_file $revThat isn't testing the "invoked from a non-repo, but --object-dir" is given case, but I think that's fine since they really are separate things.
Note also that midx_checksum doesn't exist, but it is merely a wrapper over a test-tool that prints out (for a multi_pack_index "m") m->data + m->data_len - the_hash_algo->rawsz.
So between splitting the patch, clarifying the patch message, and implementing support for this new test helper, this may be more of a project than you were bargaining for ;). Let me know if you want any help. I also don't mind taking care of it myself, since I promised in
[1] that I'd fix this issue anyway.
Thanks,
Taylor
[1]: https://lore.kernel.org/git/YQMFIljXl7sAAA%[email protected]/
Hi Taylor,
Quoted 2 lines
Thanks for CC'ing me; I have definitely been wondering about the intended behavior of
--object-diron the list recently [1].
Oh, hah. Not really being a contributor I'm not following the list, thanks for the pointer.
Quoted 7 lines
I think your patch message could use some clarifying, though. Invoking
cd $REPO/..
git multi-pack-index write --object-dir=$REPO/.git/objectshas... different behavior depending on which side of the "write" argument you put `--object-dir".
Wait what?! To be honest, I didn't expect it to even be valid on the right-hand side of "write".
Quoted 10 lines
On the left-hand side (i.e.,
"--object-dir=... write", you get something like:cd $REPO/..
git multi-pack-index --object-dir=$REPO/.git/objects writezsh: segmentation fault git.compile multi-pack-index ...
because
the_repository->objects->odbisn't initialized (so readingpathinclear_midx_files_extcrashes).
Right, this is what I was doing.
Quoted 11 lines
But in the opposite order
(i.e., "write --object-dir=...") you get:BUG: environment.c:280: git environment hasn't been setupzsh: abort git.compile multi-pack-index write
because we catch that case much earlier in
get_object_directory(). Why? Becausecmd_multi_pack_index()fills in the value ofobject_dirwithget_object_directory()if it isn't filled in already, but seeing "write" causes us to stop parsing and dispatch to the sub-commandcmd_multi_pack_index_write().
Great ...
But why do we even support both? What would the semantic difference be?
I'd be happy with either one of them working I guess :)
Quoted 8 lines
I discussed this a little in [1] also (see the part about using
RUN_SETUPinstead). There are definitely different ways to handle that; you could equally imagine only dying if we were both outside of a Git repository and didn't point at one via--object-dir.But that's separate from another issue which is fixed by your patch which is that we don't respect the value of
--object-dirwhen cleaning up MIDX .rev files viaclear_midx_files_ext().
Well, I _mostly_ meant to fix the crash, but yes, this is really the underlying issue, and indeed we should clean up the .rev files in this case as well.
Quoted 8 lines
Your fix there (to use the path of an
object_dirinstead of a repository struct) makes sense (since we don't ever fill in a repository struct corresponding to the--object-dirparameter from the MIDX code).But I think that's a separate issue than the
RUN_SETUPthing I mentioned earlier, so I would probably consider breaking this into two patches, the first which addresses theRUN_SETUPthing, and the second which is this fix.
I never wanted to fix the other issue though ;-)
And honestly, I think I don't understand the discussion at [1] well enough to really submit a patch for it.
Quoted 4 lines
Quoted 4 lines
t/t5319-multi-pack-index.sh +4
-0+test_expect_success 'multi-pack-index with --object-dir need not be in repo' ' + p="$(pwd)" && + rm -f $objdir/multi-pack-index && + cd / &&
Quoted 8 lines
t/t5319-multi-pack-index.sh test_expect_success 'write midx with twelve packs' ' · line 211
Quoted 4 lines
t/t5319-multi-pack-index.sh +4
-0+ git multi-pack-index --object-dir="$p/$objdir" write && + cd "$p" +' +I agree with Stolee that there should be a new repo created within the current working directory, that way you can "cd .." and be both outside of the repo you just created, but not outside of the test environment.
OK, fair enough, I'll resubmit.
Quoted 8 lines
But let's make sure that we're not deleting any files that we should be leaving alone. So it might be good to do something like:
git init repo &&
test_when_finished "rm -fr repo" &&
(
cd repo &&
Quoted 6 lines
test_commit base &&
git repack -d &&
) &&rev="$objdir/pack/multi-pack-index-$(midx_checksum $objdir).rev" && touch $rev &&
Hah, so you just manually pretend it was there - and meanwhile I was looking for a way to get git to generate one :)
Quoted 2 lines
git multi-pack-index write --object-dir=repo/.git/objects &&
Now this has the order of arguments the other way around, why?
Quoted 4 lines
test_path_is_file repo/.git/objects/pack/multi-pack-index &&
test_path_is_filerepo/.git/objects/multi-pack-index &&test_path_is_file$objdir/pack/multi-pack-index &&test_path_is_file$rev
Why would test_path_is_file? Seems like it should be !test_path_is_file?
Quoted 4 lines
That isn't testing the "invoked from a non-repo, but --object-dir" is given case, but I think that's fine since they really are separate things.
But that's the one thing I really want to work :)
Quoted 8 lines
Note also that
midx_checksumdoesn't exist, but it is merely a wrapper over a test-tool that prints out (for amulti_pack_index"m")m->data + m->data_len - the_hash_algo->rawsz.So between splitting the patch, clarifying the patch message, and implementing support for this new test helper, this may be more of a project than you were bargaining for ;).
Sounds like ;)
But actually we don't really care about the midx_checksum here, afaict? MIDX_WRITE_REV_INDEX isn't ever set, so the rev files are not created today?
Quoted 3 lines
Let me know if you want any
help. I also don't mind taking care of it myself, since I promised in
[1] that I'd fix this issue anyway.
Thanks :)
How about I resubmit this patch with some of the edits, especially with test for the case I care about (--object-dir used from a non-git place to point elsewhere) and then you can build on top of that?
Thanks,
johannes
Quoted 2 lines
[1]: https://lore.kernel.org/git/YQMFIljXl7sAAA%[email protected]/