archive: make --add-virtual-file honor --prefix

From: Tom Scogland <[email protected]>

The documentation for archive states:

The path of the file in the archive is built by concatenating the value of the last --prefix moption (if any) before this --add-virtual-file and <path>.

This matches the documentation for --add-file and the behavior works for that option, but --prefix is ignored for --add-virtual-file.

This commit modifies archive.c to include the prefix in the path and adds a check into the existing add-virtual-file test to ensure that it honors both the most recent prefix before the flag.

In looking for others with this issue, I found message [email protected] on the mailing list, where Stefan proposed a basically identical patch to archive.c back in February, so the main addition here is the test along with that patch.

---

    archive: make --add-virtual-file honor --prefix

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1719%2Ftrws%2Fhonor-prefix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1719/trws/honor-prefix-v1
Pull-Request: https://github.com/git/git/pull/1719

 archive.c              | 11 +++++------
 t/t5003-archive-zip.sh |  8 ++++++--
 2 files changed, 11 insertions(+), 8 deletions(-)

index 5287fcdd8e0..90086a1708f 100644
@@ -365,12 +365,11 @@ int write_archive_entries(struct archiver_args *args,
 
 		put_be64(fake_oid.hash, i + 1);
 
+		strbuf_reset(&path_in_archive);
+		if (info->base)
+			strbuf_addstr(&path_in_archive, info->base);
+		strbuf_addstr(&path_in_archive, basename(path));
 		if (!info->content) {
-			strbuf_reset(&path_in_archive);
-			if (info->base)
-				strbuf_addstr(&path_in_archive, info->base);
-			strbuf_addstr(&path_in_archive, basename(path));
-
 			strbuf_reset(&content);
 			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
 				err = error_errno(_("cannot read '%s'"), path);

@@ -381,7 +380,7 @@ int write_archive_entries(struct archiver_args *args,
 						  content.buf, content.len);
 		} else {
 			err = write_entry(args, &fake_oid,
-					  path, strlen(path),
+					  path_in_archive.buf, path_in_archive.len,
 					  canon_mode(info->stat.st_mode),
 					  info->content, info->stat.st_size);
 		}

index 961c6aac256..acc8bc4fcd6 100755
@@ -218,14 +218,18 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
 	fi &&
 	git archive --format=zip >with_file_with_content.zip \
 		--add-virtual-file=\""$PATHNAME"\": \
-		--add-virtual-file=hello:world $EMPTY_TREE &&
+		--add-virtual-file=hello:world \
+		--prefix=subdir/ --add-virtual-file=hello:world \
+		--prefix= $EMPTY_TREE &&
 	test_when_finished "rm -rf tmp-unpack" &&
 	mkdir tmp-unpack && (
 		cd tmp-unpack &&
 		"$GIT_UNZIP" ../with_file_with_content.zip &&
 		test_path_is_file hello &&
 		test_path_is_file "$PATHNAME" &&
-		test world = $(cat hello)
+		test world = $(cat hello) &&
+		test_path_is_file subdir/hello &&
+		test world = $(cat subdir/hello)
 	)
 '
 

base-commit: 786a3e4b8d754d2b14b1208b98eeb0a554ef19a8

gitgitgadget

This first paragraph was a bit hard to parse for me.

You will contrast the quoted paragraph with another one you did not quote later in this paragraph, so it is more helpful to readers if you instead said:

The documentatation for archive describes its
"--add-virtual-file" option like so:

... excerpt from --add-virtual-file description ...

This description is the same as "--add-file", and "--add-file" does behave the way as described. "--add-virtual-file" however ignores "--prefix".

Style: "This comit modifies" -> "Modify".

An obvious alternative fix is to update the documentation, which would be a much safer thing to do, given that there may be existing scripts written during the two years since --add-virtual-file option was introduced and has been behaving exactly this way. They will all be broken big time once the command starts honoring the "--prefix" option.

This pargraph should come _after_ the three-dash lines below.

The implementation looked obvious, assuming that it is a good idea to change it (I've already talked about a safer alternative fix).

	--prefix=subdir/ --add-virtual-file=good:night

Ok, I'll update the message with this and the below style.

I wouldn't mind doing this necessarily, but would want an option that follows the documentation in addition. The current implementation is harder to use consistently with --add-file and with --prefix, though it's possible to manually prefix each virtual file it's surprising that it produces (as far as I have found) the only files in the archive that don't fall under the prefix.

Certainly.

Thanks for the feedback, I'll get an updated patch posted later today.

From: Tom Scogland <[email protected]>

The documentation for archive describes the --add-virtual-file option thusly:

The path of the file in the archive is built by concatenating the value of the last --prefix moption (if any) before this --add-virtual-file and <path>.

The --add-file documentation is similar:

The path of the file in the archive is built by concatenating the value of the last --prefix option (if any) before this --add-file and the basename of <file>.

Notably both explicitly state that they honor the last --prefix option before the --add option in question. The implementation of --add-file seems to have always honored prefix, but the implementation of --add-virtual-file does not. Also note that --add-virtual-file explicitly states it will use the full path given, while --add-file uses the basename of the path it is given.

Modify archive.c to include the prefix in the path used by --add-virtual-file and add checks into
the existing add-virtual-file test to verify:

* that --prefix is honored
* that leading path components are preserved
* that both work together and separately

Changes since v1:
- Revised the commit message style
- Added tests for basename/non-basename behavior
- Fixed archive.c to use full path for virtual and basename for add-file

---

    archive: make --add-virtual-file honor --prefix

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1719%2Ftrws%2Fhonor-prefix-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1719/trws/honor-prefix-v2
Pull-Request: https://github.com/git/git/pull/1719

Range-diff vs v1:

1: 1b685d8ca1a ! 1: b9a0ef282a3 archive: make --add-virtual-file honor --prefix

     @@ Metadata
       ## Commit message ##
          archive: make --add-virtual-file honor --prefix
      
     -    The documentation for archive states:
     +    The documentation for archive describes the `--add-virtual-file` option
     +    thusly:
      
            The path of the file in the archive is built by concatenating the
            value of the last `--prefix` moption (if any) before this
            `--add-virtual-file` and <path>.
      
     -    This matches the documentation for --add-file and the behavior works for
     -    that option, but --prefix is ignored for --add-virtual-file.
     +    The `--add-file` documentation is similar:
      
     -    This commit modifies archive.c to include the prefix in the path and
     -    adds a check into the existing add-virtual-file test to ensure that it
     -    honors both the most recent prefix before the flag.
     +      The path of the file in the archive is built by concatenating the
     +      value of the last --prefix option (if any) before this --add-file and
     +      the basename of <file>.
     +
     +    Notably both explicitly state that they honor the last `--prefix` option
     +    before the `--add` option in question.  The implementation of
     +    `--add-file` seems to have always honored prefix, but the implementation
     +    of `--add-virtual-file` does not.  Also note that `--add-virtual-file`
     +    explicitly states it will use the full path given, while `--add-file`
     +    uses the basename of the path it is given.
     +
     +    Modify archive.c to include the prefix in the path used by
     +    `--add-virtual-file` and add checks into
     +    the existing add-virtual-file test to verify:
     +
     +    * that `--prefix` is honored
     +    * that leading path components are preserved
     +    * that both work together and separately
      
     -    In looking for others with this issue, I found message
     -    [email protected] on the mailing
     -    list, where Stefan proposed a basically identical patch to archive.c
     -    back in February, so the main addition here is the test along with that
     -    patch.
     +    Changes since v1:
     +    - Revised the commit message style
     +    - Added tests for basename/non-basename behavior
     +    - Fixed archive.c to use full path for virtual and basename for add-file
      
          Signed-off-by: Tom Scogland <[email protected]>
      
     @@ archive.c: int write_archive_entries(struct archiver_args *args,
      +		strbuf_reset(&path_in_archive);
      +		if (info->base)
      +			strbuf_addstr(&path_in_archive, info->base);
     -+		strbuf_addstr(&path_in_archive, basename(path));
       		if (!info->content) {
      -			strbuf_reset(&path_in_archive);
      -			if (info->base)
      -				strbuf_addstr(&path_in_archive, info->base);
     --			strbuf_addstr(&path_in_archive, basename(path));
     + 			strbuf_addstr(&path_in_archive, basename(path));
      -
       			strbuf_reset(&content);
       			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
       				err = error_errno(_("cannot read '%s'"), path);
      @@ archive.c: int write_archive_entries(struct archiver_args *args,
     + 						  canon_mode(info->stat.st_mode),
       						  content.buf, content.len);
       		} else {
     ++			strbuf_addstr(&path_in_archive, path);
       			err = write_entry(args, &fake_oid,
      -					  path, strlen(path),
      +					  path_in_archive.buf, path_in_archive.len,
     @@ t/t5003-archive-zip.sh: test_expect_success UNZIP 'git archive --format=zip --ad
       		--add-virtual-file=\""$PATHNAME"\": \
      -		--add-virtual-file=hello:world $EMPTY_TREE &&
      +		--add-virtual-file=hello:world \
     -+		--prefix=subdir/ --add-virtual-file=hello:world \
     ++		--add-virtual-file=with/dir/noprefix:withdirnopre \
     ++		--prefix=subdir/ --add-virtual-file=with/dirprefix:withdirprefix \
     ++		--prefix=subdir2/ --add-virtual-file=withoutdir:withoutdir \
      +		--prefix= $EMPTY_TREE &&
       	test_when_finished "rm -rf tmp-unpack" &&
       	mkdir tmp-unpack && (
     @@ t/t5003-archive-zip.sh: test_expect_success UNZIP 'git archive --format=zip --ad
       		test_path_is_file "$PATHNAME" &&
      -		test world = $(cat hello)
      +		test world = $(cat hello) &&
     -+		test_path_is_file subdir/hello &&
     -+		test world = $(cat subdir/hello)
     ++		test_path_is_file with/dir/noprefix &&
     ++		test withdirnopre = $(cat with/dir/noprefix) &&
     ++		test_path_is_file subdir/with/dirprefix &&
     ++		test withdirprefix = $(cat subdir/with/dirprefix) &&
     ++		test_path_is_file subdir2/withoutdir &&
     ++		test withoutdir = $(cat subdir2/withoutdir)
       	)
       '

 archive.c              | 10 +++++-----
 t/t5003-archive-zip.sh | 14 ++++++++++++--
 2 files changed, 17 insertions(+), 7 deletions(-)

index 5287fcdd8e0..64777a9870d 100644
@@ -365,12 +365,11 @@ int write_archive_entries(struct archiver_args *args,
 
 		put_be64(fake_oid.hash, i + 1);
 
+		strbuf_reset(&path_in_archive);
+		if (info->base)
+			strbuf_addstr(&path_in_archive, info->base);
 		if (!info->content) {
-			strbuf_reset(&path_in_archive);
-			if (info->base)
-				strbuf_addstr(&path_in_archive, info->base);
 			strbuf_addstr(&path_in_archive, basename(path));
-
 			strbuf_reset(&content);
 			if (strbuf_read_file(&content, path, info->stat.st_size) < 0)
 				err = error_errno(_("cannot read '%s'"), path);

@@ -380,8 +379,9 @@ int write_archive_entries(struct archiver_args *args,
 						  canon_mode(info->stat.st_mode),
 						  content.buf, content.len);
 		} else {
+			strbuf_addstr(&path_in_archive, path);
 			err = write_entry(args, &fake_oid,
-					  path, strlen(path),
+					  path_in_archive.buf, path_in_archive.len,
 					  canon_mode(info->stat.st_mode),
 					  info->content, info->stat.st_size);
 		}

index 961c6aac256..0cf3aef8ace 100755
@@ -218,14 +218,24 @@ test_expect_success UNZIP 'git archive --format=zip --add-virtual-file' '
 	fi &&
 	git archive --format=zip >with_file_with_content.zip \
 		--add-virtual-file=\""$PATHNAME"\": \
-		--add-virtual-file=hello:world $EMPTY_TREE &&
+		--add-virtual-file=hello:world \
+		--add-virtual-file=with/dir/noprefix:withdirnopre \
+		--prefix=subdir/ --add-virtual-file=with/dirprefix:withdirprefix \
+		--prefix=subdir2/ --add-virtual-file=withoutdir:withoutdir \
+		--prefix= $EMPTY_TREE &&
 	test_when_finished "rm -rf tmp-unpack" &&
 	mkdir tmp-unpack && (
 		cd tmp-unpack &&
 		"$GIT_UNZIP" ../with_file_with_content.zip &&
 		test_path_is_file hello &&
 		test_path_is_file "$PATHNAME" &&
-		test world = $(cat hello)
+		test world = $(cat hello) &&
+		test_path_is_file with/dir/noprefix &&
+		test withdirnopre = $(cat with/dir/noprefix) &&
+		test_path_is_file subdir/with/dirprefix &&
+		test withdirprefix = $(cat subdir/with/dirprefix) &&
+		test_path_is_file subdir2/withoutdir &&
+		test withoutdir = $(cat subdir2/withoutdir)
 	)
 '
 

base-commit: 786a3e4b8d754d2b14b1208b98eeb0a554ef19a8

gitgitgadget

The above is misleading.

The implementation of --add-file has always honored the prefix, while the implementation of --add-virtual-file has always ignored the prefix.

would make it easier to assess how long existing users may have been relying on the current behaviour.

Yes, this is a very good thing to mention. It is probably the reason why the implementation decided not to add prefix to the "full path" that already can have the leading directories.

Very nice job explaining the chosen design clearly (even though I do not necessarily agree with the direction this patch is going).

Also, given that this option was introduced for an explicit purpose of using it to write out diagnostics archive file, we should mention that this change does not break it in the proposed log message, at least. Of course, we should do so after verifying that is indeed the case, and better yet, after verifying that it will be hard for future changes to diagnose.c to trigger an unexpected behaviour caused by this change [*].

The "changes since v1" section does not belong to the log message proper, as v1 never happened as long as readers of "git log" are concerned. It is a very good thing to help reviewers to have below the three-dash lines that comes after your sign-off, though.

Fair, I had no intention to mislead and will reword.

Thanks for that. As to the direction, I mentioned earlier adding a different flag, or perhaps marking the filename in some fashion to express that the prefix should be honored, would you prefer that? It would, as you said, be much safer in that there's no reason for it to be a breaking change. If there's a design you prefer that would result in having an opt-in way to get the prefix behavior I wouldn't mind implementing it.

That's a very good point, and thank you for digging into it.

My apologies, this is my unfamiliarity with GitGitGadget, I'll put information like this in the PR description next time, which I think will do that.

Am 17.05.24 um 19:34 schrieb Tom Scogland via GitGitGadget:

The documentation does not actually misspell "option" as "moption".

Aligning code and docs is a good idea. Have you considered keeping the code as is and changing the documentation instead, though?

The two options are related in that they both add untracked files, but they necessarily have different arguments:

--add-file=<file>
--add-virtual-file=<path>:<content>

You can already specify any path you want with --add-virtual-file. What's the advantage of honoring --prefix as well?

René

I came into this after trying to translate an --add-file to an --add-virtual-file and being surprised that the prefix wasn't applied. anything else you add, in the repo or not, it gets the prefix. I can go back and explicitly add the prefix, but it makes the options less naturally composable in my opinion. The specific case was packaging some software that uses git metadata to set version and some other things when in a repo, and files in a tarball. The tarball's prefix is set in one part of the packaging code, files and exclusions in another, and passing that through was something we didn't have to think about using add-file, but do with add-virtual-file.

That said, it sounds like both you and Junio prefer updating the docs rather than the code, which makes me think I'm in the minority in that opinion. If that's the case, I can certainly update the docs, and I imagine we can backport that easily wherever it makes sense. I would really like to have the option to have the prefix apply though, either adding a new flag or an option to the existing one that would be invalid given current syntax or similar to provide the option.

Am 20.05.24 um 18:10 schrieb Tom Scogland:

Understandable, the documentation promised otherwise and the options have very similar names.

True, applying the prefix to all items would be simpler overall.

Speaking of simpler: --add-virtual-file could have been implemented to only take a single argument -- the content -- and rely on --prefix to provide the full path. That's more consistent with other options, as most of them only take single-valued arguments (or none). :]

I'm not sure I have an opinion on that topic, yet. Fixing the documentation is certainly easier. Adding the prefix to the path of virtual files as well is a breaking change. I feel that the easier route should at least be mentioned in the commit message and why it was not taken.

You mean like replacing a leading colon in the path with the prefix?

René

It has been a few weeks since this discussion stalled. Let me make an executive decision on the direction here---let's keep the behaviour and align the documentation so that we won't break existing users.

Thanks.

So here is to re-ignite the discussion.

----- >8 -----
Subject: [PATCH] archive: document that --add-virtual-file takes full path

Tom Scogland noticed that --add-virtual-file option uses the path specified as its value as-is, without prepending any value given to the --prefix option like --add-file does.

The behaviour has always been that way since the option was introduced, but the documentation has always been wrong and said that it would use the value of --prefix just like --add-file does.

We could modify the behaviour to make it literally work like the documentation said, but it would break existing scripts the users use.

---
 Documentation/git-archive.txt | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

index 98526f2beb..a0e3fe7996 100644
@@ -53,7 +53,7 @@ OPTIONS
 --prefix=<prefix>/::
 	Prepend <prefix>/ to paths in the archive.  Can be repeated; its
 	rightmost value is used for all tracked files.  See below which
-	value gets used by `--add-file` and `--add-virtual-file`.
+	value gets used by `--add-file`.
 
 -o <file>::
 --output=<file>::

@@ -67,9 +67,7 @@ OPTIONS
 
 --add-virtual-file=<path>:<content>::
 	Add the specified contents to the archive.  Can be repeated to add
-	multiple files.  The path of the file in the archive is built
-	by concatenating the value of the last `--prefix` option (if any)
-	before this `--add-virtual-file` and `<path>`.
+	multiple files.
 +
 The `<path>` argument can start and end with a literal double-quote
 character; the contained file name is interpreted as a C-style string,

@@ -81,6 +79,10 @@ if the path begins or ends with a double-quote character.
 The file mode is limited to a regular file, and the option may be
 subject to platform-dependent command-line limits. For non-trivial
 cases, write an untracked file and use `--add-file` instead.
++
+Note that unlike `--add-file` the path created in the archive is not
+affected by the `--prefix` option, as a full `<path>` can be given as
+the value of the option.
 
 --worktree-attributes::
 	Look for attributes in .gitattributes files in the working tree

2.45.2-683-g09b5b61c39

And it seems that nobody is interested in the topic all that much anymore?

I've read the updated text once again, and didn't see anything glaringly wrong, so I'll mark it for 'next'.

Thanks.

Am 26.06.24 um 00:43 schrieb Junio C Hamano:

I'm low-key pleased to see a documentation bug being stomped.

Perhaps this saps the energy out of efforts to make --add-virtual-file respect --prefix, but we'd probably need a new option name for that anyway to keep backward compatibility (--add-prefix-file?), so the patch adds no technical obstacle for that at least.

The patch looks good to me. I like the new paragraph about --prefix not being respected, in particular because I wouldn't thought of adding it, but it does look helpful.

Perhaps add a Reported-by: line for Tom that's findable with trailer tooling, though?

René

[jc: wrapped overly long lines]

Wouldn't

    #!/bin/sh
    prefix=foo/bar
    git archive --prefix="$prefix" --add-file=x --add-file=y \
	--add-virtual-file="$prefix/path:contents"

be an option enough? You only have to define $prefix once.