Skip to contentSkip to conversation

Difficulties of scripting git

PATCH In review Alan Mackenzie 3 participants 10 messages Jan 9, 2021 – Jan 16, 2021 Documentation

lore
Share & tools
lore
#0 lore raw

Hello, git.

I want to write a bash script (more precisely, correct an existing script) which uses git, and it is DIFFICULT!!!

I need a git pull in the script. Fine, but how does the script know if it's worked or not? I couldn't find any description of a return code in the git-pull man page (or in the git-merge man page).

The big problem is when I have modified, uncommitted files in the target repo, and those same files are to be updated in commits coming from the source repo. Sadly, git is unable to merge these changes. It just fails, putting an error message onto stderr, but doesn't tell the calling script in any way that I can see.

One idea would be always to call git stash before doing the pull, then git stash pop afterwards. Trouble is, git stash is unreliable - it doesn't always add a new stash entry, so the stash pop at the end would sometimes/often pop off an entry it shouldn't. git stash doesn't have a --force argument. git stash doesn't set a result code, either, that I can see. One way around this would be to do

    $ git stash list | wc -l

both before and after the git stash and compare the answers, but really?

So, next idea, feed the output from git status --porcelain through grep before and after the git pull, so as to find out whether there are any modified files before the git pull (thus making a stash necessary) and any files with conflicts after the git stash pop. Shouldn't be too difficult.

Except, how does one recognise a file with conflicts from this git status output? The man page says that

" For paths with merge conflicts, X' and Y' show the modification
states of each side of the merge. For paths that do not have merge conflicts, X' shows the status of the index, and Y' shows the status of the work tree. For untracked paths, XY' are ??'. Other status codes can be interpreted as follows: ...."

I've spent nearly an hour trying to make sense of this bit of man page. How is one meant to distinguish an XY of a merge conflict from the XY of an index/work tree entry? I can't find that key bit of information anywhere.

What am I missing? Why does writing scripts using git have to be so hard?

Signature · 1 line
Alan Mackenzie (Nuremberg, Germany).
#1 lore raw
Quoted 22 lines · Alan Mackenzie

Hello, git.

I want to write a bash script (more precisely, correct an existing script) which uses git, and it is DIFFICULT!!!

I need a git pull in the script. Fine, but how does the script know if it's worked or not? I couldn't find any description of a return code in the git-pull man page (or in the git-merge man page).

The big problem is when I have modified, uncommitted files in the target repo, and those same files are to be updated in commits coming from the source repo. Sadly, git is unable to merge these changes. It just fails, putting an error message onto stderr, but doesn't tell the calling script in any way that I can see.

One idea would be always to call git stash before doing the pull, then git stash pop afterwards. Trouble is, git stash is unreliable - it doesn't always add a new stash entry, so the stash pop at the end would sometimes/often pop off an entry it shouldn't. git stash doesn't have a --force argument. git stash doesn't set a result code, either, that I can see. One way around this would be to do

Quoted 3 lines
    $ git stash list | wc -l

both before and after the git stash and compare the answers, but really?

git stash doesn't consider there being nothing to stash to be an error, so it doesn't set a nonzero error code. Think of it as, "I want a clean working tree," not, "I want to create a stash." In that sense, what you wanted is already done, so it's not an error.

You are, however, not the only person who's been unhappy with this behavior. A --force option may be a useful option to have if someone wanted to create it.

Quoted 5 lines

So, next idea, feed the output from git status --porcelain through grep before and after the git pull, so as to find out whether there are any modified files before the git pull (thus making a stash necessary) and any files with conflicts after the git stash pop. Shouldn't be too difficult.

Using "git status --porcelain" before and after the stash is an easy way to find out whether there was anything to stash. If there's a difference, then a stash was created.

Quoted 9 lines

Except, how does one recognise a file with conflicts from this git status output? The man page says that

" For paths with merge conflicts, X' and Y' show the modification
states of each side of the merge. For paths that do not have merge conflicts, X' shows the status of the index, and Y' shows the status of the work tree. For untracked paths, XY' are ??'. Other status codes can be interpreted as follows: ...."

Quoted 4 lines

I've spent nearly an hour trying to make sense of this bit of man page. How is one meant to distinguish an XY of a merge conflict from the XY of an index/work tree entry? I can't find that key bit of information anywhere.

In the chart, the options in the first section are "normal" cases that occur without a merge conflict. The second section is things that happen when there's a conflict (the "unmerged" states).

So what you're looking for to find conflicts is something maybe a little like this:

  git status --porcelain | grep -E '^(AA|DD|[AUD]U|U[AUD])'

If you believe that you're likely to need to handle names with newlines, then of course you'll need -z as well.

I do agree that this is a little unclear; I went ahead and simulated a merge conflict to verify. I'll try to send a patch making the documentation reflect what I mentioned about the chart above.

Signature · 2 lines
brian m. carlson (he/him or they/them)
Houston, Texas, US
#2 lore raw

PATCH 1/1 docs: add description of status output table

In reply to Alan Mackenzie

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

Let's improve this situation by describing what each section means:
non-conflicted, conflicted, or untracked files.

---
 Documentation/git-status.txt | 4 ++++
 1 file changed, 4 insertions(+)
Documentation/git-status.txt +4 -0
index 7731b45f07..63f13c201d 100644
@@ -201,6 +201,10 @@ codes can be interpreted as follows:
 Ignored files are not listed, unless `--ignored` option is in effect,
 in which case `XY` are `!!`.
 
+In the table below, the first section indicates normal non-conflicted states for
+tracked files, the second indicates files where a merge conflict has occurred
+but not yet been resolved, and the third indicates files not tracked by Git.
+
 ....
 X          Y     Meaning
 -------------------------------------------------
#3 lore raw

Re: [PATCH] docs: add description of status output table

Quoted 4 lines · brian m. carlson

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

I agree that the addition clarifies, but it is a bit sad that we already have a beginning of the explanation; I wonder if we should improve the existing description in addition, even if it may not be sufficient to eliminate the need for this new paragraph. Here is what we already have:

For paths with merge conflicts, X and Y show the modification states of each side of the merge. For paths that do not have merge conflicts, X shows the status of the index, and Y shows the status of the work tree. For untracked paths, XY are ??. Other status codes can be interpreted as follows:

This introductory text does sort-of hint that there are three classes (merged paths, unmerged paths and untracked paths), but (1) the order the three classes are described do not match that of the table, and (2) the explanation of the untracked paths predates the addition of ignored ones to the untracked class, so the description is added after the legends as if an afterthought.

I am actually tempted to suggest rewriting the whole section, starting from the paragraph above and ending at the table, with something like this:

Three different classes of paths are shown in the same format, but the meaning of XY are different:

* For merged paths, X shows the status of the index, and Y shows the status of the working tree.

* For unmerged paths, X and Y show the modification states of each side of the merge, relative to the common ancestor.

* For untracked paths, X and Y do not convey different meaning (as, by definition, they are not known to the index); ?? is shown for untracked paths, and when --ignored option is in effect, ignored paths are shown with !!.

In the following table, these three classes are shown in separate sections, and these characters are used for X and Y fields for the first two sections that show tracked paths:

* ' ' = unmodified
* 'M' = modified

...

....
X Y Meaning
------------------------------------------------------------
[AMD] not updated

...

Hmm?

#4 lore raw

Re: [PATCH] docs: add description of status output table

Quoted 26 lines · Junio C Hamano
Quoted 4 lines · brian m. carlson

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

I agree that the addition clarifies, but it is a bit sad that we already have a beginning of the explanation; I wonder if we should improve the existing description in addition, even if it may not be sufficient to eliminate the need for this new paragraph. Here is what we already have:

For paths with merge conflicts, X and Y show the modification states of each side of the merge. For paths that do not have merge conflicts, X shows the status of the index, and Y shows the status of the work tree. For untracked paths, XY are ??. Other status codes can be interpreted as follows:

This introductory text does sort-of hint that there are three classes (merged paths, unmerged paths and untracked paths), but (1) the order the three classes are described do not match that of the table, and (2) the explanation of the untracked paths predates the addition of ignored ones to the untracked class, so the description is added after the legends as if an afterthought.

Quoted 3 lines

I am actually tempted to suggest rewriting the whole section, starting from the paragraph above and ending at the table, with something like this:

Sure, I can reroll with that. I noticed that we're using a text diagram instead of a table, so maybe I can fix that up as well in v2, depending how the output looks.

Signature · 2 lines
brian m. carlson (he/him or they/them)
Houston, Texas, US
#5 lore raw

Re: [PATCH] docs: add description of status output table

In reply to Junio C Hamano

Hello, Junio and Brian.

Thanks for such rapid and helpful responses to my post.

Quoted 13 lines · Junio C Hamano
Quoted 4 lines · brian m. carlson

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

I agree that the addition clarifies, but it is a bit sad that we already have a beginning of the explanation; I wonder if we should improve the existing description in addition, even if it may not be sufficient to eliminate the need for this new paragraph. Here is what we already have:

Quoted 5 lines

For paths with merge conflicts, X and Y show the modification states of each side of the merge. For paths that do not have merge conflicts, X shows the status of the index, and Y shows the status of the work tree. For untracked paths, XY are ??. Other status codes can be interpreted as follows:

Another problem I tripped over was that word "Other", which is ambiguous. For quite some time I read it wrongly as meaning "Other than the codes already described in this paragraph" rather than its intended meaning "Other than for untracked paths". So I was searching through the man page looking for a description of codes for conflicts.

Quoted 6 lines

This introductory text does sort-of hint that there are three classes (merged paths, unmerged paths and untracked paths), but (1) the order the three classes are described do not match that of the table, and (2) the explanation of the untracked paths predates the addition of ignored ones to the untracked class, so the description is added after the legends as if an afterthought.

When you amend this documentation, would you please consider using the term "conflicts" instead of, or as well as "unmerged". "Conflicts" is something definite and clear - "unmerged" feels a bit vague and indefinite, to me at any rate. Why might a file be "unmerged" still?

Quoted 4 lines

I am actually tempted to suggest rewriting the whole section, starting from the paragraph above and ending at the table, with something like this:

Quoted 8 lines

Three different classes of paths are shown in the same format, but the meaning of XY are different:

* For merged paths, X shows the status of the index, and Y shows the status of the working tree.

* For unmerged paths, X and Y show the modification states of each side of the merge, relative to the common ancestor.

Also missing from the current doc, I think, is a description of which "side" of the difference is represented by X, and which by Y. In my use case (having conflicts after doing a git stash pop) those "sides" would be the work tree and the repository. In other scenarios (say a merge conflict after git pull) I think they would be something else (though my head is hurting a bit, here).

Quoted 14 lines

* For untracked paths, X and Y do not convey different meaning (as, by definition, they are not known to the index); ?? is shown for untracked paths, and when --ignored option is in effect, ignored paths are shown with !!.

In the following table, these three classes are shown in separate sections, and these characters are used for X and Y fields for the first two sections that show tracked paths:

* ' ' = unmodified
* 'M' = modified

...

Quoted 8 lines

....
X Y Meaning
------------------------------------------------------------
[AMD] not updated

...

Hmm?

I would appreciate such a new formulation. Thanks!

Signature · 1 line
Alan Mackenzie (Nuremberg, Germany).
#6 lore raw

Re: [PATCH] docs: add description of status output table

Quoted 14 lines · Alan Mackenzie
Quoted 4 lines · Junio C Hamano

I am actually tempted to suggest rewriting the whole section, starting from the paragraph above and ending at the table, with something like this:

Quoted 8 lines

Three different classes of paths are shown in the same format, but the meaning of XY are different:

* For merged paths, X shows the status of the index, and Y shows the status of the working tree.

* For unmerged paths, X and Y show the modification states of each side of the merge, relative to the common ancestor.

Quoted 6 lines

Also missing from the current doc, I think, is a description of which "side" of the difference is represented by X, and which by Y. In my use case (having conflicts after doing a git stash pop) those "sides" would be the work tree and the repository. In other scenarios (say a merge conflict after git pull) I think they would be something else (though my head is hurting a bit, here).

The two sides are the two heads that are being merged and don't include the working tree at all. I think the answer about which side is which in a merge depends on whether you're merging or rebasing. Rebases do things the opposite way of how merges do them.

In your case, you're interested in the fact that there _is_ an unresolved conflict, which means you'll either get DD, AA, or something with a U in it. That will happen regardless of how you do it, with git pull, git merge, git rebase -m, or git cherry-pick, and it indicates that the working tree is "broken" (i.e., has conflict markers) and the index is in a conflicted state. Since the state of the working tree and the index are known, we use this field for listing the two heads instead.

Signature · 2 lines
brian m. carlson (he/him or they/them)
Houston, Texas, US
#7 lore raw

PATCH v2 1/1 docs: rephrase and clarify the git status --short format

In reply to Alan Mackenzie

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

Let's improve this situation by rephrasing to be more explicit about what each of the sections in the table means, to tell users in plain language which cases are occurring, and to describe what "unmerged" means.

---
This uses text from Junio's email, so his sign-off will be required here. I assume that won't be a problem, but I can send a v3 if it is.

 Documentation/git-status.txt | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
Documentation/git-status.txt +20 -8
index 7731b45f07..c0764e850a 100644
@@ -184,11 +184,26 @@ characters, that field will be quoted in the manner of a C string
 literal: surrounded by ASCII double quote (34) characters, and with
 interior special characters backslash-escaped.
 
-For paths with merge conflicts, `X` and `Y` show the modification
-states of each side of the merge. For paths that do not have merge
-conflicts, `X` shows the status of the index, and `Y` shows the status
-of the work tree.  For untracked paths, `XY` are `??`.  Other status
-codes can be interpreted as follows:
+There are three different types of states that are shown using this format, and
+each one uses the `XY` syntax differently:
+
+* When a merge is occurring and the merge was successful, or outside of a merge
+	situation, `X` shows the status of the index and `Y` shows the status of the
+	working tree.
+* When a merge conflict has occurred and has not yet been resolved, `X` and `Y`
+	show the state introduced by each head of the merge, relative to the common
+	ancestor. These paths are said to be _unmerged_.
+* When a path is untracked, `X` and `Y` are always the same, since they are
+	unknown to the index. `??` is used for untracked paths. Ignored files are
+	not listed unless `--ignored` is used; if it is, ignored files are indicated
+	by `!!`.
+
+Note that the term _merge_ here also includes rebases using the default
+`--merge` strategy, cherry-picks, and anything else using the merge machinery.
+
+In the following table, these three classes are shown in separate sections, and
+these characters are used for `X` and `Y` fields for the first two sections that
+show tracked paths:
 
 * ' ' = unmodified
 * 'M' = modified
@@ -198,9 +213,6 @@ codes can be interpreted as follows:
 * 'C' = copied
 * 'U' = updated but unmerged
 
-Ignored files are not listed, unless `--ignored` option is in effect,
-in which case `XY` are `!!`.
-
 ....
 X          Y     Meaning
 -------------------------------------------------
#8 lore raw

Re: [PATCH v2] docs: rephrase and clarify the git status --short format

Quoted 10 lines · brian m. carlson

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

Let's improve this situation by rephrasing to be more explicit about what each of the sections in the table means, to tell users in plain language which cases are occurring, and to describe what "unmerged" means.

Quoted 4 lines

---
This uses text from Junio's email, so his sign-off will be required here. I assume that won't be a problem, but I can send a v3 if it is.

Actually I find the text so vastly improved from "here is my attempt" version in the discussion that, other than the paragraph structure, there is nothing left that I can call my contribution. I'll sign the resulting commit off anyway, though ;-)

Documentation/git-status.txt characters, that field will be quoted in the manner of a C string · line 202

Quoted 6 lines
Documentation/git-status.txt +5 -0
+There are three different types of states that are shown using this format, and
+each one uses the `XY` syntax differently:
...
+
+Note that the term _merge_ here also includes rebases using the default
+`--merge` strategy, cherry-picks, and anything else using the merge machinery.

Even if rebase uses the good-old "format-patch | am -3" pipeline, it would result in an index with entries at higher stages. So I am not sure if this "Note that" helps the reader.

Quoted 1 line

+In the following table, these three classes are shown in separate sections, and

This iteration has improved the "Three different classes of paths are shown" in the "here is my attempt" version to "Three different types of states ..."; shouldn't we be doing the same here with s/classes/types of states/?

Documentation/git-status.txt characters, that field will be quoted in the manner of a C string · line 206

Quoted 2 lines
Documentation/git-status.txt +2 -0
+these characters are used for `X` and `Y` fields for the first two sections that
+show tracked paths:

Thanks.

#9 lore raw

Re: [PATCH v2] docs: rephrase and clarify the git status --short format

Quoted 9 lines · Junio C Hamano
Quoted 2 lines · brian m. carlson

This uses text from Junio's email, so his sign-off will be required here. I assume that won't be a problem, but I can send a v3 if it is.

Actually I find the text so vastly improved from "here is my attempt" version in the discussion that, other than the paragraph structure, there is nothing left that I can call my contribution. I'll sign the resulting commit off anyway, though ;-)

Well, I did use some of your text verbatim, though, and the sign-off is necessary. Not that it's easily distinguishable in our history since you're the maintainer, but I also like to be meticulous about sign-offs because I think it makes sure that people get due credit for their contributions.

Quoted 10 lines

Documentation/git-status.txt characters, that field will be quoted in the manner of a C string · line 202

Quoted 6 lines
Documentation/git-status.txt +5 -0
+There are three different types of states that are shown using this format, and
+each one uses the `XY` syntax differently:
...
+
+Note that the term _merge_ here also includes rebases using the default
+`--merge` strategy, cherry-picks, and anything else using the merge machinery.

Even if rebase uses the good-old "format-patch | am -3" pipeline, it would result in an index with entries at higher stages. So I am not sure if this "Note that" helps the reader.

I'll rephrase.

Quoted 6 lines
Quoted 1 line

+In the following table, these three classes are shown in separate sections, and

This iteration has improved the "Three different classes of paths are shown" in the "here is my attempt" version to "Three different types of states ..."; shouldn't we be doing the same here with s/classes/types of states/?

Can do.

Signature · 2 lines
brian m. carlson (he/him or they/them)
Houston, Texas, US