Skip to contentSkip to conversation

Parallel fetch and commit graph writing results in locking failure (even on linux)

PATCH New Thomas Braun 5 participants 8 messages Dec 13, 2019 – Dec 15, 2019

lore
Share & tools
lore
#0 lore raw

Hi,

on git version da72936f (Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching

$git fetch --all --jobs 12
Fordere an von origin
Fordere an von XXXX
Fordere an von YYYY
Fordere an von ZZZZ
Fordere an von EEEE
Von github.com:tango-controls/cppTango
37cc52f8..4550a743 tango-9-lts -> origin/tango-9-lts
Commit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
fatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.

Ein anderer Git-Prozess scheint in diesem Repository ausgeführt zu werden, zum Beispiel ein noch offener Editor von 'git commit'. Bitte stellen Sie sicher, dass alle Prozesse beendet wurden und versuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte ein früherer Git-Prozess in diesem Repository abgestürzt sein:
Löschen Sie die Datei manuell um fortzufahren.
Konnte 'myFork' nicht anfordern (Exit-Code: 128) Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.

(Sorry for the german text, this is not easily reproducible.) It complains that it could not create the lock file as it already exists.

I've set the following possible relevant settings:

[core]
commitGraph = true

[fetch]
prune = true
writeCommitGraph = true

[protocol]
version = 2

Anything obvious I'm doing wrong?

Thanks,
Thomas

PS: The error is also present on latest git for windows.

#1 lore raw
Quoted 35 lines · Thomas Braun

Hi,

on git version da72936f (Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching

$git fetch --all --jobs 12
Fordere an von origin
Fordere an von XXXX
Fordere an von YYYY
Fordere an von ZZZZ
Fordere an von EEEE
Von github.com:tango-controls/cppTango
37cc52f8..4550a743 tango-9-lts -> origin/tango-9-lts
Commit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
fatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.

Ein anderer Git-Prozess scheint in diesem Repository ausgeführt zu werden, zum Beispiel ein noch offener Editor von 'git commit'. Bitte stellen Sie sicher, dass alle Prozesse beendet wurden und versuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte ein früherer Git-Prozess in diesem Repository abgestürzt sein:
Löschen Sie die Datei manuell um fortzufahren.
Konnte 'myFork' nicht anfordern (Exit-Code: 128) Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.

(Sorry for the german text, this is not easily reproducible.) It complains that it could not create the lock file as it already exists.

I've set the following possible relevant settings:

[core]
commitGraph = true

Quoted 8 lines

[fetch]
prune = true
writeCommitGraph = true

[protocol]
version = 2

Anything obvious I'm doing wrong?

I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).

I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to write_commit_graph_reachable() somewhere else inside builtin/fetch.c.

I'll take a look now.

Thanks,
-Stolee

#2 lore raw
Quoted 8 lines · Derrick Stolee

I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).

I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to write_commit_graph_reachable() somewhere else inside builtin/fetch.c.

This should be fixed in master by bcb06e204c (Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.

-Peff

#3 lore raw
Quoted 6 lines · Jeff King
Quoted 4 lines · Derrick Stolee

I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).

Quoted 7 lines
Quoted 4 lines

I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to write_commit_graph_reachable() somewhere else inside builtin/fetch.c.

This should be fixed in master by bcb06e204c (Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.

Thanks, Peff. That exactly looks like the right fix. The actual commit is 7d8e72b9 ("fetch: avoid locking issues between fetch.jobs/fetch.writeCommitGraph" 2019-11-03). I had forgotten that this was already fixed.

Here is the diff, for reference:

builtin/fetch.c +2 -1
index 8d27f8abb7..20bcda09c4 100644
@@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)
                        return errcode;
        }
 
-       argv_array_pushl(&argv, "fetch", "--append", "--no-auto-gc", NULL);
+       argv_array_pushl(&argv, "fetch", "--append", "--no-auto-gc",
+                       "--no-write-commit-graph", NULL);
        add_options_to_argv(&argv);
 
        if (max_children != 1 && list->nr != 1) {

-Stolee

#4 lore raw
Quoted 15 lines

Derrick Stolee <[email protected]> hat am 13. Dezember 2019 um 20:58 geschrieben:

Quoted 11 lines · Jeff King
Quoted 8 lines · Derrick Stolee

I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).

I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to write_commit_graph_reachable() somewhere else inside builtin/fetch.c.

Quoted 7 lines
Quoted 2 lines

This should be fixed in master by bcb06e204c (Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.

Thanks, Peff. That exactly looks like the right fix. The actual commit is 7d8e72b9 ("fetch: avoid locking issues between fetch.jobs/fetch.writeCommitGraph" 2019-11-03). I had forgotten that this was already fixed.

Thanks Stolee and Peff, I'll update to a version with that fix. And in the unlikely case this does not resolve my issue, report back.

Quoted 19 lines

Here is the diff, for reference:

builtin/fetch.c +2 -2
index 8d27f8abb7..20bcda09c4 100644
@@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)
                        return errcode;
        }
 
-       argv_array_pushl(&argv, "fetch", "--append", "--no-auto-gc", NULL);
+       argv_array_pushl(&argv, "fetch", "--append", "--no-auto-gc",
+                       "--no-write-commit-graph", NULL);
        add_options_to_argv(&argv);
 
        if (max_children != 1 && list->nr != 1) {
 
-Stolee
#5 lore raw
In reply to Thomas Braun
Quoted 3 lines · Thomas Braun

Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.

(Sorry for the german text, this is not easily reproducible.)

It's unrelated to your issue, and I'm not a native German or English speaker... but that translation doesn't look quite right to me.

The original is:

_("Computing commit graph generation numbers"),

where the word "generation" isn't used in the sense of "creation" (erzeugen, generieren), but rather as in "generation gap" or "my parents are one generation older than me". So I don't think that "Generierung..." is the right word to use here, but perhaps "Generationsnummer".

But who am I to argue about German software ranslation?! :)

#6 lore raw

german language fix: Generierung vs. Generation [was: Re: Parallel fetch and commit graph writing results in locking failure (even on linux)]

Quoted 16 lines

SZEDER Gábor <[email protected]> hat am 13. Dezember 2019 um 21:15 geschrieben:

Quoted 3 lines · Thomas Braun

Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.

(Sorry for the german text, this is not easily reproducible.)

It's unrelated to your issue, and I'm not a native German or English speaker... but that translation doesn't look quite right to me.

The original is:

_("Computing commit graph generation numbers"),

Quoted 7 lines

where the word "generation" isn't used in the sense of "creation" (erzeugen, generieren), but rather as in "generation gap" or "my parents are one generation older than me". So I don't think that "Generierung..." is the right word to use here, but perhaps "Generationsnummer".

But who am I to argue about German software ranslation?! :)

Your hunch is right I would say (not that it matters much but I'm a native speaker).

So how about the following patch (CC'ing the original translator)

commit 724bf68da28b75e04a9fda34b08e95259ade9ec3 (HEAD -> master)

Author: Thomas Braun <[email protected]>
Date:   Sun Dec 15 18:28:42 2019 +0100

lang de: Reword generation numbers

The english term generation is here not used in the sense of "to generate" but in the sense of "generations of beings".

This corrects the initial translation from cf4c0c25 (l10n: update German translation, 2018-12-06).

po/de.po +1 -1
index 066326a687..773e361f6f 100644
@@ -1535,7 +1535,7 @@ msgstr "Lösche Commit-Markierungen in Commit-Graph"
 
 #: commit-graph.c:1104
 msgid "Computing commit graph generation numbers"
-msgstr "Commit-Graph Generierungsnummern berechnen"
+msgstr "Commit-Graph Generationsnummern berechnen"
 
 #: commit-graph.c:1179
 #, c-format
#7 lore raw

PATCH 1/1 l10n: de.po: Reword generation numbers

From: Thomas Braun <[email protected]>

The english term generation is here not used in the sense of "to generate" but in the sense of "generations of beings".

This corrects the initial translation from cf4c0c25 (l10n: update German translation, 2018-12-06).

---
Thank you. The change looks good to me. I'll send a pull request with this patch to the German translation repo at Github.

 po/de.po | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
po/de.po +1 -1
index 066326a687..773e361f6f 100644
@@ -1535,7 +1535,7 @@ msgstr "Lösche Commit-Markierungen in Commit-Graph"
 
 #: commit-graph.c:1104
 msgid "Computing commit graph generation numbers"
-msgstr "Commit-Graph Generierungsnummern berechnen"
+msgstr "Commit-Graph Generationsnummern berechnen"
 
 #: commit-graph.c:1179
 #, c-format
Signature · 1 line
2.24.1.735.g03f4e72817