Parallel fetch and commit graph writing results in locking failure (even on linux)
Hi,
on git version da72936f (Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching
$git fetch --all --jobs 12
Fordere an von origin
Fordere an von XXXX
Fordere an von YYYY
Fordere an von ZZZZ
Fordere an von EEEE
Von github.com:tango-controls/cppTango
37cc52f8..4550a743 tango-9-lts -> origin/tango-9-lts
Commit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
fatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.
Ein anderer Git-Prozess scheint in diesem Repository ausgeführt zu werden, zum Beispiel ein noch offener Editor von 'git commit'. Bitte stellen Sie sicher, dass alle Prozesse beendet wurden und versuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte ein früherer Git-Prozess in diesem Repository abgestürzt sein:
Löschen Sie die Datei manuell um fortzufahren.
Konnte 'myFork' nicht anfordern (Exit-Code: 128) Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
(Sorry for the german text, this is not easily reproducible.) It complains that it could not create the lock file as it already exists.
I've set the following possible relevant settings:
[core]
commitGraph = true
[fetch]
prune = true
writeCommitGraph = true
[protocol]
version = 2
Anything obvious I'm doing wrong?
Thanks,
Thomas
PS: The error is also present on latest git for windows.
Quoted 35 lines · Thomas Braun
Hi,
on git version
da72936f(Git 2.24, 2019-11-04) and debian stretch I currently get every now and then the following error during fetching$git fetch --all --jobs 12
Fordere an von origin
Fordere an von XXXX
Fordere an von YYYY
Fordere an von ZZZZ
Fordere an von EEEE
Von github.com:tango-controls/cppTango
37cc52f8..4550a743tango-9-lts -> origin/tango-9-lts
Commit-Graph Generierungsnummern berechnen: 100% (14/14), Fertig.
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
fatal: Konnte '/home/firma/devel/cppTango/.git/objects/info/commit-graphs/commit-graph-chain.lock' nicht erstellen: Die Datei existiert bereits.Ein anderer Git-Prozess scheint in diesem Repository ausgeführt zu werden, zum Beispiel ein noch offener Editor von 'git commit'. Bitte stellen Sie sicher, dass alle Prozesse beendet wurden und versuchen Sie es erneut. Falls es immer noch fehlschlägt, könnte ein früherer Git-Prozess in diesem Repository abgestürzt sein:
Löschen Sie die Datei manuell um fortzufahren.
Konnte 'myFork' nicht anfordern (Exit-Code: 128) Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig. Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.(Sorry for the german text, this is not easily reproducible.) It complains that it could not create the lock file as it already exists.
I've set the following possible relevant settings:
[core]
commitGraph = true
Quoted 8 lines
[fetch]
prune = true
writeCommitGraph = true[protocol]
version = 2Anything obvious I'm doing wrong?
I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).
I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to write_commit_graph_reachable() somewhere else inside builtin/fetch.c.
I'll take a look now.
Thanks,
-Stolee
Quoted 8 lines · Derrick Stolee
I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).
I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to
write_commit_graph_reachable()somewhere else insidebuiltin/fetch.c.
This should be fixed in master by bcb06e204c (Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.
-Peff
Quoted 6 lines · Jeff King
Quoted 4 lines · Derrick Stolee
I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).
Quoted 7 lines
Quoted 4 lines
I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to
write_commit_graph_reachable()somewhere else insidebuiltin/fetch.c.This should be fixed in master by
bcb06e204c(Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.
Thanks, Peff. That exactly looks like the right fix. The actual commit is 7d8e72b9 ("fetch: avoid locking issues between fetch.jobs/fetch.writeCommitGraph" 2019-11-03). I had forgotten that this was already fixed.
Here is the diff, for reference:
builtin/fetch.c +2 -1
@@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)return errcode; } -argv_array_pushl (&argv,"fetch" ,"--append" ,"--no-auto-gc" ,NULL ); +argv_array_pushl (&argv,"fetch" ,"--append" ,"--no-auto-gc" , +"--no-write-commit-graph" ,NULL );add_options_to_argv (&argv);if (max_children !=1 && list->nr !=1 ) {
-Stolee
Quoted 15 lines
Derrick Stolee <[email protected]> hat am 13. Dezember 2019 um 20:58 geschrieben:
Quoted 11 lines · Jeff King
Quoted 8 lines · Derrick Stolee
I don't think so. I think you just found a bug where the fetch.writeCommitGraph logic doesn't work with parallel fetch jobs (only one can write at a time).
I believe the fix would be to write the commit-graph after all of the jobs have completed, which should mean we need to move the call to
write_commit_graph_reachable()somewhere else insidebuiltin/fetch.c.
Quoted 7 lines
Quoted 2 lines
This should be fixed in master by
bcb06e204c(Merge branch 'js/fetch-multi-lockfix', 2019-12-01), I think.Thanks, Peff. That exactly looks like the right fix. The actual commit is
7d8e72b9("fetch: avoid locking issues between fetch.jobs/fetch.writeCommitGraph" 2019-11-03). I had forgotten that this was already fixed.
Thanks Stolee and Peff, I'll update to a version with that fix. And in the unlikely case this does not resolve my issue, report back.
Quoted 19 lines
Here is the diff, for reference:
builtin/fetch.c +2
-2@@ -1602,7 +1602,8 @@ static int fetch_multiple(struct string_list *list, int max_children)return errcode; } -argv_array_pushl (&argv,"fetch" ,"--append" ,"--no-auto-gc" ,NULL ); +argv_array_pushl (&argv,"fetch" ,"--append" ,"--no-auto-gc" , +"--no-write-commit-graph" ,NULL );add_options_to_argv (&argv);if (max_children !=1 && list->nr !=1 ) { -Stolee
Quoted 3 lines · Thomas Braun
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
(Sorry for the german text, this is not easily reproducible.)
It's unrelated to your issue, and I'm not a native German or English speaker... but that translation doesn't look quite right to me.
The original is:
_("Computing commit graph generation numbers"),
where the word "generation" isn't used in the sense of "creation" (erzeugen, generieren), but rather as in "generation gap" or "my parents are one generation older than me". So I don't think that "Generierung..." is the right word to use here, but perhaps "Generationsnummer".
But who am I to argue about German software ranslation?! :)
german language fix: Generierung vs. Generation [was: Re: Parallel fetch and commit graph writing results in locking failure (even on linux)]
Quoted 16 lines
SZEDER Gábor <[email protected]> hat am 13. Dezember 2019 um 21:15 geschrieben:
Quoted 3 lines · Thomas Braun
Commit-Graph Generierungsnummern berechnen: 100% (13/13), Fertig.
(Sorry for the german text, this is not easily reproducible.)
It's unrelated to your issue, and I'm not a native German or English speaker... but that translation doesn't look quite right to me.
The original is:
_("Computing commit graph generation numbers"),
Quoted 7 lines
where the word "generation" isn't used in the sense of "creation" (erzeugen, generieren), but rather as in "generation gap" or "my parents are one generation older than me". So I don't think that "Generierung..." is the right word to use here, but perhaps "Generationsnummer".
But who am I to argue about German software ranslation?! :)
Your hunch is right I would say (not that it matters much but I'm a native speaker).
So how about the following patch (CC'ing the original translator)
commit 724bf68da28b75e04a9fda34b08e95259ade9ec3 (HEAD -> master)
Author: Thomas Braun <[email protected]> Date: Sun Dec 15 18:28:42 2019 +0100
lang de: Reword generation numbers
The english term generation is here not used in the sense of "to generate" but in the sense of "generations of beings".
This corrects the initial translation from cf4c0c25 (l10n: update German translation, 2018-12-06).
po/de.po +1 -1
@@ -1535,7 +1535,7 @@ msgstr "Lösche Commit-Markierungen in Commit-Graph" #: commit-graph.c:1104 msgid "Computing commit graph generation numbers" -msgstr "Commit-Graph Generierungsnummern berechnen" +msgstr "Commit-Graph Generationsnummern berechnen" #: commit-graph.c:1179 #, c-format
PATCH 1/1 l10n: de.po: Reword generation numbers
From: Thomas Braun <[email protected]>
The english term generation is here not used in the sense of "to generate" but in the sense of "generations of beings".
This corrects the initial translation from cf4c0c25 (l10n: update German translation, 2018-12-06).
---
Thank you. The change looks good to me. I'll send a pull request with this patch to the German translation repo at Github.
po/de.po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
po/de.po +1 -1
@@ -1535,7 +1535,7 @@ msgstr "Lösche Commit-Markierungen in Commit-Graph" #: commit-graph.c:1104 msgid "Computing commit graph generation numbers" -msgstr "Commit-Graph Generierungsnummern berechnen" +msgstr "Commit-Graph Generationsnummern berechnen" #: commit-graph.c:1179 #, c-format
Signature · 1 line
2.24.1.735.g03f4e72817