fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
RFC PATCH 1/1 fetch2/git: Prevent git fetcher from fetching gitlab repository metadata
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>
--- lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/fetch2/git.py +1 -1
@@ -382,7 +382,7 @@ class Git(FetchMethod): runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir) runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir) - fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl)) + fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl)) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d)
Signature · 1 line
2.35.1
Quoted 46 lines
-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadataThe bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>--- lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/bb/fetch2/git.py +1
-2@@ -382,7 +382,7 @@ class Git(FetchMethod): runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir) runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir) - fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl)) + fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl)) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d) --
Quoted 1 line
2.35.1
Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....
//Peter
Hi,
Quoted 39 lines · Peter Kjellerstedt
Quoted 39 lines
-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadataThe bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>--- lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)lib/bb/fetch2/git.py +0
-0@@ -382,7 +382,7 @@ class Git(FetchMethod): runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
Quoted 14 lines
Quoted 8 lines
lib/bb/fetch2/git.py +1
-2runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir) - fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl)) + fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl)) if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d) --2.35.1
Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....
As user of Gerrit, I fear this will cause problems. In my case developers are used to creating test topics and using git hashes in recipes which are not yet released, e.g. not yet in release branches or tags. This can of course create problems when such changes end up in real releases.
Workaround is that developers can create throw away testing branches and refer to them in recipes.
From one side this is an improvement to have less data in caches, but on the other side this adds extra actions to developers who want to test changes to their recipes. Can't decide which one is more important though :/
Cheers,
-Mikko
Can be solved with a parameter to a fetcher perhaps?
Alex
Quoted 94 lines · Mikko Rapeli
Hi,
Quoted 5 lines · Peter Kjellerstedt
Quoted 5 lines
-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>;Peter Kjellerstedt <[email protected]>; Richard Purdie < [email protected]>
Quoted 1 line
Quoted 1 line
Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching
gitlab repository metadata
Quoted 6 lines
Quoted 6 lines
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Quoted 8 lines
Quoted 8 lines
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher
such
Quoted 1 line
Quoted 1 line
that it only fetches refs/heads/* and refs/tags/* . Avoid using
negative
Quoted 11 lines
Quoted 11 lines
refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>--- lib/bb/fetch2/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 6 lines
lib/bb/fetch2/git.py class Git(FetchMethod): · line 382
Quoted 6 lines
lib/bb/fetch2/git.py +0
-0@@ -382,7 +382,7 @@ class Git(FetchMethod): runfetchcmd("%s remote rm origin" % ud.basecmd, d,workdir=ud.clonedir)
Quoted 2 lines
lib/bb/fetch2/git.py class Git(FetchMethod): · line 383
Quoted 2 lines
lib/bb/fetch2/git.py +0
-0runfetchcmd("%s remote add --mirror=fetch origin %s" %(ud.basecmd,
shlex.quote(repourl)), d, workdir=ud.clonedir)Quoted 1 line
lib/bb/fetch2/git.py class Git(FetchMethod): · line 384
Quoted 1 line
lib/bb/fetch2/git.py +0
-1- fetch_cmd = "LANG=C %s fetch -f --progress %srefs/*:refs/*" % (ud.basecmd,
shlex.quote(repourl))Quoted 1 line
lib/bb/fetch2/git.py class Git(FetchMethod): · line 384
Quoted 1 line
lib/bb/fetch2/git.py +1
-0+ fetch_cmd = "LANG=C %s fetch -f --progress %srefs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd,
shlex.quote(repourl))Quoted 11 lines
Quoted 5 lines
lib/bb/fetch2/git.py +0
-1if ud.proto.lower() != 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler = GitProgressHandler(d) --2.35.1
Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....
As user of Gerrit, I fear this will cause problems. In my case developers are used to creating test topics and using git hashes in recipes which are not yet released, e.g. not yet in release branches or tags. This can of course create problems when such changes end up in real releases.
Workaround is that developers can create throw away testing branches and refer to them in recipes.
From one side this is an improvement to have less data in caches, but on the other side this adds extra actions to developers who want to test changes to their recipes. Can't decide which one is more important though :/
Cheers,
-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#13910):
https://lists.openembedded.org/g/bitbake-devel/message/13910
Mute This Topic: https://lists.openembedded.org/mt/93128921/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 1 line · Alexander Kanavin
Can be solved with a parameter to a fetcher perhaps?
Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.
This could be used for fetching the changes too, to limit the scope.
Cheers,
-Mikko
Quoted 7 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin
Can be solved with a parameter to a fetcher perhaps?
Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.
This could be used for fetching the changes too, to limit the scope.
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
Quoted 10 lines · Marek Vasut
Quoted 7 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin
Can be solved with a parameter to a fetcher perhaps?
Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.
This could be used for fetching the changes too, to limit the scope.
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
Quoted 5 lines · Marek Vasut
Quoted 4 lines · Marek Vasut
Quoted 3 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin
Can be solved with a parameter to a fetcher perhaps?
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.
This could be used for fetching the changes too, to limit the scope.
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
But then, if gerrit works with nobranch=1, then gerrit must be generating tags which contain the commits you test ?
And since this patch fetches refs/tags/ , then this patch won't break the gerrit setup ?
Hi,
Quoted 6 lines · Marek Vasut
Quoted 5 lines · Marek Vasut
Quoted 4 lines · Marek Vasut
Quoted 3 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin
Can be solved with a parameter to a fetcher perhaps?
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.
This could be used for fetching the changes too, to limit the scope.
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
To me this would be the way to go.
Quoted 7 lines
Quoted 1 line
No, this won't do, nobranch expects the commit to be in a tag.
But then, if gerrit works with nobranch=1, then gerrit must be generating tags which contain the commits you test ?
And since this patch fetches refs/tags/ , then this patch won't break the gerrit setup ?
nobranch=1 works with any branch or tag or open gerrit review commit id. At least with sumo and dunfell.
Cheers,
-Mikko
Quoted 4 lines · Marek Vasut
Quoted 2 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Alex
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Documentation says it does:
https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n45
"
- nobranch
Don't check the SHA validation for branch. set this option for the
recipe
referring to commit which is valid in tag instead of branch. The default is "0", set nobranch=1 if needed.
"
Quoted 12 lines · Marek Vasut
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Documentation says it does:
Quoted 7 lines
https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n45
"
- nobranch
Don't check the SHA validation for branch. set this option for the recipe referring to commit which is valid in tag instead of branch. The default is "0", set nobranch=1 if needed.
"
Only the first sentence is enforced. The change can still be in a branch, in a tag, in random other namespace as long as the commit is found at checkout time.
Cheers,
-Mikko
Hi Marek,
Quoted 10 lines · Marek Vasut
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Quoted 7 lines
Documentation says it does:
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
Don't check the SHA validation for branch. set this option for the
recipe
referring to commit which is valid in tag instead of branch.
I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).
The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.
Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?
(We need to update the docs to reflect that in that case).
Cheers,
Quentin
Quoted 12 lines
The default is "0", set nobranch=1 if needed.
"-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13918): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_message_13918&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=g4KWyxwbq71V3gbvIJNG-oA9Gdvj3A5wqfz8Kws5qZg&e=
Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93128921_6293953&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=KMelWJJL5NtG7NWmtiS3jFAONb4GRttyl1ziLzEHhr8&e=
Group Owner: [email protected]
Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_unsub&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=nyYO56WzR2jmLa4g95pgXToervc-fJhqbVjnOOUDm0g&e= [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 20 lines · Quentin Schulz
Hi Marek,
Quoted 8 lines · Marek Vasut
Quoted 7 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?
No, this won't do, nobranch expects the commit to be in a tag.
Quoted 8 lines
Quoted 1 line
I don't think it expects that.
Documentation says it does:
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
Don't check the SHA validation for branch. set this option for the recipe
referring to commit which is valid in tag instead of branch.
Quoted 7 lines
I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).
The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.
Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?
(We need to update the docs to reflect that in that case).
In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?
Quoted 31 lines · Marek Vasut
Quoted 30 lines · Quentin Schulz
Hi Marek,
Quoted 11 lines · Marek Vasut
Quoted 9 lines · Alexander Kanavin
Quoted 5 lines · Marek Vasut
Quoted 3 lines
So maybe the easy way out is, if nobranch=1 then fetch everything, else
just heads and tags ?No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Quoted 7 lines
Documentation says it does:
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
Don't check the SHA validation for branch. set this option for the
recipe
referring to commit which is valid in tag instead of branch.I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).
The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.
Quoted 7 lines
Quoted 4 lines
Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?
(We need to update the docs to reflect that in that case).
In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?
This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.
Cheers,
Richard
Quoted 61 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 augusti 2022 16:17
To: Marek Vasut <[email protected]>; Quentin Schulz <quentin.schulz@theobroma- systems.com>; Alexander Kanavin <[email protected]>
Cc: Mikko Rapeli <[email protected]>; Martin Jansa <[email protected]>; bitbake-devel <bitbake- [email protected]>; Peter Kjellerstedt <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadataQuoted 8 lines · Marek Vasut
Quoted 7 lines · Quentin Schulz
Hi Marek,
Quoted 4 lines · Marek Vasut
Quoted 3 lines · Alexander Kanavin
Quoted 1 line · Marek Vasut
Quoted 1 line
So maybe the easy way out is, if nobranch=1 then fetch
everything,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
else
just heads and tags ?No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Documentation says it does:
3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py- 23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq 8yBP6m6qZZ4njZguQhZhkI_- 172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6o I&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- nobranch
Don't check the SHA validation for branch. set this option forthe
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
recipe
referring to commit which is valid in tag instead of branch.I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong
to
Quoted 4 lines
Quoted 4 lines
any branch anymore (force-push for example, or branch deletion).
The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list
<branch>
Quoted 9 lines
Quoted 6 lines
to check whether a commit exists and can be used by bitbake.
Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?
(We need to update the docs to reflect that in that case).
In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?
Quoted 7 lines
This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.
Cheers,
Richard
I made some testing, and for Gerrit to continue to work it would be enough to use:
fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))
This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.
However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...
//Peter
Quoted 80 lines · Peter Kjellerstedt
Quoted 61 lines
-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 augusti 2022 16:17
To: Marek Vasut <[email protected]>; Quentin Schulz <quentin.schulz@theobroma- systems.com>; Alexander Kanavin <[email protected]>
Cc: Mikko Rapeli <[email protected]>; Martin Jansa <[email protected]>; bitbake-devel <bitbake- [email protected]>; Peter Kjellerstedt <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadataQuoted 8 lines · Marek Vasut
Quoted 7 lines · Quentin Schulz
Hi Marek,
Quoted 4 lines · Marek Vasut
Quoted 3 lines · Alexander Kanavin
Quoted 1 line · Marek Vasut
Quoted 1 line
So maybe the easy way out is, if nobranch=1 then fetch
everything,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
else
just heads and tags ?No, this won't do, nobranch expects the commit to be in a tag.
I don't think it expects that.
Documentation says it does:
3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py- 23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq 8yBP6m6qZZ4njZguQhZhkI_- 172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6o I&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
- nobranch
Don't check the SHA validation for branch. set this option forthe
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
recipe
referring to commit which is valid in tag instead of branch.I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong
to
Quoted 4 lines
Quoted 4 lines
any branch anymore (force-push for example, or branch deletion).
The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list
<branch>
Quoted 9 lines
Quoted 6 lines
to check whether a commit exists and can be used by bitbake.
Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?
(We need to update the docs to reflect that in that case).
In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?
Quoted 7 lines
This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.
Cheers,
Richard
I made some testing, and for Gerrit to continue to work it would be enough to use:
fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.
Quoted 5 lines
However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...
With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.
On 8/22/22 18:39, Marek Vasut wrote:
Hi,
[...]
Quoted 11 lines
Quoted 11 lines
I made some testing, and for Gerrit to continue to work it would be enough to use:
fetch_cmd= "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd,shlex.quote(repourl))This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.
Quoted 7 lines
Quoted 5 lines
However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...
With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.
I keep running into this problem with mesa, how can we proceed to fix it upstream ?
Quoted 25 lines · Marek Vasut
On 8/22/22 18:39, Marek Vasut wrote:
Hi,
[...]
Quoted 19 lines
Quoted 18 lines
I made some testing, and for Gerrit to continue to work it would be enough to use:
fetch_cmd= "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd,shlex.quote(repourl))This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.
However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...
Quoted 6 lines
Quoted 3 lines
With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.
I keep running into this problem with mesa, how can we proceed to fix it upstream ?
We probably need a version of the patch which restricts by default but allows it restriction to be turned off on a per url basis with a parameter.
That restriction needs to be reflected in the mirror tarball name too.
Cheers,
Richard
Hi Marek,
Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
Quoted 5 lines
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>
Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:
WARNING: mesa-2_22.1.6-r0 do_package_qa: QA Issue: File /usr/lib/dri/nouveau_dri.so in package mesa-megadriver contains reference to TMPDIR
Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdio
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 32 lines
-----Original Message-----
From: Luca Ceresoli <[email protected]>
Sent: den 22 augusti 2022 18:03
To: Marek Vasut <[email protected]>
Cc: [email protected]; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadataHi Marek,
Quoted 16 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Quoted 7 lines
Quoted 1 line
Signed-off-by Marek Vasut <[email protected]>Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:
WARNING: mesa-2_22.1.6-r0
do_package_qa: QA Issue: File/usr/lib/dri/nouveau_dri.soin package mesa-megadriver contains reference to TMPDIR
I cannot see any reason how they can be related.
Quoted 8 lines
Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/ste ps/32/logs/stdio--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
//Peter
Quoted 23 lines · Luca Ceresoli
Hi Marek,
Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
Quoted 5 lines
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>
Quoted 9 lines
Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:
WARNING: mesa-2_22.1.6-r0
do_package_qa: QA Issue: File/usr/lib/dri/nouveau_dri.soin package mesa-megadriver contains reference to TMPDIRFull log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdio
This is a known issue with an open bug assigned to me (unfortunately), it isn't related. It is intermittent as it is llvm-native related and we don't commonly rebuild this codepath.
Cheers,
Richard
Hi Richard, Peter,
Quoted 30 lines · On Mon, 22 Aug 2022 17:07:50 +0100 "Richard Purdie"…
Quoted 29 lines · Luca Ceresoli
Hi Marek,
Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:
The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.
Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.
Quoted 5 lines
To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.
Signed-off-by Marek Vasut <[email protected]>Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:
WARNING: mesa-2_22.1.6-r0
do_package_qa: QA Issue: File/usr/lib/dri/nouveau_dri.soin package mesa-megadriver contains reference to TMPDIR
Quoted 7 lines
Quoted 3 lines
Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdioThis is a known issue with an open bug assigned to me (unfortunately), it isn't related. It is intermittent as it is llvm-native related and we don't commonly rebuild this codepath.
Indeed, added to https://bugzilla.yoctoproject.org/show_bug.cgi?id=14897
Thanks for the hint and apologies for the noise.
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com