Skip to contentSkip to conversation

fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

RFC PATCH In review Marek Vasut 7 participants 23 messages Aug 19, 2022 – Sep 2, 2022

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>

---
 lib/bb/fetch2/git.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/fetch2/git.py +1 -1
index 4534bd75..b5fc0a51 100644
@@ -382,7 +382,7 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
Signature · 1 line
2.35.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 46 lines

-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>

---
 lib/bb/fetch2/git.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/fetch2/git.py +1 -2
index 4534bd75..b5fc0a51 100644
@@ -382,7 +382,7 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
--
Quoted 1 line

2.35.1

Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....

//Peter

#2 lore raw

Hi,

Quoted 39 lines · Peter Kjellerstedt
Quoted 39 lines

-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>

---
 lib/bb/fetch2/git.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/bb/fetch2/git.py +0 -0
index 4534bd75..b5fc0a51 100644
@@ -382,7 +382,7 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir=ud.clonedir)
 
Quoted 14 lines
Quoted 8 lines
lib/bb/fetch2/git.py +1 -2
             runfetchcmd("%s remote add --mirror=fetch origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)
-            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))
+            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
--

2.35.1

Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....

As user of Gerrit, I fear this will cause problems. In my case developers are used to creating test topics and using git hashes in recipes which are not yet released, e.g. not yet in release branches or tags. This can of course create problems when such changes end up in real releases.

Workaround is that developers can create throw away testing branches and refer to them in recipes.

From one side this is an improvement to have less data in caches, but on the other side this adds extra actions to developers who want to test changes to their recipes. Can't decide which one is more important though :/

Cheers,

-Mikko

#3 lore raw

Can be solved with a parameter to a fetcher perhaps?

Alex

Quoted 94 lines · Mikko Rapeli

Hi,

Quoted 5 lines · Peter Kjellerstedt
Quoted 5 lines

-----Original Message-----
From: Marek Vasut <[email protected]>
Sent: den 19 augusti 2022 18:55
To: [email protected]
Cc: Marek Vasut <[email protected]>; Martin Jansa <[email protected]>;

Peter Kjellerstedt <[email protected]>; Richard Purdie < [email protected]>

Quoted 1 line
Quoted 1 line

Subject: [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching

gitlab repository metadata

Quoted 6 lines
Quoted 6 lines

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Quoted 8 lines
Quoted 8 lines

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher

such

Quoted 1 line
Quoted 1 line

that it only fetches refs/heads/* and refs/tags/* . Avoid using

negative

Quoted 11 lines
Quoted 11 lines

refspecs as those are only available in new git versions.

---
Cc: Martin Jansa <[email protected]>
Cc: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>

---
 lib/bb/fetch2/git.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 6 lines

lib/bb/fetch2/git.py class Git(FetchMethod): · line 382

Quoted 6 lines
lib/bb/fetch2/git.py +0 -0
index 4534bd75..b5fc0a51 100644
@@ -382,7 +382,7 @@ class Git(FetchMethod):
               runfetchcmd("%s remote rm origin" % ud.basecmd, d,

workdir=ud.clonedir)

Quoted 2 lines

lib/bb/fetch2/git.py class Git(FetchMethod): · line 383

Quoted 2 lines
lib/bb/fetch2/git.py +0 -0
             runfetchcmd("%s remote add --mirror=fetch origin %s" %

(ud.basecmd, shlex.quote(repourl)), d, workdir=ud.clonedir)

Quoted 1 line

lib/bb/fetch2/git.py class Git(FetchMethod): · line 384

Quoted 1 line
lib/bb/fetch2/git.py +0 -1
-            fetch_cmd = "LANG=C %s fetch -f --progress %s

refs/*:refs/*" % (ud.basecmd, shlex.quote(repourl))

Quoted 1 line

lib/bb/fetch2/git.py class Git(FetchMethod): · line 384

Quoted 1 line
lib/bb/fetch2/git.py +1 -0
+            fetch_cmd = "LANG=C %s fetch -f --progress %s

refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*" % (ud.basecmd, shlex.quote(repourl))

Quoted 11 lines
Quoted 5 lines
lib/bb/fetch2/git.py +0 -1
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
--

2.35.1

Seems like the right thing to do. We use Gerrit, which also has its metadata in special refs/ spaces. One repository I tested with grew from 3 MB to 35 MB when I fetched using refs/* while another grew from 20 MB to 120 MB, so there is definitely space and time to be saved by only fetching the refs/heads and refs/tags spaces....

As user of Gerrit, I fear this will cause problems. In my case developers are used to creating test topics and using git hashes in recipes which are not yet released, e.g. not yet in release branches or tags. This can of course create problems when such changes end up in real releases.

Workaround is that developers can create throw away testing branches and refer to them in recipes.

From one side this is an improvement to have less data in caches, but on the other side this adds extra actions to developers who want to test changes to their recipes. Can't decide which one is more important though :/

Cheers,

-Mikko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#13910):
https://lists.openembedded.org/g/bitbake-devel/message/13910
Mute This Topic: https://lists.openembedded.org/mt/93128921/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
Quoted 1 line · Alexander Kanavin

Can be solved with a parameter to a fetcher perhaps?

Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.

This could be used for fetching the changes too, to limit the scope.

Cheers,

-Mikko

#5 lore raw
Quoted 7 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin

Can be solved with a parameter to a fetcher perhaps?

Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.

This could be used for fetching the changes too, to limit the scope.

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

#6 lore raw
Quoted 10 lines · Marek Vasut
Quoted 7 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin

Can be solved with a parameter to a fetcher perhaps?

Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.

This could be used for fetching the changes too, to limit the scope.

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

#7 lore raw
Quoted 5 lines · Marek Vasut
Quoted 4 lines · Marek Vasut
Quoted 3 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin

Can be solved with a parameter to a fetcher perhaps?

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.

This could be used for fetching the changes too, to limit the scope.

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

But then, if gerrit works with nobranch=1, then gerrit must be generating tags which contain the commits you test ?

And since this patch fetches refs/tags/ , then this patch won't break the gerrit setup ?

#8 lore raw

Hi,

Quoted 6 lines · Marek Vasut
Quoted 5 lines · Marek Vasut
Quoted 4 lines · Marek Vasut
Quoted 3 lines · On 8/22/22 09:38, [email protected] wrote:
Quoted 1 line · Alexander Kanavin

Can be solved with a parameter to a fetcher perhaps?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Frequently developers know to change the git URL in recipes from "branch=master" to "nobranch=1" for their test commits.

This could be used for fetching the changes too, to limit the scope.

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

To me this would be the way to go.

Quoted 7 lines
Quoted 1 line

No, this won't do, nobranch expects the commit to be in a tag.

But then, if gerrit works with nobranch=1, then gerrit must be generating tags which contain the commits you test ?

And since this patch fetches refs/tags/ , then this patch won't break the gerrit setup ?

nobranch=1 works with any branch or tag or open gerrit review commit id. At least with sumo and dunfell.

Cheers,

-Mikko

#9 lore raw
In reply to Marek Vasut
Quoted 4 lines · Marek Vasut
Quoted 2 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Alex

#10 lore raw
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Documentation says it does:

https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n45
"
- nobranch
Don't check the SHA validation for branch. set this option for the
recipe
referring to commit which is valid in tag instead of branch. The default is "0", set nobranch=1 if needed.
"

#11 lore raw
Quoted 12 lines · Marek Vasut
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Documentation says it does:

Quoted 7 lines

https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/git.py#n45
"
- nobranch
Don't check the SHA validation for branch. set this option for the recipe referring to commit which is valid in tag instead of branch. The default is "0", set nobranch=1 if needed.
"

Only the first sentence is enforced. The change can still be in a branch, in a tag, in random other namespace as long as the commit is found at checkout time.

Cheers,

-Mikko

#12 lore raw
In reply to Marek Vasut

Hi Marek,

Quoted 10 lines · Marek Vasut
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Quoted 7 lines

Documentation says it does:

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
   Don't check the SHA validation for branch. set this option for the
recipe
   referring to commit which is valid in tag instead of branch.

I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).

The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.

Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?

(We need to update the docs to reflect that in that case).

Cheers,
Quentin

Quoted 12 lines

   The default is "0", set nobranch=1 if needed.
"

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#13918): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_message_13918&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=g4KWyxwbq71V3gbvIJNG-oA9Gdvj3A5wqfz8Kws5qZg&e=
Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93128921_6293953&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=KMelWJJL5NtG7NWmtiS3jFAONb4GRttyl1ziLzEHhr8&e=
Group Owner: [email protected]
Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_unsub&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=nyYO56WzR2jmLa4g95pgXToervc-fJhqbVjnOOUDm0g&e= [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#13 lore raw
Quoted 20 lines · Quentin Schulz

Hi Marek,

Quoted 8 lines · Marek Vasut
Quoted 7 lines · Alexander Kanavin
Quoted 4 lines · Marek Vasut
Quoted 2 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

Quoted 8 lines
Quoted 1 line

I don't think it expects that.

Documentation says it does:

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
    Don't check the SHA validation for branch. set this option for the recipe
    referring to commit which is valid in tag instead of branch.

Quoted 7 lines

I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).

The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.

Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?

(We need to update the docs to reflect that in that case).

In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?

#14 lore raw
Quoted 31 lines · Marek Vasut
Quoted 30 lines · Quentin Schulz

Hi Marek,

Quoted 11 lines · Marek Vasut
Quoted 9 lines · Alexander Kanavin
Quoted 5 lines · Marek Vasut
Quoted 3 lines

So maybe the easy way out is, if nobranch=1 then fetch everything, else
just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Quoted 7 lines

Documentation says it does:

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6oI&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
- nobranch
    Don't check the SHA validation for branch. set this option for the
recipe
    referring to commit which is valid in tag instead of branch.

I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong to any branch anymore (force-push for example, or branch deletion).

The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list <branch> to check whether a commit exists and can be used by bitbake.

Quoted 7 lines
Quoted 4 lines

Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?

(We need to update the docs to reflect that in that case).

In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?

This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.

Cheers,

Richard

#15 lore raw
Quoted 61 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 augusti 2022 16:17
To: Marek Vasut <[email protected]>; Quentin Schulz <quentin.schulz@theobroma- systems.com>; Alexander Kanavin <[email protected]>
Cc: Mikko Rapeli <[email protected]>; Martin Jansa <[email protected]>; bitbake-devel <bitbake- [email protected]>; Peter Kjellerstedt <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

Quoted 8 lines · Marek Vasut
Quoted 7 lines · Quentin Schulz

Hi Marek,

Quoted 4 lines · Marek Vasut
Quoted 3 lines · Alexander Kanavin
Quoted 1 line · Marek Vasut
Quoted 1 line

So maybe the easy way out is, if nobranch=1 then fetch

everything,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

else
just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Documentation says it does:

https://urldefense.proofpoint.com/v2/url?u=https-

3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py- 23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq 8yBP6m6qZZ4njZguQhZhkI_- 172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6o I&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

- nobranch
    Don't check the SHA validation for branch. set this option for

the

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

recipe
    referring to commit which is valid in tag instead of branch.

I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong

to

Quoted 4 lines
Quoted 4 lines

any branch anymore (force-push for example, or branch deletion).

The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list

<branch>

Quoted 9 lines
Quoted 6 lines

to check whether a commit exists and can be used by bitbake.

Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?

(We need to update the docs to reflect that in that case).

In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?

Quoted 7 lines

This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.

Cheers,

Richard

I made some testing, and for Gerrit to continue to work it would be enough to use:

            fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))

This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.

However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...

//Peter

#16 lore raw
Quoted 80 lines · Peter Kjellerstedt
Quoted 61 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 augusti 2022 16:17
To: Marek Vasut <[email protected]>; Quentin Schulz <quentin.schulz@theobroma- systems.com>; Alexander Kanavin <[email protected]>
Cc: Mikko Rapeli <[email protected]>; Martin Jansa <[email protected]>; bitbake-devel <bitbake- [email protected]>; Peter Kjellerstedt <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

Quoted 8 lines · Marek Vasut
Quoted 7 lines · Quentin Schulz

Hi Marek,

Quoted 4 lines · Marek Vasut
Quoted 3 lines · Alexander Kanavin
Quoted 1 line · Marek Vasut
Quoted 1 line

So maybe the easy way out is, if nobranch=1 then fetch

everything,

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

else
just heads and tags ?

No, this won't do, nobranch expects the commit to be in a tag.

I don't think it expects that.

Documentation says it does:

https://urldefense.proofpoint.com/v2/url?u=https-

3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py- 23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq 8yBP6m6qZZ4njZguQhZhkI_- 172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6o I&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

- nobranch
    Don't check the SHA validation for branch. set this option for

the

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

recipe
    referring to commit which is valid in tag instead of branch.

I assume this was meant to give the example of tags which aren't necessarily in a branch (annotated tags or tags of commits not belong

to

Quoted 4 lines
Quoted 4 lines

any branch anymore (force-push for example, or branch deletion).

The git fetcher does a git log --pretty=oneline -n 1 <hash> when nobranch is set, otherwise git branch --contains <hash> --list

<branch>

Quoted 9 lines
Quoted 6 lines

to check whether a commit exists and can be used by bitbake.

Considering this check, I assume nobranch=1 is working for any commit that was fetched by the git fetcher?

(We need to update the docs to reflect that in that case).

In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head refs/tags' otherwise ?

Quoted 7 lines

This does get a bit more complex though since you now need two different mirror tarballs, one for each option. The code can do that if setup correctly but we do need to cover that issue.

Cheers,

Richard

I made some testing, and for Gerrit to continue to work it would be enough to use:

             fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))

This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.

Quoted 5 lines

However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...

With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.

#17 lore raw

On 8/22/22 18:39, Marek Vasut wrote:

Hi,

[...]

Quoted 11 lines
Quoted 11 lines

I made some testing, and for Gerrit to continue to work it would be enough to use:

             fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))

This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.

Quoted 7 lines
Quoted 5 lines

However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...

With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.

I keep running into this problem with mesa, how can we proceed to fix it upstream ?

#18 lore raw
Quoted 25 lines · Marek Vasut

On 8/22/22 18:39, Marek Vasut wrote:

Hi,

[...]

Quoted 19 lines
Quoted 18 lines

I made some testing, and for Gerrit to continue to work it would be enough to use:

             fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))

This should not affect other Git servers and should avoid using different fetch commands depending on the URL. The drawback is of course that for Gerrit, there would be only marginal benefits to this change since the majority of its metadata is in the refs/changes space.

However, I wonder if the suggested change actually has any significant effect, given that the initial clone is done using --mirror, which means all refs/ spaces are fetched. If I remove the --mirror option from the clone command the change works as expected, but I have no idea if that has any other significant impact...

Quoted 6 lines
Quoted 3 lines

With this change, I am able to actually fetch mesa from gitlab.freedesktop.org without local CI proxy terminating the connection in the process. So yes, it does have effect.

I keep running into this problem with mesa, how can we proceed to fix it upstream ?

We probably need a version of the patch which restricts by default but allows it restriction to be turned off on a per url basis with a parameter.

That restriction needs to be reflected in the mirror tarball name too.

Cheers,

Richard

#19 lore raw
In reply to Marek Vasut

Hi Marek,

Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

Quoted 5 lines

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:

WARNING: mesa-2_22.1.6-r0 do_package_qa: QA Issue: File /usr/lib/dri/nouveau_dri.so in package mesa-megadriver contains reference to TMPDIR

Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdio

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#20 lore raw
Quoted 32 lines

-----Original Message-----
From: Luca Ceresoli <[email protected]>
Sent: den 22 augusti 2022 18:03
To: Marek Vasut <[email protected]>
Cc: [email protected]; Martin Jansa <[email protected]>; Peter Kjellerstedt <[email protected]>; Richard Purdie <[email protected]>
Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

Hi Marek,

Quoted 16 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

Quoted 7 lines
Quoted 1 line

Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:

WARNING: mesa-2_22.1.6-r0 do_package_qa: QA Issue: File /usr/lib/dri/nouveau_dri.so in package mesa-megadriver contains reference to TMPDIR

I cannot see any reason how they can be related.

Quoted 8 lines

Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/ste ps/32/logs/stdio

--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

//Peter

#21 lore raw
In reply to Luca Ceresoli
Quoted 23 lines · Luca Ceresoli

Hi Marek,

Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

Quoted 5 lines

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

Quoted 9 lines

Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:

WARNING: mesa-2_22.1.6-r0 do_package_qa: QA Issue: File /usr/lib/dri/nouveau_dri.so in package mesa-megadriver contains reference to TMPDIR

Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdio

This is a known issue with an open bug assigned to me (unfortunately), it isn't related. It is intermittent as it is llvm-native related and we don't commonly rebuild this codepath.

Cheers,

Richard

#22 lore raw

Hi Richard, Peter,

Quoted 30 lines · On Mon, 22 Aug 2022 17:07:50 +0100 "Richard Purdie"…
Quoted 29 lines · Luca Ceresoli

Hi Marek,

Quoted 12 lines · On Fri, 19 Aug 2022 18:54:55 +0200 "Marek Vasut" <[email protected]> wrote:

The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such.

Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output.

Quoted 5 lines

To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions.

Of course this might become irrelevant with whatever implementation will be in v2, however when testing with this patch applied I got the following warning and wonder whether they are related:

WARNING: mesa-2_22.1.6-r0 do_package_qa: QA Issue: File /usr/lib/dri/nouveau_dri.so in package mesa-megadriver contains reference to TMPDIR

Quoted 7 lines
Quoted 3 lines

Full log:
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/5753/steps/32/logs/stdio

This is a known issue with an open bug assigned to me (unfortunately), it isn't related. It is intermittent as it is llvm-native related and we don't commonly rebuild this codepath.

Indeed, added to https://bugzilla.yoctoproject.org/show_bug.cgi?id=14897

Thanks for the hint and apologies for the noise.

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com