BB_NO_NETWORK: Fallback to local source for git lsremote
PATCH 1/1 BB_NO_NETWORK: Fallback to local source for git lsremote
Change made to enable building offline also with git tag references in SRC_URI.
If BB_NO_NETWORK is set, fallback to search for tags and revisions in local DL_DIR / GITDIR in order to avoid network access.
---
bitbake/lib/bb/fetch2/git.py | 36 ++++++++++++++++++++++++------------
1 file changed, 24 insertions(+), 12 deletions(-)
bitbake/lib/bb/fetch2/git.py +0 -0
@@ -131,12 +131,6 @@ class Git(FetchMethod):
ud.setup_revisons(d)
- for name in ud.names:
- # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
- if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
- if ud.revisions[name]:
- ud.unresolvedrev[name] = ud.revisions[name]
- ud.revisions[name] = self.latest_revision(ud, d, name)
gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.'))
if gitsrcname.startswith('.'):
@@ -156,6 +150,13 @@ class Git(FetchMethod):
ud.localfile = ud.clonedir
+ for name in ud.names:
+ # Ensure anything that doesn't look like a sha256 checksum/revision is translated into one
+ if not ud.revisions[name] or len(ud.revisions[name]) != 40 or (False in [c in "abcdef0123456789" for c in ud.revisions[name]]):
+ if ud.revisions[name]:
+ ud.unresolvedrev[name] = ud.revisions[name]
+ ud.revisions[name] = self.latest_revision(ud, d, name)
+
def localpath(self, ud, d):
return ud.clonedir
@@ -345,12 +346,23 @@ class Git(FetchMethod):
repourl = self._get_repo_url(ud)
cmd = "%s ls-remote %s %s" % \
(ud.basecmd, repourl, search)
- if ud.proto.lower() != 'file':
- bb.fetch2.check_network_access(d, cmd)
- output = runfetchcmd(cmd, d, True)
- if not output:
- raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
- return output
+ try:
+ if ud.proto.lower() != 'file':
+ bb.fetch2.check_network_access(d, cmd)
+ output = runfetchcmd(cmd, d, True)
+ if not output:
+ raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+ return output
+ except bb.fetch2.NetworkAccess as exc:
+ # Fallback to local DL_DIR/GITDIR if BB_NO_NETWORK is set
+ cmd = "%s ls-remote %s://%s %s" % \
+ (ud.basecmd, 'file', self.localpath(ud, d), search)
+ output = runfetchcmd(cmd, d, True)
+ if output.split() == []:
+ bb.warn('failed: ', cmd)
+ if not output:
+ raise bb.fetch2.FetchError("The command %s gave empty output unexpectedly" % cmd, ud.url)
+ return output
def _latest_revision(self, ud, d, name):
"""
Signature · 1 line
2.1.4
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.
I'm not sure I agree with this since you'd get different build results depending on whether BB_NO_NETWORK is set or not.
I think if BB_NO_NETWORK is set, its reasonable to assume that the configuration should be locked down and that BB_NO_NETWORK should not have other side effects like this.
It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
Cheers,
Richard
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.
Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
This problem means that we have to set all our SRC_URI to git SHA instead of the much more readable and easy to check git tags.
you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Regards
Jérémy Rosen
Quoted 3 lines
Cheers,
Richard
Quoted 25 lines · Jérémy Rosen
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
Quoted 7 lines
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use BB_GENERATE_MIRROR_TARBALLS like mentioned here
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
tags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
Quoted 33 lines · Khem Raj
Quoted 23 lines · Jérémy Rosen
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
Quoted 7 lines
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLSlike mentioned here
Quoted 7 lines
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
tags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
I know... but the fact that NO_NETWORK can't work if any git uses a tag seems... not what I would expect. Especially if the tag is available locally. I understand what you say, but I think it breaks the purpose of NO_NETWORK
If a git reference is floating, then building twice won't produce the same result anyway... this sounds more like a QA problem to me than something that bitbake should enforce. But i'm really not a BB architect, it's just that this patch would solve one of our problem and i'd like to see if there is a way to do that without loosing other aspects of bitbake...
Quoted 39 lines · Jérémy Rosen
Quoted 33 lines · Khem Raj
Quoted 23 lines · Jérémy Rosen
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
Quoted 7 lines
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLSlike mentioned here
Quoted 8 lines
Quoted 7 lines
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source_download_mirror_.3F
tags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
I know... but the fact that
NO_NETWORKcan't work if any git uses a tag seems... not what I would expect. Especially if the tag is available locally. I understand what you say, but I think it breaks the purpose ofNO_NETWORK
There is an assumption here you are making, which is that tags are fixed in git which is not true. However when you use tags in your recipe you want folks to use that tag and it could be moved/displaced under the hood and user would not notice some tags like “last_good” “last_sane_build” are moving tags the are practiced. Tags are in same category as AUTOREV
BB_NO_NETWORK means you need to ensure that metadata can operate on disconnected snapshot which is not true for tags.
Quoted 3 lines
If a git reference is floating, then building twice won't produce the same result anyway... this sounds more like a QA problem to me than something that bitbake should enforce. But i'm really not a BB architect, it's just that this patch would solve one of our problem and i'd like to see if there is a way to do that without loosing other aspects of bitbake…
We should be warning about this case as it does today and to make builds more predictable, this patch takes us in reverse direction. otherwise soon folks have no idea what they are building every developer has his own snapshot think of supporting hundreds of developers where you have no idea what combination they built to get into trouble that they are seeing. It turns into a nightmare very quickly.
Quoted 1 line · Khem Raj
tags are floating types in git, you can not use them reliably.
This is not always true. We could choose to trust tags set according to our processes to not move (e.g. enforced rules that disallows deletion/updates of tags), and only use tags for these recipes, and rely on commit object ids eleswhere.
Tags are only a problem in git if people insist on modifying/deleting them.
Signature · 1 line
olofjn
Quoted 7 lines · Olof Johansson
Quoted 1 line · Khem Raj
tags are floating types in git, you can not use them reliably.
This is not always true.
you are confusing how a tool is specifying its functions with process that could emulate a different behavior. OE does not have any control over the internal processes of its users and it should not have.
Quoted 7 lines
We could choose to trust tags set
according to our processes to not move (e.g. enforced rules that disallows deletion/updates of tags), and only use tags for these recipes, and rely on commit object ids eleswhere.Tags are only a problem in git if people insist on modifying/deleting them.
Fact is, its allowed rewrite tags in git, how people use it or not is irrelevant here.
Quoted 3 lines
--
olofjn
Quoted 4 lines · Khem Raj
you are confusing how a tool is specifying its functions with process that could emulate a different behavior. OE does not have any control over the internal processes of its users and it should not have.
No, I don't think I'm confusing anything. Are you confusing OE and bitbake? We maintain and develop an in-house Poky based distribution, and unlike OE, we know a lot about our users/developers.
Quoted 5 lines
Quoted 2 lines
Tags are only a problem in git if people insist on modifying/deleting them.
Fact is, its allowed rewrite tags in git, how people use it or not is irrelevant here.
Nope, Git is a protocol; git the tool can't do nothing about a server refusing to (re)move a tag. Your view of what a git tags is yours. Tags are meant to be static, otherwise they wouldn't have been synced to all clones in such a silly namespace.
While I agreed with Tobias' patch ("if you use tags in your builds, and use BB_NO_NETWORK, you are ok with falling back to what you have locally"), RP had a point about this making BB_NO_NETWORK have non-obvious side-effects, and also making it harder to catch cases where tags aren't wanted (like in community maintained layers).
But *we* still want to use human readable tags as the only version string internally (for our layers), where we know we are in control. So, introducing a variable that lets us opt-in to this behavior sounds like something that could be accepted?
Signature · 1 line
olofjn
Quoted 2 lines · On 09/08/2016 10:36, "[email protected] on behalf…
and unlike OE, we know a lot about our
users/developers.
We actually know a thing or two about our users, but having worked on both OE and commercial products I have learnt that the degree of variability in the user base and the kind of constraints you can enforce upon them are very different in both contexts.
To be fair, OE has made a significant investment on user-centered approaches. Between other things, by accepting an interaction designer and user researcher (yours truly) between its contributors, something pretty much unheard of among FOSS projects making tools for software engineers.
Cheers
Belén
Quoted 9 lines · Olof Johansson
Quoted 4 lines · Khem Raj
you are confusing how a tool is specifying its functions with process that could emulate a different behavior. OE does not have any control over the internal processes of its users and it should not have.
Quoted 4 lines
No, I don't think I'm confusing anything. Are you confusing OE and bitbake? We maintain and develop an in-house Poky based distribution, and unlike OE, we know a lot about our users/developers.
its a typo.
Thats fine. You can maintain this change locally for your use case.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Tags are only a problem in git if people insist on modifying/deleting them.
Fact is, its allowed rewrite tags in git, how people use it or not is irrelevant here.
Nope, Git is a protocol; git the tool can't do nothing about a
Too literal, I guess a reader is getting what I was referring to
Quoted 11 lines
server refusing to (re)move a tag. Your view of what a git tags is yours. Tags are meant to be static, otherwise they wouldn't have been synced to all clones in such a silly namespace.
While I agreed with Tobias' patch ("if you use tags in your builds, and use
BB_NO_NETWORK, you are ok with falling back to what you have locally"), RP had a point about this makingBB_NO_NETWORKhave non-obvious side-effects, and also making it harder to catch cases where tags aren't wanted (like in community maintained layers).
Quoted 4 lines
But *we* still want to use human readable tags as the only version string internally (for our layers), where we know we are in control. So, introducing a variable that lets us opt-in to this behavior sounds like something that could be accepted?
If you were to think of a tool to generate the srcrevs from tags, that IMO can be better solution here. Like Paul suggested.
Quoted 1 line · Khem Raj
Too literal, I guess a reader is getting what I was referring to
No, a reader did not understood what you are talking about. Quite frankly, you're not being constructive here Khem. Assuming that I'm confusing things because we have differences in opinions, and that it doesn't matter that I don't understand what you are saying as long as you think that other people do is not very helpful. What's the point in discussing anything with you?
Quoted 3 lines
If you were to think of a tool to generate the srcrevs from tags, that IMO can be better solution here. Like Paul suggested.
Yes, Paul's suggestion was very constructive (thanks!), and that's something we might do. But it still forces us to deal with sha1 revision ids. Would require effort on our part to hide this from our users and automate it somehow. Some issues that we would like to handle:
- changing PV won't have any effect on what source is fetched/used
- changing SRCREV to another revision won't affect PV
Signature · 1 line
olofjn
Olof
Quoted 11 lines · Olof Johansson
Quoted 1 line · Khem Raj
Too literal, I guess a reader is getting what I was referring to
No, a reader did not understood what you are talking about. Quite frankly, you're not being constructive here Khem. Assuming that I'm confusing things because we have differences in opinions, and that it doesn't matter that I don't understand what you are saying as long as you think that other people do is not very helpful. What's the point in discussing anything with you?
I am sorry if its coming across this way.I for one was trying to convey that there are reasons to not have such a patch e.g. predictable builds and general support problems with BB_NO_NETWORK.
We have been through this very same problem and decided to use srcrev collection approach which is similar to buildhistory collection approach. It was a bit of getting used to, but in the end it was a very good way.
Quoted 5 lines
Quoted 4 lines
If you were to think of a tool to generate the srcrevs from tags, that IMO can be better solution here. Like Paul suggested.
Quoted 7 lines
Yes, Paul's suggestion was very constructive (thanks!), and that's something we might do. But it still forces us to deal with sha1 revision ids. Would require effort on our part to hide this from our users and automate it somehow. Some issues that we would like to handle:
- changing PV won't have any effect on what source is fetched/used
I believe it should not.
Quoted 1 line
- changing SRCREV to another revision won't affect PV
If you do not utilize SRCPV in PV variable. SRCREV should not reflect in the PV
Quoted 3 lines
--
olofjn
Quoted 8 lines · Khem Raj
I am sorry if its coming across this way.I for one was trying to convey that there are reasons to not have such a patch e.g. predictable builds and general support problems with
BB_NO_NETWORK.We have been through this very same problem and decided to use srcrev collection approach which is similar to buildhistory collection approach. It was a bit of getting used to, but in the end it was a very good way.
Thanks for clarifying, this is a reasonable stance, and I can see where you are coming from. But I'm not advocating that this behavior should be forced upon anybody. What I'm trying to convey is that not all environments have issues with volatile tags, and from my pov, it's reasonably easy to fight people (within an organization) moving tags if you own the Git infrastructure =).
(Having reproducible builds is another way to alleviate the risks of using possibly volatile tags. If the build output is different, then the input obviously is different as well (incl. implicit inputs like datetime, TZ or locale). Just putting it out there. People in the Debian community and elsewhere are doing a lot of work in this area. [1])
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
If you were to think of a tool to generate the srcrevs from tags, that IMO can be better solution here. Like Paul suggested.
Yes, Paul's suggestion was very constructive (thanks!), and that's something we might do. But it still forces us to deal with sha1 revision ids. Would require effort on our part to hide this from our users and automate it somehow. Some issues that we would like to handle:
Quoted 3 lines
Quoted 1 line
- changing PV won't have any effect on what source is fetched/used
I believe it should not.
Yes, that's the heart of our disagreement, I think. We currently set SRCREV = "${PV}". We believe that in our environment, we are safe to do this even without syncing with the remote.
Quoted 4 lines
Quoted 1 line
- changing SRCREV to another revision won't affect PV
If you do not utilize SRCPV in PV variable. SRCREV should not reflect in the PV
Exactly, so that's the problem for me. I'm hesitent to introduce sha1s into the metadata in our layers as it will make us have two variables to keep track of the version. I fear that they will become out of sync.
If somebody tries to only update the SRCREV, the package manifests and elsewhere will be confusing for people trying to troubleshoot as the PV is that of the old version (the same way a moved tag can cause massive confusion). If somebody tries to only update the PV, the package manifest will look correct, but the sources is that of something else. Is this something you have any experience with?
Signature · 3 lines
olofjn 1: https://reproducible-builds.org/
Quoted 12 lines · Olof Johansson
Quoted 10 lines · Khem Raj
I am sorry if its coming across this way.I for one was trying to convey
that there are reasons to not have such a patch e.g. predictable builds and general support problems withBB_NO_NETWORK.We have been through this very same problem and decided to use srcrev
collection approach which is similar to buildhistory collection approach. It was a bit of getting used to, but in the end it was a very good way.
Quoted 6 lines
Thanks for clarifying, this is a reasonable stance, and I can see where you are coming from. But I'm not advocating that this behavior should be forced upon anybody. What I'm trying to convey is that not all environments have issues with volatile tags, and from my pov, it's reasonably easy to fight people (within an organization) moving tags if you own the Git infrastructure =).
The trouble is that whilst you and others posting on this mailing list understand this problem (in some cases now its been explained), in the general case people don't realise that git tags can be rewritten.
If there is an "easy" option to make a failure go away, people will take it every time, without understanding why it can be a bad idea. People prefer not to think if they don't need to.
I've watched git tags get rewritten by upstreams who you'd have thought would have known better, with all kinds of nasty fallout from the result as few people understood what happened, why or how to fix it. The usual "solution" is to blame bitbake or sstate, since they're complex, people don't understand them and they're therefore clearly at fault and broken.
So my position on this is based on experience of how badly you can get burnt by it. I was very reluctant to make bitbake behave in the way it does today originally but in the end various discussions brought me to that conclusion.
What I'm trying to avoid in the fetcher is too many code paths. We already have a lot of them, probably too many, and it makes maintaining and enhancing the fetchers very very hard as you generally end up trampling on some weird usecase someone relies upon. I therefore in general don't like magic switches that drastically alter behaviour, prefering to have one standard path we all use. Add to that the concerns above and the idea of making this optional for people makes me very nervous.
Having better descriptions of revisions is a semi-tangential issue. There have been a few different approaches to using the output of git describe or counting revisions and injecting it into PV. The gitpkgv.bbclass was one similar idea and we did end up putting a method into the fetcher to better enable this where the revision number was based on numbers of commits in a branch. I'd be ok with extending or supplementing that idea to work with git describe so that package output revisions are human readable. I would however want the recipes and fetcher to primarily work with the real sha revisions as those are what define a specific revision in git.
Cheers,
Richard
Quoted 38 lines · On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
Quoted 29 lines · Jérémy Rosen
Quoted 10 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.Quoted 5 lines
you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLSlike mentioned here
Quoted 8 lines
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
_download_mirror_.3Ftags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
FYI, another way to handle this is to enable buildhistory, run your build and then use the buildhistory-collect-srcrevs script. This will produce output that you can save to .inc file that will lock down the SRCREV values to exact hashes, which you can then include and commit with your sources. This allows you to keep your tags and even ${AUTOREV} if you want within each of your recipes and still have everything locked down for future reproducibility.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
I am not very good at internal git magic, but how possible would it be to use the output of git-describe --long in SRC_URI...
This output contains both the tag name and the (abreviated) SHA, so it's both readable for the human user, easy to check and can't change...
Quoted 54 lines · Paul Eggleton
Quoted 35 lines · On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
Quoted 27 lines · Jérémy Rosen
Quoted 9 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.Quoted 5 lines
you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLSlike mentioned hereQuoted 8 lines
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
_download_mirror_.3Ftags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
FYI, another way to handle this is to enable buildhistory, run your build and then use the buildhistory-collect-srcrevs script. This will produce output that you can save to .inc file that will lock down the SRCREV values to exact hashes, which you can then include and commit with your sources. This allows you to keep your tags and even ${AUTOREV} if you want within each of your recipes and still have everything locked down for future reproducibility.
Cheers,
Paul
Thanks for the tip Paul,
I was not aware of this support-script.
This is a good workaround in the current Bitbake environment.
We strive to make it easier for our users to continue building if going offline and as such a step, I will submit a new patchset with some optional flag to use local tags (so that no unexpected side-effect is default in a-action). This since it would be a little less overhead/steps to take when building offline.
Cheers,
Tobias
________________________________________
From: Paul Eggleton <[email protected]>
Sent: Tuesday, August 9, 2016 5:22 AM
To: [email protected]
Cc: Khem Raj; Jérémy Rosen; Tobias Hagelborn; Richard Purdie
Subject: Re: [bitbake-devel] [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote
Quoted 38 lines · On Mon, 08 Aug 2016 08:35:38 Khem Raj wrote:
Quoted 29 lines · Jérémy Rosen
Quoted 10 lines · Richard Purdie
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.I'm not sure I agree with this since you'd get different build results depending on whether
BB_NO_NETWORKis set or not.Quoted 6 lines
I think if
BB_NO_NETWORKis set, its reasonable to assume that the configuration should be locked down and thatBB_NO_NETWORKshould not have other side effects like this.It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
To be honest, this is a real world problem I have on a regular basis...
The problem we have is that our customers want to keep an archive for (very) long term support and they test offline mode.
This problem means that we have to set all our
SRC_URIto git SHA instead of the much more readable and easy to check git tags.Quoted 5 lines
you think to assume that a reference is a branch, but the real-life problem is when reference is a tag. Tag don't move so this is a case of locked configuration, but i'm not sure how to handle it.
Is there a way to have the best of both world somehow ?
Don’t use tags instead rely on SHA numbers in SRCREV and use
BB_GENERATE_MIRROR_TARBALLSlike mentioned here
Quoted 8 lines
https://wiki.yoctoproject.org/wiki/How_do_I#Q:_How_do_I_create_my_own_source
_download_mirror_.3Ftags are floating types in git, you can not use them reliably. You may tool in a method to automate updating the recipes SRCREVs if you dont want to do it by hand.
That should help hopefully
FYI, another way to handle this is to enable buildhistory, run your build and then use the buildhistory-collect-srcrevs script. This will produce output that you can save to .inc file that will lock down the SRCREV values to exact hashes, which you can then include and commit with your sources. This allows you to keep your tags and even ${AUTOREV} if you want within each of your recipes and still have everything locked down for future reproducibility.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
Hi Richard,
I think I understand your concern. This patch takes whatever is available locally and that means it is a side-effect compared to not setting BB_NO_NETWORK. Something the user might not expect.
I don't know if you would accept the use of a new flag that the user can set to agree with this side effect.
Something along the lines of "FORCE_LOCAL_TAGS".
Then the user would have given some consent on that side-affect.
We think it is still very useful to us to be able to build locally with whatever you have fetched in cases where no NW is available and you basically know that few/no tags have moved.
Regards
Tobias
________________________________________
From: Richard Purdie <[email protected]>
Sent: Monday, August 8, 2016 11:55 AM
To: Tobias Hagelborn; [email protected]
Subject: Re: [bitbake-devel] [PATCH] BB_NO_NETWORK: Fallback to local source for git lsremote
Quoted 5 lines · Tobias Hagelborn
Change made to enable building offline also with git tag references in
SRC_URI.If
BB_NO_NETWORKis set, fallback to search for tags and revisions in localDL_DIR/ GITDIR in order to avoid network access.
I'm not sure I agree with this since you'd get different build results depending on whether BB_NO_NETWORK is set or not.
I think if BB_NO_NETWORK is set, its reasonable to assume that the configuration should be locked down and that BB_NO_NETWORK should not have other side effects like this.
It would also make it very hard to detect unlocked configurations during testing which currently are quite easily identified.
Cheers,
Richard