fetch2: Unify BB_FETCH_PREMIRRORONLY
cover fetch2: Unify BB_FETCH_PREMIRRORONLY
The following changes since commit 753e2a0ede4449917c75353b57f13bbafe70fac8:
build-appliance-image: Update to master head revision (2019-03-19 23:56:29 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/premirror
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/premirror
Robert Yang (2):
fetch2: runfetchcmd(): Print workdir in debug message
fetch2: Unify BB_FETCH_PREMIRRORONLY
bitbake/lib/bb/fetch2/__init__.py | 5 ++++- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/2 fetch2: runfetchcmd(): Print workdir in debug message
The workdir is an important message when run git command, for exmample, before the patch:
Running git -c core.fsyncobjectfiles=0 branch --contains [snip]
We don't know where it is running, now it is:
Running 'git -c core.fsyncobjectfiles=0 branch --contains [snip]' in /path/to/git.opensvc.com.multipath-tools..git
Which is easier to debug.
--- bitbake/lib/bb/fetch2/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/__init__.py +4 -1
@@ -858,7 +858,10 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None): # Disable pseudo as it may affect ssh, potentially causing it to hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd - logger.debug(1, "Running %s", cmd) + if workdir: + logger.debug(1, "Running '%s' in %s" % (cmd, workdir)) + else: + logger.debug(1, "Running %s", cmd) success = False error_message = ""
Signature · 1 line
2.7.4
PATCH 2/2 fetch2: Unify BB_FETCH_PREMIRRORONLY
In reply to Robert YangThe fetch2/__init__.py checks whether "BB_FETCH_PREMIRRORONLY" == "1", but fetch2/git.py and hg.py checks whether it is None, this makes it discontinuous, and BB_FETCH_PREMIRRORONLY = "0" doens't work as expected in the later case, so unify it to the previous one. (As BB_NO_NETWORK does).
--- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
bitbake/lib/bb/fetch2/git.py +1 -1
@@ -318,7 +318,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.clonedir): return False
bitbake/lib/bb/fetch2/hg.py +1 -1
@@ -99,7 +99,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.moddir): return False
Signature · 1 line
2.7.4
Quoted 39 lines · Robert Yang
The
fetch2/__init__.pychecks whether "BB_FETCH_PREMIRRORONLY" == "1", butfetch2/git.pyandhg.pychecks whether it is None, this makes it discontinuous,
andBB_FETCH_PREMIRRORONLY= "0" doens't work as expected in the later case,
so unify it to the previous one. (AsBB_NO_NETWORKdoes).Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -318,7 +318,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout withonly premirrors
# is not possiblebitbake/lib/bb/fetch2/git.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.clonedir): return Falsebitbake/lib/bb/fetch2/hg.py +0
-0@@ -99,7 +99,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout withonly premirrors
# is not possible
bitbake/lib/bb/fetch2/hg.py class Hg(FetchMethod): · line 105
Quoted 5 lines
bitbake/lib/bb/fetch2/hg.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.moddir): return False
Can we use bb.utils.to_boolean() in all cases to be consistent please?
Cheers,
Richard
Hi RP,
Quoted 39 lines · Richard Purdie
Quoted 38 lines · Robert Yang
The
fetch2/__init__.pychecks whether "BB_FETCH_PREMIRRORONLY" == "1", butfetch2/git.pyandhg.pychecks whether it is None, this makes it discontinuous,
andBB_FETCH_PREMIRRORONLY= "0" doens't work as expected in the later case,
so unify it to the previous one. (AsBB_NO_NETWORKdoes).Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -318,7 +318,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout withonly premirrors
# is not possiblebitbake/lib/bb/fetch2/git.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.clonedir): return Falsebitbake/lib/bb/fetch2/hg.py +0
-0@@ -99,7 +99,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout withonly premirrors
Quoted 8 lines
bitbake/lib/bb/fetch2/hg.py class Hg(FetchMethod): · line 105
Quoted 6 lines
# is not possiblebitbake/lib/bb/fetch2/hg.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.moddir): return FalseCan we use
bb.utils.to_boolean()in all cases to be consistent please?
Thanks, I will update it.
I have another question about try_premirror(), are there any side effects if we make try_premirror() always return true? I mean that:
def try_premirror(self, ud, d):
return True
I've met a problem recently, when rework in an old build, e.g, suppose there is a foo_git.bb:
1) Add foo.git to PREMIRROR
2) In builddir
$ bitbake foo Then foo.git will be fetched into DL_DIR, which is premirror_foo.git.
3) Upgrade foo_git.bb's SRCREV, and this SRCREV isn't in premirror_foo.git since it's very new.
4) Upgrade foo.git in PREMIRROR, now it contains the SRCREV of step 3
5) Go to the old builddir
$ bitbake foo
Then it won't update from PREMIRROR if BB_FETCH_PREMIRRORONLY is not set, and it would fail to build if BB_NO_NETWORK = "1" since SRCREV can't be found in DL_DIR. Now the problem is:
- Set BB_FETCH_PREMIRRORONLY = "0" and BB_NO_NETWORK = "1" will make foo fail to build.
- Set BB_FETCH_PREMIRRORONLY = "1" will make it work for foo_git.bb, but it may not work for another recipe such as foo_X_git.bb even if BB_NO_NETWORK = "0".(Imagine the PREMIRROR doesn't contain foo_X_git.bb's SRCREV)
- Set BB_FETCH_PREMIRRORONLY = "0" and BB_NO_NETWORK = "0" will make foo work, but it won't upgrade from PREMIRROR, though everything is up-to-date there. It will update DL_DIR from SRC_URI, this isn't good when network is slow.
Things will be much easier if make try_premirror() always return True.
// Robert
Quoted 7 lines
Cheers,
Richard
Quoted 82 lines · Robert Yang
Hi RP,
Quoted 42 lines · Richard Purdie
Quoted 41 lines · Robert Yang
The
fetch2/__init__.pychecks whether "BB_FETCH_PREMIRRORONLY" == "1", butfetch2/git.pyandhg.pychecks whether it is None, this makes it discontinuous,
andBB_FETCH_PREMIRRORONLY= "0" doens't work as expected in the later case,
so unify it to the previous one. (AsBB_NO_NETWORKdoes).Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -318,7 +318,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkoutwith
only premirrors# is not possiblebitbake/lib/bb/fetch2/git.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.clonedir): return Falsebitbake/lib/bb/fetch2/hg.py +0
-0@@ -99,7 +99,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkoutwith
only premirrors# is not possibleQuoted 8 lines
bitbake/lib/bb/fetch2/hg.py class Hg(FetchMethod): · line 105
Quoted 5 lines
bitbake/lib/bb/fetch2/hg.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.moddir): return FalseCan we use
bb.utils.to_boolean()in all cases to be consistent please?Thanks, I will update it.
I have another question about
try_premirror(), are there any side effects
if we maketry_premirror()always return true? I mean that:def
try_premirror(self, ud, d):
return TrueI've met a problem recently, when rework in an old build, e.g, suppose
there is a foo_git.bb:1) Add foo.git to PREMIRROR
2) In builddir
$ bitbake foo Then foo.git will be fetched into DL_DIR, which is premirror_foo.git.3) Upgrade foo_git.bb's SRCREV, and this SRCREV isn't in premirror_foo.git since
it's very new.4) Upgrade foo.git in PREMIRROR, now it contains the SRCREV of step 3
5) Go to the old builddir
$ bitbake foo
Quoted 8 lines
Then it won't update from PREMIRROR if
BB_FETCH_PREMIRRORONLYis not set,
and it would fail to build ifBB_NO_NETWORK= "1" since SRCREV can't be found
inDL_DIR. Now the problem is:- Set
BB_FETCH_PREMIRRORONLY= "0" andBB_NO_NETWORK= "1" will make foo fail to build.
That is expected to fail as the user has said specifically not to use premirrors.
Quoted 6 lines
- Set
BB_FETCH_PREMIRRORONLY= "1" will make it work for foo_git.bb, but it
may not work for another recipe such as foo_X_git.bb even ifBB_NO_NETWORK= "0".(Imagine the PREMIRROR doesn't contain
foo_X_git.bb's
SRCREV)
Here, the user has said to use premirrors *only* so that is what the code should do.
Quoted 6 lines
- Set
BB_FETCH_PREMIRRORONLY= "0" andBB_NO_NETWORK= "0" will make foo work,
but it won't upgrade from PREMIRROR, though everything is up-to-
date there.
It will updateDL_DIRfromSRC_URI, this isn't good when network
is slow.
The behaviour of BB_FETCH_PREMIRRORONLY = "0" will change after your patch, it will make it behave as if BB_FETCH_PREMIRRORONLY was unset so I think your patch will fix this case?
Quoted 2 lines
Things will be much easier if make
try_premirror()always return True.
This gets a little complex, for example if premirrors is a tarball mirror of a git repo rather than a full repo.
If you change this to always return True, it would always download the mirror tarball for a git repo rather than trying to do a git pull in the main repo which would just download the delta.
The correct behaviour would probably be to work out whether we can do a "git fetch" update from any source, then go back to pulling mirror tarballs if we can't get the revision by the other means.
We should probably file an enhancement bug for that and put a comment about this in the code as this is a key behaviour we rely upon. I don't want to change this part of the fetcher at this point in the release cycle though as this kind of change is fraught with risk :(.
Cheers,
Richard
Hi RP,
Quoted 112 lines · On 3/22/19 7:18 PM, [email protected] wrote:
Quoted 82 lines · Robert Yang
Hi RP,
Quoted 42 lines · Richard Purdie
Quoted 41 lines · Robert Yang
The
fetch2/__init__.pychecks whether "BB_FETCH_PREMIRRORONLY" == "1", butfetch2/git.pyandhg.pychecks whether it is None, this makes it discontinuous,
andBB_FETCH_PREMIRRORONLY= "0" doens't work as expected in the later case,
so unify it to the previous one. (AsBB_NO_NETWORKdoes).Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 2 +- bitbake/lib/bb/fetch2/hg.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -318,7 +318,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkoutwith
only premirrors# is not possiblebitbake/lib/bb/fetch2/git.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.clonedir): return Falsebitbake/lib/bb/fetch2/hg.py +0
-0@@ -99,7 +99,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkoutwith
only premirrors# is not possibleQuoted 8 lines
bitbake/lib/bb/fetch2/hg.py class Hg(FetchMethod): · line 105
Quoted 5 lines
bitbake/lib/bb/fetch2/hg.py +1
-1- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if d.getVar("BB_FETCH_PREMIRRORONLY") == "1": return True if os.path.exists(ud.moddir): return FalseCan we use
bb.utils.to_boolean()in all cases to be consistent please?Thanks, I will update it.
I have another question about
try_premirror(), are there any side effects
if we maketry_premirror()always return true? I mean that:def
try_premirror(self, ud, d):
return TrueI've met a problem recently, when rework in an old build, e.g, suppose
there is a foo_git.bb:1) Add foo.git to PREMIRROR
2) In builddir
$ bitbake foo Then foo.git will be fetched into DL_DIR, which is premirror_foo.git.3) Upgrade foo_git.bb's SRCREV, and this SRCREV isn't in premirror_foo.git since
it's very new.4) Upgrade foo.git in PREMIRROR, now it contains the SRCREV of step 3
5) Go to the old builddir
$ bitbake fooQuoted 8 lines
Then it won't update from PREMIRROR if
BB_FETCH_PREMIRRORONLYis not set,
and it would fail to build ifBB_NO_NETWORK= "1" since SRCREV can't be found
inDL_DIR. Now the problem is:- Set
BB_FETCH_PREMIRRORONLY= "0" andBB_NO_NETWORK= "1" will make foo fail to build.That is expected to fail as the user has said specifically not to use premirrors.
Quoted 6 lines
- Set
BB_FETCH_PREMIRRORONLY= "1" will make it work for foo_git.bb, but it
may not work for another recipe such as foo_X_git.bb even ifBB_NO_NETWORK= "0".(Imagine the PREMIRROR doesn't contain
foo_X_git.bb's
SRCREV)Here, the user has said to use premirrors *only* so that is what the code should do.
Quoted 6 lines
- Set
BB_FETCH_PREMIRRORONLY= "0" andBB_NO_NETWORK= "0" will make foo work,
but it won't upgrade from PREMIRROR, though everything is up-to-
date there.
It will updateDL_DIRfromSRC_URI, this isn't good when network
is slow.
Quoted 3 lines
The behaviour of
BB_FETCH_PREMIRRORONLY= "0" will change after your patch, it will make it behave as ifBB_FETCH_PREMIRRORONLYwas unset so I think your patch will fix this case?
Unfortunately, no, my patch doesn't fix this problem. My patches fixes BB_FETCH_PREMIRRORONLY's inconsistent behavior between fetch2/__init__.py and fetch2/git.py or hg.py.
Quoted 11 lines
Quoted 3 lines
Things will be much easier if make
try_premirror()always return True.This gets a little complex, for example if premirrors is a tarball mirror of a git repo rather than a full repo.
If you change this to always return True, it would always download the mirror tarball for a git repo rather than trying to do a git pull in the main repo which would just download the delta.
Quoted 5 lines
The correct behaviour would probably be to work out whether we can do a "git fetch" update from any source, then go back to pulling mirror tarballs if we can't get the revision by the other means.
We should probably file an enhancement bug for that and put a comment
Sound good, I filed a enhancement bug for 2.8 M1:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13233
Bug 13233 - fetch2: try_premirror(): improve on updating repo from mirror
I will send a V2 to use bb.utils.to_boolean() atm.
// Robert
Quoted 11 lines
about this in the code as this is a key behaviour we rely upon. I don't want to change this part of the fetcher at this point in the release cycle though as this kind of change is fraught with risk :(.
Cheers,
Richard