fetch2: try_mirrors(): Skip invalid local url
cover fetch2: try_mirrors(): Skip invalid local url
From: Robert Yang <[email protected]>
The following changes since commit 3b2f9c31dc17c3ba5f6a9971413afed3d450d826:
poky.yaml.in: increase required disk space to 140G (2025-08-08 23:41:11 +0100)
are available in the Git repository at:
https://github.com/robertlinux/bitbake rbt/mirror
https://github.com/robertlinux/bitbake/tree/rbt/mirror
Robert Yang (1):
fetch2: try_mirrors(): Skip invalid local url
bitbake/lib/bb/fetch2/__init__.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.49.0
PATCH 1/1 fetch2: try_mirrors(): Skip invalid local url
From: Robert Yang <[email protected]>
Skip fetching from an invalid local url since it would be failed.
--- bitbake/lib/bb/fetch2/__init__.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/__init__.py +5 -1
@@ -1168,7 +1168,11 @@ def try_mirrors(fetch, d, origud, mirrors, check = False): uris, uds = build_mirroruris(origud, mirrors, ld) for index, uri in enumerate(uris): - ret = try_mirror_url(fetch, origud, uds[index], ld, check) + ud = uds[index] + # Skip fetching it when the local url's path doesn't exist + if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path): + continue + ret = try_mirror_url(fetch, origud, ud, ld, check) if ret: return ret return None
Signature · 1 line
2.49.0
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(fetch, d, origud, mirrors, check = False): · line 1175
Quoted 7 lines · Robert Yang via lists.openembedded.org
Skip fetching from an invalid local url since it would be failed.
bitbake/lib/bb/fetch2/__init__.py +5
-1- ret = try_mirror_url(fetch, origud, uds[index], ld, check) + ud = uds[index] + # Skip fetching it when the local url's path doesn't exist + if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path): + continue + ret = try_mirror_url(fetch, origud, ud, ld, check)
Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.
Alex
Hi Alex,
Quoted 10 lines · Alexander Kanavin
bitbake/lib/bb/fetch2/__init__.py def try_mirrors(fetch, d, origud, mirrors, check = False): · line 1175
Quoted 7 lines · Robert Yang via lists.openembedded.org
Skip fetching from an invalid local url since it would be failed.
bitbake/lib/bb/fetch2/__init__.py +5
-1- ret = try_mirror_url(fetch, origud, uds[index], ld, check) + ud = uds[index] + # Skip fetching it when the local url's path doesn't exist + if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path): + continue + ret = try_mirror_url(fetch, origud, ud, ld, check)
Quoted 2 lines
Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.
There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.
// Robert
Quoted 2 lines
Alex
Quoted 8 lines · Robert Yang
Quoted 2 lines
Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.
There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.
But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?
Alex
Quoted 4 lines · Alexander Kanavin
Quoted 3 lines · Robert Yang
Quoted 2 lines
Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.
Quoted 8 lines
Quoted 5 lines
There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?
Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.
// Robert
Quoted 2 lines
Alex
Quoted 5 lines · Robert Yang
Quoted 2 lines
But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?
Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.
So the goal of the patch is remove clutter from log.do_fetch, and nothing else? This should be spelled out in the commit message, with examples of logs before the change and after.
I also wonder if this should be inside try_mirror_url (or further down), and not at the top level function. Otherwise other users of the function will have the same issue.
Alex
Quoted 10 lines · Alexander Kanavin
Quoted 5 lines · Robert Yang
Quoted 2 lines
But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?
Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.
So the goal of the patch is remove clutter from log.do_fetch, and nothing else? This should be spelled out in the commit message, with examples of logs before the change and after.
It also fixed a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source from the PREMIRROR:
Fast shallow clone failed, try to skip fast mode now.
This is because:
def clone_shallow_local(self, ud, dest, d):
"""
Shallow fetch from ud.clonedir (${DL_DIR}/git2/<gitrepo> by default):
- For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
- For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev """
progresshandler = GitProgressHandler(d)
repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest)
init_cmd = "%s init -q" % ud.basecmd
if ud.bareclone:
init_cmd += " --bare"
runfetchcmd(init_cmd, d, workdir=dest)
# Use repourl when creating a fast initial shallow clone # Prefer already existing full bare clones if available if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir):
remote = shlex.quote(repourl)
The repourl is an invalid local url from PREMIRRORS which caused the warnings.
Quoted 4 lines
I also wonder if this should be inside
try_mirror_url(or further down), and not at the top level function. Otherwise other users of the function will have the same issue.
Yes, that makes sense, I will send a V2 for it.
// Robert
Quoted 2 lines
Alex