Skip to contentSkip to conversation

fetch2: try_mirrors(): Skip invalid local url

PATCH Superseded [email protected] 2 participants 8 messages Aug 11, 2025

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover fetch2: try_mirrors(): Skip invalid local url

From: Robert Yang <[email protected]>

The following changes since commit 3b2f9c31dc17c3ba5f6a9971413afed3d450d826:

poky.yaml.in: increase required disk space to 140G (2025-08-08 23:41:11 +0100)

are available in the Git repository at:

https://github.com/robertlinux/bitbake rbt/mirror
https://github.com/robertlinux/bitbake/tree/rbt/mirror

Robert Yang (1):
fetch2: try_mirrors(): Skip invalid local url

 bitbake/lib/bb/fetch2/__init__.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.49.0
#1 lore raw

PATCH 1/1 fetch2: try_mirrors(): Skip invalid local url

From: Robert Yang <[email protected]>

Skip fetching from an invalid local url since it would be failed.

---
 bitbake/lib/bb/fetch2/__init__.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/__init__.py +5 -1
index 0ad987c596..e4c9cecd86 100644
@@ -1168,7 +1168,11 @@ def try_mirrors(fetch, d, origud, mirrors, check = False):
     uris, uds = build_mirroruris(origud, mirrors, ld)
 
     for index, uri in enumerate(uris):
-        ret = try_mirror_url(fetch, origud, uds[index], ld, check)
+        ud = uds[index]
+        # Skip fetching it when the local url's path doesn't exist
+        if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path):
+            continue
+        ret = try_mirror_url(fetch, origud, ud, ld, check)
         if ret:
             return ret
     return None
Signature · 1 line
2.49.0
#2 lore raw

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(fetch, d, origud, mirrors, check = False): · line 1175

Quoted 7 lines · Robert Yang via lists.openembedded.org

Skip fetching from an invalid local url since it would be failed.

bitbake/lib/bb/fetch2/__init__.py +5 -1
-        ret = try_mirror_url(fetch, origud, uds[index], ld, check)
+        ud = uds[index]
+        # Skip fetching it when the local url's path doesn't exist
+        if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path):
+            continue
+        ret = try_mirror_url(fetch, origud, ud, ld, check)

Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.

Alex

#3 lore raw

Hi Alex,

Quoted 10 lines · Alexander Kanavin

bitbake/lib/bb/fetch2/__init__.py def try_mirrors(fetch, d, origud, mirrors, check = False): · line 1175

Quoted 7 lines · Robert Yang via lists.openembedded.org

Skip fetching from an invalid local url since it would be failed.

bitbake/lib/bb/fetch2/__init__.py +5 -1
-        ret = try_mirror_url(fetch, origud, uds[index], ld, check)
+        ud = uds[index]
+        # Skip fetching it when the local url's path doesn't exist
+        if ud.parm.get('protocol', '') == 'file' and not os.path.exists(ud.path):
+            continue
+        ret = try_mirror_url(fetch, origud, ud, ld, check)
Quoted 2 lines

Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.

There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.

// Robert

Quoted 2 lines

Alex

#4 lore raw
Quoted 8 lines · Robert Yang
Quoted 2 lines

Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.

There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.

But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?

Alex

#5 lore raw
Quoted 4 lines · Alexander Kanavin
Quoted 3 lines · Robert Yang
Quoted 2 lines

Can you please explain the use case? I'd think failing is the correct action, so that the invalid url can be removed from configuration.

Quoted 8 lines
Quoted 5 lines

There can be multiple PREMIRRORs each PREMIRROR conatains specifics sources for each layer, each recipe will try the PREMIRRORs one
by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean.

But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?

Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.

// Robert

Quoted 2 lines

Alex

#6 lore raw
Quoted 5 lines · Robert Yang
Quoted 2 lines

But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?

Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.

So the goal of the patch is remove clutter from log.do_fetch, and nothing else? This should be spelled out in the commit message, with examples of logs before the change and after.

I also wonder if this should be inside try_mirror_url (or further down), and not at the top level function. Otherwise other users of the function will have the same issue.

Alex

#7 lore raw
Quoted 10 lines · Alexander Kanavin
Quoted 5 lines · Robert Yang
Quoted 2 lines

But why is it specific to file protocol then? If the item doesn't exist on some remote server, there would be the same issue?

Yes, they same issues to other protocols, but there isn't an easy way to check that for other protocols, so we have to let the code go on for other protocols.

So the goal of the patch is remove clutter from log.do_fetch, and nothing else? This should be spelled out in the commit message, with examples of logs before the change and after.

It also fixed a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source from the PREMIRROR:

Fast shallow clone failed, try to skip fast mode now.

This is because:
def clone_shallow_local(self, ud, dest, d):
"""
Shallow fetch from ud.clonedir (${DL_DIR}/git2/<gitrepo> by default):
- For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
- For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev """

progresshandler = GitProgressHandler(d)
repourl = self._get_repo_url(ud) bb.utils.mkdirhier(dest)
init_cmd = "%s init -q" % ud.basecmd
if ud.bareclone:
init_cmd += " --bare"
runfetchcmd(init_cmd, d, workdir=dest)
# Use repourl when creating a fast initial shallow clone # Prefer already existing full bare clones if available if not ud.shallow_skip_fast and not os.path.exists(ud.clonedir):
remote = shlex.quote(repourl)

The repourl is an invalid local url from PREMIRRORS which caused the warnings.

Quoted 4 lines

I also wonder if this should be inside try_mirror_url (or further down), and not at the top level function. Otherwise other users of the function will have the same issue.

Yes, that makes sense, I will send a V2 for it.

// Robert

Quoted 2 lines

Alex