fetch2: try_mirror_url(): Skip invalid local url
v3 cover fetch2: try_mirror_url(): Skip invalid local url
From: Robert Yang <[email protected]>
* V3
- Check both repo and rep.git since git works with both of them.
* V2
- Move the code to try_mirror_url() as Alex suggested
- Add more commit messages.
* V1
- Initial version
// Robert
The following changes since commit 3b2f9c31dc17c3ba5f6a9971413afed3d450d826:
poky.yaml.in: increase required disk space to 140G (2025-08-08 23:41:11 +0100)
are available in the Git repository at:
https://github.com/robertlinux/bitbake rbt/mirror
https://github.com/robertlinux/bitbake/tree/rbt/mirror
Robert Yang (1):
fetch2: try_mirror_url(): Skip invalid local url
bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Signature · 1 line
2.49.0
PATCH v3 1/1 fetch2: try_mirror_url(): Skip invalid local url
From: Robert Yang <[email protected]>
There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for each layer, each recipe will try the PREMIRRORs one by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean, and tt also can fix a warning when BB_GIT_SHALLOW and is enabled and failed to fetch the source from the PREMIRROR:
WARNING: Fast shallow clone failed, try to skip fast mode now.
--- bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
bitbake/lib/bb/fetch2/__init__.py +16 -0
@@ -1067,6 +1067,22 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): # Return of None or a value means we're finished # False means try another url + # Skip fetching it when the local url's path doesn't exist + if ud.parm.get('protocol', '') == 'file': + found = False + check_paths = [ud.path] + # Git works with or without '.git' suffix + if ud.path.endswith('.git'): + check_paths.append(ud.path[:-4]) + else: + check_paths.append('%s.git' % ud.path) + for check_path in check_paths: + if os.path.exists(check_path): + found = True + break + if not found: + return False + if ud.lockfile and ud.lockfile != origud.lockfile: lf = bb.utils.lockfile(ud.lockfile)
Signature · 1 line
2.49.0
Quoted 36 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for each layer, each recipe will try the PREMIRRORs one by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean, and tt also can fix a warning when
BB_GIT_SHALLOWand is enabled and failed to fetch the source from the PREMIRROR:WARNING: Fast shallow clone failed, try to skip fast mode now.
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++1 file changed, 16 insertions(+)bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -1067,6 +1067,22 @@ def try_mirror_url(fetch, origud, ud, ld,check = False):
# Return of None or a value means we're finished # False means try another url
+ # Skip fetching it when the local url's path doesn't exist + ifud.parm.get('protocol', '')== 'file':
+ found = False
+check_paths= [ud.path]
+ # Git works with or without '.git' suffix
+ ifud.path.endswith('.git'):
+check_paths.append(ud.path[:-4])
+ else:
+check_paths.append('%s.git' % ud.path)
+ forcheck_pathincheck_paths:
Quoted 8 lines
+ if
os.path.exists(check_path):
+ found = True
+ break
+ if not found:
+ return False
+
if ud.lockfile and ud.lockfile != origud.lockfile:
lf =bb.utils.lockfile(ud.lockfile)
Absolutely not.
This is adding git specific code into the main fetcher functions and the whole idea is this kind of thing is meant to be abstracted.
The fetcher logging is a mess and we should probably be improving that up rather than trying to add special cases to the code.
Cheers,
Richard
Hi RP,
Quoted 49 lines · Richard Purdie
Quoted 36 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
There can be multiple PREMIRRORs each PREMIRROR contains specifics sources for each layer, each recipe will try the PREMIRRORs one by one until succeed, but the trying would be failed if the PREMIRROR doesn't contain the required sources, so return it immediately to make log.do_fetch clean, and tt also can fix a warning when
BB_GIT_SHALLOWand is enabled and failed to fetch the source from the PREMIRROR:WARNING: Fast shallow clone failed, try to skip fast mode now.
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/__init__.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)bitbake/lib/bb/fetch2/__init__.py +0
-0@@ -1067,6 +1067,22 @@ def try_mirror_url(fetch, origud, ud, ld,check = False):
# Return of None or a value means we're finished # False means try another url+ # Skip fetching it when the local url's path doesn't exist + if
ud.parm.get('protocol', '')== 'file':
+ found = False
+check_paths= [ud.path]
+ # Git works with or without '.git' suffix
+ ifud.path.endswith('.git'):
+check_paths.append(ud.path[:-4])
+ else:
+check_paths.append('%s.git' % ud.path)
+ forcheck_pathincheck_paths:Quoted 8 lines
+ if
os.path.exists(check_path):
+ found = True
+ break
+ if not found:
+ return False
+
if ud.lockfile and ud.lockfile != origud.lockfile:
lf =bb.utils.lockfile(ud.lockfile)Absolutely not.
Quoted 4 lines
This is adding git specific code into the main fetcher functions and the whole idea is this kind of thing is meant to be abstracted.
The fetcher logging is a mess and we should probably be improving that
The problem is the following warning:
WARNING: Fast shallow clone failed, try to skip fast mode now.
So how about move the code into git.py?
Add the code to __init__.py can also reduce the logs in log.do_fetch, so I added it here.
// Robert
Quoted 5 lines
up rather than trying to add special cases to the code.
Cheers,
Richard