fetch2/git: Use basename for gitsrcname when ud.proto is file
cover fetch2/git: Use basename for gitsrcname when ud.proto is file
From: Robert Yang <[email protected]>
* Test info
- bitbake world --runall=fetch works well when BB_GENERATE_MIRROR_TARBALLS and BB_GIT_SHALLOW are enabled or disabled
- bitbake-selftest works well
// Robert
The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:
bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)
are available in the Git repository at:
https://github.com/robertlinux/yocto rbt/fetch_git
https://github.com/robertlinux/yocto/tree/rbt/fetch_git
Robert Yang (1):
fetch2/git: Use basename for gitsrcname when ud.proto is file
bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Signature · 1 line
2.45.1
PATCH 1/1 fetch2/git: Use basename for gitsrcname when ud.proto is file
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch
* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof
* When enable BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz
The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.
Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof
And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz
Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz
And the tarballs can be re-used by do_fetch.
--- bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +4 -1
@@ -277,7 +277,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s' % os.path.basename(ud.path) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]
Signature · 1 line
2.45.1
Quoted 45 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py | 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] =
self.latest_revision(ud, d, name)
Quoted 8 lines
- gitsrcname = '%s%s' % (
ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)
+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) ifgitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]
I started to review this, then I started wondering why we have mirror tarballs being created for local file:// urls? Does that make sense?
Cheers,
Richard
Hi RP,
Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)
Quoted 11 lines
bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 286
Quoted 8 lines
bitbake/lib/bb/fetch2/git.py +4
-1- gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s' % os.path.basename(ud.path) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]I started to review this, then I started wondering why we have mirror tarballs being created for local file:// urls? Does that make sense?
This patch fixed two issues:
1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://
Create git mirror tarball for file:// doesn't make much sense, but the problem is the first issue, there would be the error when len(path) > 255 (The NAME_MAX).
// Robert
Quoted 4 lines
Cheers,
Richard
Quoted 79 lines · Robert Yang
Hi RP,
Quoted 65 lines · Richard Purdie
Quoted 63 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had
two potential issues when fetch from local PREMIRROR, for exapmle:Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long
since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for
'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in
the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the
the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py | 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] =
self.latest_revision(ud, d, name)
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+ if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))Quoted 8 lines
Quoted 3 lines
if
gitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]I started to review this, then I started wondering why we have mirror
tarballs being created for local file:// urls? Does that make sense?This patch fixed two issues:
Quoted 7 lines
1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://Create git mirror tarball for file:// doesn't make much sense, but the problem
is the first issue, there would be the error when len(path) > 255 (TheNAME_MAX).
I understand that. However:
a) why it is looking for a file:// url in PREMIRRORS?
b) why does it consider mirror tarballs for a file:// url?
I'm not sure it should be doing either of those things?
Cheers,
Richard
Hi RP,
Quoted 83 lines · Richard Purdie via lists.openembedded.org
Quoted 82 lines · Robert Yang
Hi RP,
Quoted 65 lines · Richard Purdie
Quoted 63 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had
two potential issues when fetch from local PREMIRROR, for exapmle:Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long
since the path is translated to filename and added to
downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake
looks for
'git2_github.com.lsof-org.lsof.tar.gz', but the
'path.to.local.PREMIRROR' in
the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the
the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-1@@ -277,7 +277,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'),
ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+ if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))Quoted 8 lines
Quoted 3 lines
if
gitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]I started to review this, then I started wondering why we have mirror
tarballs being created for local file:// urls? Does that make sense?This patch fixed two issues:
1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://
Quoted 8 lines
Quoted 4 lines
Create git mirror tarball for file:// doesn't make much sense, but the problem
is the first issue, there would be the error when len(path) > 255 (TheNAME_MAX).I understand that. However:
a) why it is looking for a file:// url in PREMIRRORS?
This is a little WRLinux specific, we release git repos with the product, and set PREMIRROS as:
PREMIRRORS:append = " \
git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \
gitsm://.*/.* gitsm://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \Quoted 1 line
b) why does it consider mirror tarballs for a file:// url?
I found the issue when I tried to generate shallow tarballs from the above PREMIRRORS.
// Robert.
Quoted 17 lines
I'm not sure it should be doing either of those things?
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16421): https://lists.openembedded.org/g/bitbake-devel/message/16421
Mute This Topic: https://lists.openembedded.org/mt/106812686/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 44 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py | 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] =
self.latest_revision(ud, d, name)
Quoted 8 lines
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)
+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) ifgitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]
I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe. If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.
Just as a note, '%s' % x is also poor code as you can just assign the value.
Cheers,
Richard
Quoted 6 lines · Richard Purdie via lists.openembedded.org
I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe. If you have two repositories with the same name, this can break since
basename()on the URI is not a unique value.
I just tested this and verified this is the behaviour. With a recipe that fetches a local git repo called xyzzy it currently does a clone to $DL_DIR/git2/home.rosbur01.xyzzy but this changes that path to $DL_DIR/git2/xyzzy. My fear is that this is too ambiguous for a DL_DIR path.
Ross
Hi RP and Ross,
Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)
Quoted 11 lines
bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 285
Quoted 7 lines
bitbake/lib/bb/fetch2/git.py +4
-1- gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s' % os.path.basename(ud.path) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.
Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.
Quoted 2 lines
If you have two repositories with the same name, this can break since
basename()on the URI is not a unique value.
If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.
Quoted 3 lines
Just as a note, '%s' % x is also poor code as you can just assign the value.
Yes, absolutely, that's my mistake.
// Robert
Quoted 7 lines
Cheers,
Richard
Quoted 64 lines · Robert Yang
Hi RP and Ross,
Quoted 49 lines · Richard Purdie
Quoted 48 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py | 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name]
ud.revisions[name] =self.latest_revision(ud, d, name)
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)Quoted 8 lines
Quoted 4 lines
+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) ifgitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.
Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.
Quoted 6 lines
Quoted 2 lines
If you have two repositories with the same name, this can break since
basename()on the URI is not a unique value.If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.
The trouble is that this doesn't just affect local repos, the contents of DL_DIR can and will be shared as a public mirror so it is any two repos with the same name, not just two local repos.
This is partly why I still think mirrors of local file repos don't make sense.
Cheers,
Richard
Hi RP,
Quoted 68 lines · Richard Purdie
Quoted 67 lines · Robert Yang
Hi RP and Ross,
Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)Quoted 11 lines
bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 285
Quoted 7 lines
bitbake/lib/bb/fetch2/git.py +4
-1- gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s' % os.path.basename(ud.path) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.
Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.
Quoted 2 lines
If you have two repositories with the same name, this can break since
basename()on the URI is not a unique value.
Quoted 7 lines
Quoted 3 lines
If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.
The trouble is that this doesn't just affect local repos, the contents of
DL_DIRcan and will be shared as a public mirror so it is any two repos with the same name, not just two local repos.
This only affects protocol=file, without this patch, the
git://path/to/local/path/gitrepo.git:protocol=file
after do_fetch, it will be:
downloads/git2/path.to.local.path.gitrepo.git
The above name can't be used on public mirror because it can't be matched, unless different people and projects use the same local path, but this isn't the reality, for example, my local path is:
git://home/robert/to/gitrepo.git;protocol=file
When I upload it to a public mirror, it will be:
home.robert.to.gitrepo.git
For you, it might be:git://home/rp/to/gitrepo.git;protocol=file
What your do_fetch looking for is:
home.rp.to.gitrepo.git
So it can't be fetched from public mirror, unless you use:git://home/robert/to/gitrepo.git;protocol=file
But this isn't the reality
Quoted 3 lines
This is partly why I still think mirrors of local file repos don't make sense.
It's not only for mirror, but will cause "Filename too long error" for normal build's do_fetch when:
len(/home/robert/to/long/dir/gitrepo.git) > 256
You can simulate the error with the following command on Ubuntu 20.04 or 22.04:
$ touch $(for i in $(seq 1 30); do echo -n longpart$i; done)
File name too long
This is because the current code changes the path to filename, and len(filename) > 256.
// Robert
Quoted 5 lines
Cheers,
Richard
Quoted 11 lines · Robert Yang via lists.openembedded.org
Hi RP,
Quoted 7 lines · Richard Purdie
On Tue, 2024-08-06 at 21:42 +0800, Robert Yang wrote:
The trouble is that this doesn't just affect local repos, the contents
ofDL_DIRcan and will be shared as a public mirror so it is any two
repos with the same name, not just two local repos.
Quoted 7 lines
This only affects protocol=file, without this patch, the
git://path/to/local/path/gitrepo.git:protocol=fileafter
do_fetch, it will be:downloads/git2/path.to.local.path.gitrepo.git
Currently we don't support mirroring of file protocol git urls as they don't really make sense.
Quoted 5 lines
The above name can't be used on public mirror because it can't be matched, unless different people and projects use the same local path, but this isn't the reality, for example, my local path is:
git://home/robert/to/gitrepo.git;protocol=file
Correct, but the path is uniquely identified.
Quoted 6 lines
When I upload it to a public mirror, it will be:
home.robert.to.gitrepo.gitFor you, it might be:
git://home/rp/to/gitrepo.git;protocol=file
Quoted 7 lines
What your
do_fetchlooking for is:
home.rp.to.gitrepo.gitSo it can't be fetched from public mirror, unless you use:
git://home/robert/to/gitrepo.git;protocol=fileBut this isn't the reality
Agreed, they simply don't work today and are not supported.
Quoted 9 lines
Quoted 3 lines
This is partly why I still think mirrors of local file repos don't make
sense.It's not only for mirror, but will cause "Filename too long error" for normal build's
do_fetchwhen:len(/home/robert/to/long/dir/gitrepo.git) > 256
Quoted 8 lines
You can simulate the error with the following command on Ubuntu 20.04 or 22.04:
$ touch $(for i in $(seq 1 30); do echo -n longpart$i; done)File name too long
This is because the current code changes the path to filename, and len(filename) > 256.
Sure, but this isn't a good reason to adopt an unsafe naming scheme for something we don't currently support and don't really want to add support for.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Robert Yang via lists.openembedded.org
Hi RP and Ross,
Sorry, correct Ross' email.
// Robert
Quoted 96 lines · Richard Purdie
Quoted 50 lines · Robert Yang via lists.openembedded.org
Quoted 49 lines
From: Robert Yang <[email protected]>
The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:
Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/
$ bitbake lsof -cfetch* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof* When enable
BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gzThe above tarball can't be re-used by
do_fetchsince bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.Use
os.path.basename(ud.path)as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsofAnd the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gzOr when
BB_GIT_SHALLOWis enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gzAnd the tarballs can be re-used by
do_fetch.Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py | 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +277,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] =
self.latest_revision(ud, d, name)
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file":
+ gitsrcname = '%s' %os.path.basename(ud.path)
+ else:Quoted 8 lines
Quoted 4 lines
+ gitsrcname = '%s%s' % (
ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) ifgitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.
Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.
Quoted 2 lines
If you have two repositories with the same name, this can break since
basename()on the URI is not a unique value.If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.
Quoted 3 lines
Just as a note, '%s' % x is also poor code as you can just assign the value.
Yes, absolutely, that's my mistake.
// Robert
Quoted 7 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16447): https://lists.openembedded.org/g/bitbake-devel/message/16447
Mute This Topic: https://lists.openembedded.org/mt/106812686/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hello,
Any comments on this patch, please?
// Robert
Quoted 35 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
* Test info
- bitbake world --runall=fetch works well whenBB_GENERATE_MIRROR_TARBALLSandBB_GIT_SHALLOWare enabled or disabled- bitbake-selftest works well
// Robert
The following changes since commit
5d88faa0f35f0205c1475893d8589d1e6533dcc0:bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in
get_unihashes(2024-06-18 08:45:22 +0100)are available in the Git repository at:
https://github.com/robertlinux/yocto rbt/fetch_git
https://github.com/robertlinux/yocto/tree/rbt/fetch_gitRobert Yang (1):
fetch2/git: Use basename for gitsrcname when ud.proto is filebitbake/lib/bb/fetch2/git.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16371): https://lists.openembedded.org/g/bitbake-devel/message/16371
Mute This Topic: https://lists.openembedded.org/mt/106812685/7304958
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
I submitted a bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856
the bitbake on my env generated a git shallow tar for glibc, but with local mirror path in filename. and linked to it as a symbollink located in git2.
when unpacking, the bitbake can't find the shallow with official shallow (which file name should contail sourceware.xxx), can can't find bare repo (which linked to the shallow file).
I think the latest patch for git shallow result it:
fetch2/git: Add support for fast initial shallow fetch 457288b2fda86fd00cdcaefac616129b0029e1f9
the patch author said that the bitbake don't want to generate shallow for local mirror, but why I got it? If so, maybe we should set shallow_skip_fast as True if protocol is "file":
- ud.shallow_skip_fast = False
+ ud.shallow_skip_fast = (ud.proto == "file")
Hi
Definition of ud.shallow_skip_fast = False is:
Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones.
Before BitBake 2.12:
- Create a bare clone of the repo
- Create a shallow tarball of that bare clone
Beginning with BitBake 2.12
- Do not create a bare clone of the repo (but use it when it already exists)
- Instead, use origin directly to create the tarball
- Only in case of failure, switch to the old behaviour by setting ud.shallow_skip_fast = True and retry
General example using raw GIT commands with the Linux-CIP repo git://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git:
- depth=1 (shallow) data transfer for Linux-CIP via git/ssh is about 148 MiB (default branch)
- bare mirror clone data transfer is about 3.91 GiB
=> So the bare clone needs 27 times more data transfer than one shallow clone.
=> The shallow clone size is similar to the size of usual kernel tarballs.
=> Fast shallow archieves that small clone about 148 MiB.
To fix that issue here:
- EITHER adapt the mentioned patch from Robert "Use basename for gitsrcname when ud.proto is file"
- OR set ud.shallow_skip_fast = True in case of PREMIRROR/file proto (are there other cases, too?)
Maybe it was only a bit of luck that the PREMIRROR has worked, because BitBake has always created a bare clone, even when using shallow.
Regards
Stefan
Quoted 32 lines · Yi Qingliang via lists.openembedded.org
I submitted a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856the bitbake on my env generated a git shallow tar for glibc, but with local mirror path in filename.
and linked to it as a symbollink located in git2.when unpacking, the bitbake can't find the shallow with official shallow (which file name should contail sourceware.xxx), can can't find bare repo (which linked to the shallow file).
I think the latest patch for git shallow result it:
fetch2/git: Add support for fast initial shallow fetch457288b2fda86fd00cdcaefac616129b0029e1f9
the patch author said that the bitbake don't want to generate shallow for local mirror, but why I got it? If so, maybe we should setshallow_skip_fastas True if protocol is "file":- ud.shallow_skip_fast = False
+ ud.shallow_skip_fast = (ud.proto == "file")-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#17600): https://lists.openembedded.org/g/bitbake-
devel/message/17600
Mute This Topic: https://lists.openembedded.org/mt/106812685/7767788
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-
devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Stefan Koch Siemens AG www.siemens.com
upload my glic recipe as attachement.
my download directory is "/mnt/datum/downloads/".
premirror of glibc is: /mnt/datum/repositories/sourceware.org/git/glibc.git
the file system structure generated by bitbake in download is:
lrwxrwxrwx 1 ll ll 120 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.done
-rw-r--r-- 1 ll ll 0 May 7 17:09 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.lock
-rw-r--r-- 1 ll ll 24413461 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz
-rw-r--r-- 1 ll ll 0 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz.done
when unpacking, the bitbake expect a bare repo at /mnt/datum/downloads/git2/sourceware.org.git.glibc.git, but indeed, it is a symbollink to gitshallow file. and then bitbake expect a git shallow file with name "gitshallow_sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz", also no luck.
so bitbake report error finally.
Tthe patch Robert provided not work, shallow file's name changed to gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz , which still result unpacking error.
no bare repo, and shallow file's name is not expected (no sourceware.org).
indeed, what's I want is to compile program, which is expected running on OS with lower version glibc.
now, I'm using my older glibc to compile library, and use official latest one to compile my application,
the library can linked by user directly, at the same time the application is provided to user with a full root file system (running with chroot).
I checked the static link glibc solution, which is not recommended.
and my library link to external 3rd binary library, so I can't use musl libc solution.
for library:
now I have ported the 2.17 glibc to the latest bitbake/oe-core, static linking gcc/c++ related library, dynamic linking glibc, to generate my library.
for application:
The Appimage looks like is too complicated, and need to check every so on all OS, to simplify it, I selected chroot+root filesystem (with mounting some directory, dev, var, proc, ...) as solution for my application.
maybe there is a better solution?