Skip to contentSkip to conversation

fetch2/git: Use basename for gitsrcname when ud.proto is file

PATCH In review [email protected] 5 participants 19 messages Jun 22, 2024 – May 13, 2025 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover fetch2/git: Use basename for gitsrcname when ud.proto is file

From: Robert Yang <[email protected]>

* Test info
- bitbake world --runall=fetch works well when BB_GENERATE_MIRROR_TARBALLS and BB_GIT_SHALLOW are enabled or disabled

- bitbake-selftest works well

// Robert

The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:

bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)

are available in the Git repository at:

https://github.com/robertlinux/yocto rbt/fetch_git
https://github.com/robertlinux/yocto/tree/rbt/fetch_git

Robert Yang (1):
fetch2/git: Use basename for gitsrcname when ud.proto is file

 bitbake/lib/bb/fetch2/git.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Signature · 1 line
2.45.1
#1 lore raw

PATCH 1/1 fetch2/git: Use basename for gitsrcname when ud.proto is file

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
The tarball name will be:
git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
 bitbake/lib/bb/fetch2/git.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +4 -1
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
Signature · 1 line
2.45.1
#2 lore raw
Quoted 45 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
  path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
  The tarball name will be:
  git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

  The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
 bitbake/lib/bb/fetch2/git.py | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):

                     ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)

Quoted 8 lines

-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) +        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]

I started to review this, then I started wondering why we have mirror tarballs being created for local file:// urls? Does that make sense?

Cheers,

Richard

#3 lore raw

Hi RP,

Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
  path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
  The tarball name will be:
  git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

  The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
  bitbake/lib/bb/fetch2/git.py | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):
                      ud.unresolvedrev[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud, d, name)
 
Quoted 11 lines

bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 286

Quoted 8 lines
bitbake/lib/bb/fetch2/git.py +4 -1
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]
 

I started to review this, then I started wondering why we have mirror tarballs being created for local file:// urls? Does that make sense?

This patch fixed two issues:

1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://

Create git mirror tarball for file:// doesn't make much sense, but the problem is the first issue, there would be the error when len(path) > 255 (The NAME_MAX).

// Robert

Quoted 4 lines

Cheers,

Richard

#4 lore raw
Quoted 79 lines · Robert Yang

Hi RP,

Quoted 65 lines · Richard Purdie
Quoted 63 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had
two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long
   since the path is translated to filename and added to downloads/git2/ as:
   path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
   The tarball name will be:
   git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

   The above tarball can't be re-used by do_fetch since bitbake looks for
   'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in
   the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the
the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
  bitbake/lib/bb/fetch2/git.py | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):

                      ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)

-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path) +        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))

Quoted 8 lines
Quoted 3 lines

          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I started to review this, then I started wondering why we have mirror
tarballs being created for local file:// urls? Does that make sense?

This patch fixed two issues:

Quoted 7 lines

1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://

Create git mirror tarball for file:// doesn't make much sense, but the problem
is the first issue, there would be the error when len(path) > 255 (The NAME_MAX).

I understand that. However:

a) why it is looking for a file:// url in PREMIRRORS?
b) why does it consider mirror tarballs for a file:// url?

I'm not sure it should be doing either of those things?

Cheers,

Richard

#5 lore raw

Hi RP,

Quoted 83 lines · Richard Purdie via lists.openembedded.org
Quoted 82 lines · Robert Yang

Hi RP,

Quoted 65 lines · Richard Purdie
Quoted 63 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had
two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long
   since the path is translated to filename and added to
downloads/git2/ as:
   path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
   The tarball name will be:
   git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

   The above tarball can't be re-used by do_fetch since bitbake
looks for
   'git2_github.com.lsof-org.lsof.tar.gz', but the
'path.to.local.PREMIRROR' in
   the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the
the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
   bitbake/lib/bb/fetch2/git.py | 5 ++++-
   1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -1
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):
                       ud.unresolvedrev[name] = ud.revisions[name]
                   ud.revisions[name] = self.latest_revision(ud,
 d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'),

ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path) +        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))

Quoted 8 lines
Quoted 3 lines

          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I started to review this, then I started wondering why we have mirror
tarballs being created for local file:// urls? Does that make sense?

This patch fixed two issues:

1) "File name too long error" when fetch from PREMIRROR
2) Git mirror tarball contains local path when file://

Quoted 8 lines
Quoted 4 lines

Create git mirror tarball for file:// doesn't make much sense, but the problem
is the first issue, there would be the error when len(path) > 255 (The NAME_MAX).

I understand that. However:

a) why it is looking for a file:// url in PREMIRRORS?

This is a little WRLinux specific, we release git repos with the product, and set PREMIRROS as:

PREMIRRORS:append = " \
     git://.*/.* git://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \
     gitsm://.*/.* gitsm://${LAYERDIR}/git/MIRRORNAME;protocol=file \n \
Quoted 1 line

b) why does it consider mirror tarballs for a file:// url?

I found the issue when I tried to generate shallow tarballs from the above PREMIRRORS.

// Robert.

Quoted 17 lines

I'm not sure it should be doing either of those things?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16421): https://lists.openembedded.org/g/bitbake-devel/message/16421
Mute This Topic: https://lists.openembedded.org/mt/106812686/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
In reply to [email protected]
Quoted 44 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
  path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
  The tarball name will be:
  git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

  The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
 bitbake/lib/bb/fetch2/git.py | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):

                     ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)

Quoted 8 lines


-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) +        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe. If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

Just as a note, '%s' % x is also poor code as you can just assign the value.

Cheers,

Richard

#7 lore raw
Quoted 6 lines · Richard Purdie via lists.openembedded.org

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe. If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

I just tested this and verified this is the behaviour. With a recipe that fetches a local git repo called xyzzy it currently does a clone to $DL_DIR/git2/home.rosbur01.xyzzy but this changes that path to $DL_DIR/git2/xyzzy. My fear is that this is too ambiguous for a DL_DIR path.

Ross

#8 lore raw
In reply to Richard Purdie

Hi RP and Ross,

Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
  path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
  The tarball name will be:
  git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

  The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
  bitbake/lib/bb/fetch2/git.py | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):
                      ud.unresolvedrev[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud, d, name)
 
Quoted 11 lines

bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 285

Quoted 7 lines
bitbake/lib/bb/fetch2/git.py +4 -1
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.

Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.

Quoted 2 lines

If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.

Quoted 3 lines

Just as a note, '%s' % x is also poor code as you can just assign the value.

Yes, absolutely, that's my mistake.

// Robert

Quoted 7 lines

Cheers,

Richard

#9 lore raw
Quoted 64 lines · Robert Yang

Hi RP and Ross,

Quoted 49 lines · Richard Purdie
Quoted 48 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
   path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
   The tarball name will be:
   git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

   The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
  bitbake/lib/bb/fetch2/git.py | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):

                      ud.unresolvedrev[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud, d, name)

-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) +        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)

Quoted 8 lines
Quoted 4 lines

+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.

Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.

Quoted 6 lines
Quoted 2 lines

If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.

The trouble is that this doesn't just affect local repos, the contents of DL_DIR can and will be shared as a public mirror so it is any two repos with the same name, not just two local repos.

This is partly why I still think mirrors of local file repos don't make sense.

Cheers,

Richard

#10 lore raw

Hi RP,

Quoted 68 lines · Richard Purdie
Quoted 67 lines · Robert Yang

Hi RP and Ross,

Quoted 46 lines · Richard Purdie
Quoted 45 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
   path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
   The tarball name will be:
   git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

   The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
   bitbake/lib/bb/fetch2/git.py | 5 ++++-
   1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):
                       ud.unresolvedrev[name] = ud.revisions[name]
                   ud.revisions[name] = self.latest_revision(ud, d, name)
 
Quoted 11 lines

bitbake/lib/bb/fetch2/git.py class Git(FetchMethod): · line 285

Quoted 7 lines
bitbake/lib/bb/fetch2/git.py +4 -1
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
           if gitsrcname.startswith('.'):
               gitsrcname = gitsrcname[1:]

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.

Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.

Quoted 2 lines

If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

Quoted 7 lines
Quoted 3 lines

If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.

The trouble is that this doesn't just affect local repos, the contents of DL_DIR can and will be shared as a public mirror so it is any two repos with the same name, not just two local repos.

This only affects protocol=file, without this patch, the

git://path/to/local/path/gitrepo.git:protocol=file

after do_fetch, it will be:

downloads/git2/path.to.local.path.gitrepo.git

The above name can't be used on public mirror because it can't be matched, unless different people and projects use the same local path, but this isn't the reality, for example, my local path is:

git://home/robert/to/gitrepo.git;protocol=file

When I upload it to a public mirror, it will be:
home.robert.to.gitrepo.git

For you, it might be:
git://home/rp/to/gitrepo.git;protocol=file

What your do_fetch looking for is:
home.rp.to.gitrepo.git

So it can't be fetched from public mirror, unless you use:
git://home/robert/to/gitrepo.git;protocol=file

But this isn't the reality

Quoted 3 lines

This is partly why I still think mirrors of local file repos don't make sense.

It's not only for mirror, but will cause "Filename too long error" for normal build's do_fetch when:

len(/home/robert/to/long/dir/gitrepo.git) > 256

You can simulate the error with the following command on Ubuntu 20.04 or 22.04:

$ touch $(for i in $(seq 1 30); do echo -n longpart$i; done)

File name too long

This is because the current code changes the path to filename, and len(filename) > 256.

// Robert

Quoted 5 lines

Cheers,

Richard

#11 lore raw
Quoted 11 lines · Robert Yang via lists.openembedded.org

Hi RP,

Quoted 7 lines · Richard Purdie

On Tue, 2024-08-06 at 21:42 +0800, Robert Yang wrote:

The trouble is that this doesn't just affect local repos, the contents
of DL_DIR can and will be shared as a public mirror so it is any two
repos with the same name, not just two local repos.

Quoted 7 lines

This only affects protocol=file, without this patch, the

git://path/to/local/path/gitrepo.git:protocol=file

after do_fetch, it will be:

downloads/git2/path.to.local.path.gitrepo.git

Currently we don't support mirroring of file protocol git urls as they don't really make sense.

Quoted 5 lines

The above name can't be used on public mirror because it can't be matched, unless different people and projects use the same local path, but this isn't the reality, for example, my local path is:

git://home/robert/to/gitrepo.git;protocol=file

Correct, but the path is uniquely identified.

Quoted 6 lines

When I upload it to a public mirror, it will be:
home.robert.to.gitrepo.git

For you, it might be:
git://home/rp/to/gitrepo.git;protocol=file

Quoted 7 lines

What your do_fetch looking for is:
home.rp.to.gitrepo.git

So it can't be fetched from public mirror, unless you use:
git://home/robert/to/gitrepo.git;protocol=file

But this isn't the reality

Agreed, they simply don't work today and are not supported.

Quoted 9 lines
Quoted 3 lines

This is partly why I still think mirrors of local file repos don't make
sense.

It's not only for mirror, but will cause "Filename too long error" for normal build's do_fetch when:

len(/home/robert/to/long/dir/gitrepo.git) > 256

Quoted 8 lines

You can simulate the error with the following command on Ubuntu 20.04 or 22.04:

$ touch $(for i in $(seq 1 30); do echo -n longpart$i; done)

File name too long

This is because the current code changes the path to filename, and len(filename) > 256.

Sure, but this isn't a good reason to adopt an unsafe naming scheme for something we don't currently support and don't really want to add support for.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw
Quoted 1 line · Robert Yang via lists.openembedded.org

Hi RP and Ross,

Sorry, correct Ross' email.

// Robert

Quoted 96 lines · Richard Purdie
Quoted 50 lines · Robert Yang via lists.openembedded.org
Quoted 49 lines

From: Robert Yang <[email protected]>

The previous code added local path to generated mirror tarball name which had two potential issues when fetch from local PREMIRROR, for exapmle:

Add github.com.lsof-org.lsof to /path/to/local/PREMIRROR/

$ bitbake lsof -cfetch

* There might be "Fix File name too long error" when the local path is long since the path is translated to filename and added to downloads/git2/ as:
   path.to.local.PREMIRROR.github.com.lsof-org.lsof

* When enable BB_GENERATE_MIRROR_TARBALLS:
   The tarball name will be:
   git2_path.to.local.PREMIRROR.github.com.lsof-org.lsof.tar.gz

   The above tarball can't be re-used by do_fetch since bitbake looks for 'git2_github.com.lsof-org.lsof.tar.gz', but the 'path.to.local.PREMIRROR' in the file name breaks the match.

Use os.path.basename(ud.path) as gitsrcname when ud.proto is file can fix the the above two issues, now the file name in downloads/git2/ is:
github.com.lsof-org.lsof

And the generated tarball is:
git2_github.com.lsof-org.lsof.tar.gz

Or when BB_GIT_SHALLOW is enabled:
gitshallow_github.com.lsof-org.lsof_2e4c7a1-1_master.tar.gz

And the tarballs can be re-used by do_fetch.

---
  bitbake/lib/bb/fetch2/git.py | 5 ++++-

  1 file changed, 4 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdf..89d3268f7f 100644
@@ -277,7 +277,10 @@ class Git(FetchMethod):

                      ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'),
ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) +        if ud.proto == "file":
+            gitsrcname = '%s' % os.path.basename(ud.path)
+        else:

Quoted 8 lines
Quoted 4 lines

+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

I did look at this patch again. I'm still not convinced about the original use case but ignoring that, the patch itself isn't too safe.

Thanks for looking into this, the main problem is that we need a way to fix the filename too long error as described in the commit message above.

Quoted 2 lines

If you have two repositories with the same name, this can break since basename() on the URI is not a unique value.

If the two repos are identical, then it isn't a problem since bitbake can handle them correctly, if the two local repos with the same name but have different contents, I think that the user should fix their repos.

Quoted 3 lines

Just as a note, '%s' % x is also poor code as you can just assign the value.

Yes, absolutely, that's my mistake.

// Robert

Quoted 7 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16447): https://lists.openembedded.org/g/bitbake-devel/message/16447
Mute This Topic: https://lists.openembedded.org/mt/106812686/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#14 lore raw
In reply to [email protected]

Hello,

Any comments on this patch, please?

// Robert

Quoted 35 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

* Test info
- bitbake world --runall=fetch works well when BB_GENERATE_MIRROR_TARBALLS and BB_GIT_SHALLOW are enabled or disabled

- bitbake-selftest works well

// Robert

The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:

bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)

are available in the Git repository at:

https://github.com/robertlinux/yocto rbt/fetch_git
https://github.com/robertlinux/yocto/tree/rbt/fetch_git

Robert Yang (1):
fetch2/git: Use basename for gitsrcname when ud.proto is file

  bitbake/lib/bb/fetch2/git.py | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16371): https://lists.openembedded.org/g/bitbake-devel/message/16371
Mute This Topic: https://lists.openembedded.org/mt/106812685/7304958
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#15 lore raw

I submitted a bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856

the bitbake on my env generated a git shallow tar for glibc, but with local mirror path in filename. and linked to it as a symbollink located in git2.

when unpacking, the bitbake can't find the shallow with official shallow (which file name should contail sourceware.xxx), can can't find bare repo (which linked to the shallow file).

I think the latest patch for git shallow result it:
fetch2/git: Add support for fast initial shallow fetch 457288b2fda86fd00cdcaefac616129b0029e1f9

the patch author said that the bitbake don't want to generate shallow for local mirror, but why I got it? If so, maybe we should set shallow_skip_fast as True if protocol is "file":

-        ud.shallow_skip_fast = False
+        ud.shallow_skip_fast = (ud.proto == "file")

#16 lore raw

Hi

Definition of ud.shallow_skip_fast = False is:
Prefer an initial shallow clone over an initial full bare clone, while still utilizing any already existing full bare clones.

Before BitBake 2.12:
- Create a bare clone of the repo
- Create a shallow tarball of that bare clone

Beginning with BitBake 2.12
- Do not create a bare clone of the repo (but use it when it already exists)
- Instead, use origin directly to create the tarball
- Only in case of failure, switch to the old behaviour by setting ud.shallow_skip_fast = True and retry

General example using raw GIT commands with the Linux-CIP repo git://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git:
- depth=1 (shallow) data transfer for Linux-CIP via git/ssh is about 148 MiB (default branch)
- bare mirror clone data transfer is about 3.91 GiB
=> So the bare clone needs 27 times more data transfer than one shallow clone.
=> The shallow clone size is similar to the size of usual kernel tarballs.
=> Fast shallow archieves that small clone about 148 MiB.

To fix that issue here:
- EITHER adapt the mentioned patch from Robert "Use basename for gitsrcname when ud.proto is file"
- OR set ud.shallow_skip_fast = True in case of PREMIRROR/file proto (are there other cases, too?)

Maybe it was only a bit of luck that the PREMIRROR has worked, because BitBake has always created a bare clone, even when using shallow.

Regards

Stefan

Quoted 32 lines · Yi Qingliang via lists.openembedded.org

I submitted a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15856

the bitbake on my env generated a git shallow tar for glibc, but with local mirror path in filename.
and linked to it as a symbollink located in git2.

when unpacking, the bitbake can't find the shallow with official shallow (which file name should contail sourceware.xxx), can can't find bare repo (which linked to the shallow file).

I think the latest patch for git shallow result it:
fetch2/git: Add support for fast initial shallow fetch 457288b2fda86fd00cdcaefac616129b0029e1f9

the patch author said that the bitbake don't want to generate shallow for local mirror, but why I got it? If so, maybe we should set shallow_skip_fast as True if protocol is "file":

-        ud.shallow_skip_fast = False
+        ud.shallow_skip_fast = (ud.proto == "file")

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#17600): https://lists.openembedded.org/g/bitbake-
devel/message/17600
Mute This Topic: https://lists.openembedded.org/mt/106812685/7767788
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-
devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
Stefan Koch
Siemens AG
www.siemens.com
#17 lore raw

upload my glic recipe as attachement.

my download directory is "/mnt/datum/downloads/".
premirror of glibc is: /mnt/datum/repositories/sourceware.org/git/glibc.git

the file system structure generated by bitbake in download is:
lrwxrwxrwx 1 ll ll      120 May  7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll        6 May  7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.done
-rw-r--r-- 1 ll ll        0 May  7 17:09 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.lock
-rw-r--r-- 1 ll ll 24413461 May  7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz
-rw-r--r-- 1 ll ll        0 May  7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz.done

when unpacking, the bitbake expect a bare repo at /mnt/datum/downloads/git2/sourceware.org.git.glibc.git, but indeed, it is a symbollink to gitshallow file. and then bitbake expect a git shallow file with name "gitshallow_sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz", also no luck.

so bitbake report error finally.

#18 lore raw

Tthe patch Robert provided not work, shallow file's name changed to gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz , which still result unpacking error.

no bare repo, and shallow file's name is not expected (no sourceware.org).

#19 lore raw

indeed, what's I want is to compile program, which is expected running on OS with lower version glibc.
now, I'm using my older glibc to compile library, and use official latest one to compile my application,
the library can linked by user directly, at the same time the application is provided to user with a full root file system (running with chroot).

I checked the static link glibc solution, which is not recommended.
and my library link to external 3rd binary library, so I can't use musl libc solution.

for library:
now I have ported the 2.17 glibc to the latest bitbake/oe-core, static linking gcc/c++ related library, dynamic linking glibc, to generate my library.

for application:
The Appimage looks like is too complicated, and need to check every so on all OS, to simplify it, I selected chroot+root filesystem (with mounting some directory, dev, var, proc, ...) as solution for my application.

maybe there is a better solution?