Skip to contentSkip to conversation

fetch2/git: Fix File name too long error

PATCH In review [email protected] 2 participants 5 messages Apr 8, 2024 – Jun 22, 2024 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover fetch2/git: Fix File name too long error

From: Robert Yang <[email protected]>

The following changes since commit 110ee701b33600f14e40544bf9354774cd85c5ce:

sstatesig: fix netrc.NetrcParseError exception (2024-04-04 21:50:24 +0100)

are available in the Git repository at:

https://github.com/robertlinux/yocto rbt/long
https://github.com/robertlinux/yocto/tree/rbt/long

Robert Yang (1):
fetch2/git: Fix File name too long error

 bitbake/lib/bb/fetch2/git.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/1 fetch2/git: Fix File name too long error

From: Robert Yang <[email protected]>

There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)).

This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255.

Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:

$ bitbake mtd-utils-native -cfetch
$ ls downloads/git2
1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git.done

The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.

---
 bitbake/lib/bb/fetch2/git.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +5 -1
index c7ff769fdfe..7ed51af336d 100644
@@ -83,6 +83,7 @@ import subprocess
 import tempfile
 import bb
 import bb.progress
+import hashlib
 from contextlib import contextmanager
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
@@ -277,7 +278,10 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path))
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
Signature · 1 line
2.42.0
#2 lore raw

I believe this causes the following error:

======================================================================
FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules
self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount
self.assertEqual(expected_count, actual_count, msg='Object count %d is not the expected %d' % (actual_count, expected_count))
AssertionError: 1 != 2 : Object count 2 is not the expected 1

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio

Quoted 61 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)).

This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255.

Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:

$ bitbake mtd-utils-native -cfetch
$ ls downloads/git2
1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git.done

The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.

---
 bitbake/lib/bb/fetch2/git.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +5 -1
index c7ff769fdfe..7ed51af336d 100644
@@ -83,6 +83,7 @@ import subprocess
 import tempfile
 import bb
 import bb.progress
+import hashlib
 from contextlib import contextmanager
 from   bb.fetch2 import FetchMethod
 from   bb.fetch2 import runfetchcmd
@@ -277,7 +278,10 @@ class Git(FetchMethod):
                     ud.unresolvedrev[name] = ud.revisions[name]
                 ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path))
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
         if gitsrcname.startswith('.'):
             gitsrcname = gitsrcname[1:]
 
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064
Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw
Quoted 2 lines · Alexandre Belloni

I believe this causes the following error:

Sorry, after more thoughts, there are two issues:
* The generated mirror tarball can't be used since the local path is added to to the tarball name.

* There might be file name too long error when the local path is too long.

It doesn't make sense to add local path to the generated mirror tarball name, but causes the above two issue, use basename can fix both of them.

I will send a V2 for it.

// Robert

Quoted 77 lines

======================================================================
FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules
self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount
self.assertEqual(expected_count, actual_count, msg='Object count %d is not the expected %d' % (actual_count, expected_count))
AssertionError: 1 != 2 : Object count 2 is not the expected 1

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio

Quoted 61 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)).

This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255.

Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:

$ bitbake mtd-utils-native -cfetch
$ ls downloads/git2
1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git.done

The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.

---
  bitbake/lib/bb/fetch2/git.py | 6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +5 -1
index c7ff769fdfe..7ed51af336d 100644
@@ -83,6 +83,7 @@ import subprocess
  import tempfile
  import bb
  import bb.progress
+import hashlib
  from contextlib import contextmanager
  from   bb.fetch2 import FetchMethod
  from   bb.fetch2 import runfetchcmd
@@ -277,7 +278,10 @@ class Git(FetchMethod):
                      ud.unresolvedrev[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud, d, name)
 
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
+        if ud.proto == "file":
+            gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path))
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_'))
          if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]
 
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064
Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw
Quoted 10 lines · Robert Yang via lists.openembedded.org
Quoted 2 lines · Alexandre Belloni

I believe this causes the following error:

Sorry, after more thoughts, there are two issues:
* The generated mirror tarball can't be used since the local path is added to to the tarball name.

Quoted 6 lines

* There might be file name too long error when the local path is too long.

It doesn't make sense to add local path to the generated mirror tarball name, but causes the above two issue, use basename can fix both of them.

I will send a V2 for it.

I've sent a new version rather than V2 since the subject is changed:

[bitbake-devel] [PATCH 0/1] fetch2/git: Use basename for gitsrcname when ud.proto is file

// Robert

Quoted 85 lines

// Robert

Quoted 11 lines

======================================================================
FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
   File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount self.assertEqual(expected_count, actual_count, msg='Object count %d is not the expected %d' % (actual_count, expected_count))
AssertionError: 1 != 2 : Object count 2 is not the expected 1

Quoted 61 lines

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio

Quoted 56 lines · Robert Yang via lists.openembedded.org

From: Robert Yang <[email protected]>

There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)).

This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255.

Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:

$ bitbake mtd-utils-native -cfetch
$ ls downloads/git2
1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git
git.infradead.org.mtd-utils.git.done

The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.

---
  bitbake/lib/bb/fetch2/git.py | 6 +++++-

  1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +0 -0
index c7ff769fdfe..7ed51af336d 100644
@@ -83,6 +83,7 @@ import subprocess

  import tempfile
  import bb
  import bb.progress
+import hashlib
  from contextlib import contextmanager
  from   bb.fetch2 import FetchMethod
  from   bb.fetch2 import runfetchcmd

bitbake/lib/bb/fetch2/git.py +0 -0
@@ -277,7 +278,10 @@ class Git(FetchMethod):

                      ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name)
-        gitsrcname = '%s%s' % (ud.host.replace(':', '.'),
ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) +        if ud.proto == "file":
+            gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path))
+        else:
+            gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'):
              gitsrcname = gitsrcname[1:]

Signature · 1 line
2.42.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16370): https://lists.openembedded.org/g/bitbake-devel/message/16370
Mute This Topic: https://lists.openembedded.org/mt/105397591/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-