fetch2/git: Fix File name too long error
cover fetch2/git: Fix File name too long error
From: Robert Yang <[email protected]>
The following changes since commit 110ee701b33600f14e40544bf9354774cd85c5ce:
sstatesig: fix netrc.NetrcParseError exception (2024-04-04 21:50:24 +0100)
are available in the Git repository at:
https://github.com/robertlinux/yocto rbt/long
https://github.com/robertlinux/yocto/tree/rbt/long
Robert Yang (1):
fetch2/git: Fix File name too long error
bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
Signature · 1 line
2.42.0
PATCH 1/1 fetch2/git: Fix File name too long error
From: Robert Yang <[email protected]>
There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The NAME_MAX)).
This is becuase ud.path.replace() changes the path to filename, and the errors when happens when len(path) > 255.
Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:
$ bitbake mtd-utils-native -cfetch $ ls downloads/git2 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git.done
The 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.
--- bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/git.py +5 -1
@@ -83,6 +83,7 @@ import subprocess import tempfile import bb import bb.progress +import hashlib from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd
@@ -277,7 +278,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]
Signature · 1 line
2.42.0
I believe this causes the following error:
======================================================================
FAIL: test_shallow_submodules (bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, in test_shallow_submodules
self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount
self.assertEqual(expected_count, actual_count, msg='Object count %d is not the expected %d' % (actual_count, expected_count))
AssertionError: 1 != 2 : Object count 2 is not the expected 1
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio
Quoted 61 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The
NAME_MAX)).This is becuase
ud.path.replace()changes the path to filename, and the errors when happens when len(path) > 255.Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:
$ bitbake mtd-utils-native -cfetch $ ls downloads/git2 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git.doneThe 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +5
-1@@ -83,6 +83,7 @@ import subprocess import tempfile import bb import bb.progress +import hashlib from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd@@ -277,7 +278,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]Signature · 11 lines
2.42.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064 Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 2 lines · Alexandre Belloni
I believe this causes the following error:
Sorry, after more thoughts, there are two issues:
* The generated mirror tarball can't be used since the local path is added to to the tarball name.
* There might be file name too long error when the local path is too long.
It doesn't make sense to add local path to the generated mirror tarball name, but causes the above two issue, use basename can fix both of them.
I will send a V2 for it.
// Robert
Quoted 77 lines
======================================================================
FAIL:test_shallow_submodules(bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, intest_shallow_submodules
self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount
self.assertEqual(expected_count,actual_count, msg='Object count%dis not the expected%d' % (actual_count,expected_count))
AssertionError: 1 != 2 : Object count2is not the expected1https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio
Quoted 61 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The
NAME_MAX)).This is becuase
ud.path.replace()changes the path to filename, and the errors when happens when len(path) > 255.Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:
$ bitbake mtd-utils-native -cfetch $ ls downloads/git2 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git.doneThe 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.
Signed-off-by Robert Yang <[email protected]>--- bitbake/lib/bb/fetch2/git.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +5
-1@@ -83,6 +83,7 @@ import subprocess import tempfile import bb import bb.progress +import hashlib from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd@@ -277,7 +278,10 @@ class Git(FetchMethod): ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] = self.latest_revision(ud, d, name) - gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file": + gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(), os.path.basename(ud.path)) + else: + gitsrcname = '%s%s' % (ud.host.replace(':', '.'), ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) if gitsrcname.startswith('.'): gitsrcname = gitsrcname[1:]Signature · 11 lines
2.42.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16064): https://lists.openembedded.org/g/bitbake-devel/message/16064 Mute This Topic: https://lists.openembedded.org/mt/105397591/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Quoted 10 lines · Robert Yang via lists.openembedded.org
Quoted 2 lines · Alexandre Belloni
I believe this causes the following error:
Sorry, after more thoughts, there are two issues:
* The generated mirror tarball can't be used since the local path is added to to the tarball name.
Quoted 6 lines
* There might be file name too long error when the local path is too long.
It doesn't make sense to add local path to the generated mirror tarball name, but causes the above two issue, use basename can fix both of them.
I will send a V2 for it.
I've sent a new version rather than V2 since the subject is changed:
[bitbake-devel] [PATCH 0/1] fetch2/git: Use basename for gitsrcname when ud.proto is file
// Robert
Quoted 85 lines
// Robert
Quoted 11 lines
======================================================================
FAIL:test_shallow_submodules(bb.tests.fetch.GitShallowTest.test_shallow_submodules)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1946, intest_shallow_submodulesself.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/tests/fetch.py", line 1728, in assertRevCount self.assertEqual(expected_count,actual_count, msg='Object count%dis not the expected%d' % (actual_count,expected_count))
AssertionError: 1 != 2 : Object count2is not the expected1Quoted 61 lines
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6634/steps/11/logs/stdio
Quoted 56 lines · Robert Yang via lists.openembedded.org
From: Robert Yang <[email protected]>
There are "File name too long" errors when local PREMIRROR/MIRROR is in a deep directory (len(path) > 255 (The
NAME_MAX)).This is becuase
ud.path.replace()changes the path to filename, and the errors when happens when len(path) > 255.Use hashlib.md5(ud.path.encode('utf-8')).hexdigest() can fix the problem and avoid potential file name conflicts, now the files are like:
$ bitbake mtd-utils-native -cfetch $ ls downloads/git2 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git -> /path/to/build/downloads/git2/1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git git.infradead.org.mtd-utils.git.doneThe 1625d71bc771b3c4d479337faeef324f.git.infradead.org.mtd-utils.git is gitsrcname.
Signed-off-by Robert Yang <[email protected]>---
bitbake/lib/bb/fetch2/git.py| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)bitbake/lib/bb/fetch2/git.py +0
-0@@ -83,6 +83,7 @@ import subprocessimport tempfile
import bb
import bb.progress
+import hashlib
from contextlib import contextmanager
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmdbitbake/lib/bb/fetch2/git.py +0
-0@@ -277,7 +278,10 @@ class Git(FetchMethod):ud.unresolvedrev[name] = ud.revisions[name] ud.revisions[name] =
self.latest_revision(ud, d, name)
- gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) + if ud.proto == "file":
+ gitsrcname = '%s.%s' % (hashlib.md5(ud.path.encode('utf-8')).hexdigest(),os.path.basename(ud.path))
+ else:
+ gitsrcname = '%s%s' % (ud.host.replace(':', '.'),ud.path.replace('/', '.').replace('*', '.').replace(' ','_').replace('(', '_').replace(')', '_')) ifgitsrcname.startswith('.'):
gitsrcname = gitsrcname[1:]Signature · 1 line
2.42.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16370): https://lists.openembedded.org/g/bitbake-devel/message/16370
Mute This Topic: https://lists.openembedded.org/mt/105397591/3616940
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-