fetch2/gitsm: Unpack shallow mirror tarballs
PATCH 1/1 fetch2/gitsm: Unpack shallow mirror tarballs
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.
--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +6 -0
@@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to be unpacked so + # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) self.process_submodules(ud, ud.clonedir, download_submodule, d) def unpack(self, ud, destdir, d):
Signature · 1 line
2.20.1
Quoted 15 lines · Paul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.
Signed-off-by Paul Barker <[email protected]>--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)lib/bb/fetch2/gitsm.py +0
-0@@ -169,6 +169,12 @@ class GitSM(Git): raise
lib/bb/fetch2/gitsm.py class GitSM(Git): · line 180
Quoted 10 lines
lib/bb/fetch2/gitsm.py +6
-0Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to be unpacked so + # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) self.process_submodules(ud, ud.clonedir, download_submodule, d) def unpack(self, ud, destdir, d):
Does this mean we're missing a test?
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Paul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.
Signed-off-by Paul Barker <[email protected]>--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)lib/bb/fetch2/gitsm.py +6
-0@@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to be unpacked so + # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) self.process_submodules(ud, ud.clonedir, download_submodule, d)
Quoted 3 lines
lib/bb/fetch2/gitsm.py class GitSM(Git): · line 180
Quoted 1 line
lib/bb/fetch2/gitsm.py +0
-0def unpack(self, ud, destdir, d):Does this mean we're missing a test?
Possibly. Maybe a two step test with a recipe using a gitsm:// URL:
1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.
Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.Signed-off-by Paul Barker <[email protected]>--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)lib/bb/fetch2/gitsm.py +2
-0@@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to beunpacked so
lib/bb/fetch2/gitsm.py +2
-0+ # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and
Git.need_update(self, ud, d):lib/bb/fetch2/gitsm.py +2
-0+ bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d,workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,download_submodule, d)Quoted 3 lines
Quoted 1 line
def
unpack(self, ud, destdir, d):Does this mean we're missing a test?
Possibly. Maybe a two step test with a recipe using a
gitsm://URL:
1) Fetch the recipe withBB_GENERATE_MIRROR_TARBALLS,BB_GENERATE_SHALLOW_TARBALLSandBB_GIT_SHALLOWset. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw awayDL_DIR,SSTATE_DIRand TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.
Quoted 2 lines
Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.
It sounds like it might be reasonable, I'm not sure why we can't do that in lib/bb/tests/fetch.py though? There are other tests which do similar sequences of commands.
You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?
Sorry to push testing btw but the git fetcher is one of our better tested pieces of code so I do want to try and maintain good test coverage!
Cheers,
Richard
Quoted 49 lines · Richard Purdie
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.Signed-off-by Paul Barker <[email protected]>--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)lib/bb/fetch2/gitsm.py +2
-0@@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to beunpacked so
lib/bb/fetch2/gitsm.py +2
-0+ # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and
Git.need_update(self, ud, d):lib/bb/fetch2/gitsm.py +2
-0+ bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d,workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,download_submodule, d)Quoted 3 lines
Quoted 1 line
def
unpack(self, ud, destdir, d):Does this mean we're missing a test?
Possibly. Maybe a two step test with a recipe using a
gitsm://URL:
1) Fetch the recipe withBB_GENERATE_MIRROR_TARBALLS,BB_GENERATE_SHALLOW_TARBALLSandBB_GIT_SHALLOWset. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw awayDL_DIR,SSTATE_DIRand TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.Quoted 2 lines
Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.
Quoted 7 lines
It sounds like it might be reasonable, I'm not sure why we can't do that in
lib/bb/tests/fetch.pythough? There are other tests which do similar sequences of commands.You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?
Ok, I've looked more closely at GitMakeShallowTest in lib/bb/tests/fetch.py. We could write a new test_shallow_submodule_mirrors function based on test_shallow_submodules and test_shallow_mirrors.
The case I would want to catch is do_unpack failing. I guess we could check that fetcher.unpack() works as a proxy for that.
I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.
Quoted 72 lines · Paul Barker
Quoted 49 lines · Richard Purdie
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker
When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.Signed-off-by Paul Barker <[email protected]>--- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+)lib/bb/fetch2/gitsm.py +2
-0@@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to beunpacked so
lib/bb/fetch2/gitsm.py +2
-0+ # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and
Git.need_update(self, ud, d):lib/bb/fetch2/gitsm.py +2
-0+ bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d,workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,download_submodule, d)Quoted 3 lines
Quoted 1 line
def
unpack(self, ud, destdir, d):Does this mean we're missing a test?
Possibly. Maybe a two step test with a recipe using a
gitsm://URL:
1) Fetch the recipe withBB_GENERATE_MIRROR_TARBALLS,BB_GENERATE_SHALLOW_TARBALLSandBB_GIT_SHALLOWset. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw awayDL_DIR,SSTATE_DIRand TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.Quoted 2 lines
Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.
Quoted 7 lines
It sounds like it might be reasonable, I'm not sure why we can't do that in
lib/bb/tests/fetch.pythough? There are other tests which do similar sequences of commands.You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?
Ok, I've looked more closely at GitMakeShallowTest in
lib/bb/tests/fetch.py. We could write a newtest_shallow_submodule_mirrorsfunction based ontest_shallow_submodulesandtest_shallow_mirrors.The case I would want to catch is
do_unpackfailing. I guess we could check thatfetcher.unpack()works as a proxy for that.
--
_______________________________________________
bitbake-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel
Signature · 4 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Senior Software Engineer, Mentor Graphics
Quoted 2 lines · Christopher Larson
I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.
Any suggestions on where we can unpack the shallow tarball then? We need it to be unpacked somewhere so that we can examine the .gitmodules file.
Quoted 8 lines · Paul Barker
Quoted 2 lines · Christopher Larson
I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.
Any suggestions on where we can unpack the shallow tarball then? We need it to be unpacked somewhere so that we can examine the .gitmodules file.
So my downloads directory blew up and I saw lots of fetch errors for git SRC_URIs on builds overnight after testing this patch.
We could extract to a temporary directory as needed when downloading a gitsm shallow mirror tarball and then throw this away once we've checked .gitmodules. That would mean repeating the tarball extraction if the download method is executed again but that should be fairly rare. How does that sound?
The alternative would be to create a new 'gitsmshallow' directory under DL_DIR for unpacking gitsm shallow mirror tarballs only. The names of directories under 'gitsmshallow' would need to include the commit hash given in the shallow tarball filename so that we don't have clashes when DL_DIR is shared between builds which use different SRCREVs of the same gitsm SRC_URI.
I also have a test case ready to go which I'll include in v2 of this patch once we've decided where to extract the tarballs.
Thanks,
Paul