Skip to contentSkip to conversation

fetch2/gitsm: Unpack shallow mirror tarballs

PATCH Superseded Paul Barker 3 participants 8 messages Feb 24, 2020 – Feb 25, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fetch2/gitsm: Unpack shallow mirror tarballs

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +6 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be unpacked so
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d):
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir)
         self.process_submodules(ud, ud.clonedir, download_submodule, d)
 
     def unpack(self, ud, destdir, d):
Signature · 1 line
2.20.1
#1 lore raw
Quoted 15 lines · Paul Barker

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +0 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 

lib/bb/fetch2/gitsm.py class GitSM(Git): · line 180

Quoted 10 lines
lib/bb/fetch2/gitsm.py +6 -0
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be unpacked so
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d):
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir)
         self.process_submodules(ud, ud.clonedir, download_submodule, d)
 
     def unpack(self, ud, destdir, d):

Does this mean we're missing a test?

Cheers,

Richard

#2 lore raw
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Paul Barker

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +6 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be unpacked so
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d):
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir)
         self.process_submodules(ud, ud.clonedir, download_submodule, d)
 
Quoted 3 lines

lib/bb/fetch2/gitsm.py class GitSM(Git): · line 180

Quoted 1 line
lib/bb/fetch2/gitsm.py +0 -0
     def unpack(self, ud, destdir, d):

Does this mean we're missing a test?

Possibly. Maybe a two step test with a recipe using a gitsm:// URL:
1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.

Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.

#3 lore raw
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +2 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be

unpacked so

lib/bb/fetch2/gitsm.py +2 -0
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and

Git.need_update(self, ud, d):

lib/bb/fetch2/gitsm.py +2 -0
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,

workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,
download_submodule, d)

Quoted 3 lines
Quoted 1 line

def unpack(self, ud, destdir, d):

Does this mean we're missing a test?

Possibly. Maybe a two step test with a recipe using a gitsm:// URL:
1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.

Quoted 2 lines

Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.

It sounds like it might be reasonable, I'm not sure why we can't do that in lib/bb/tests/fetch.py though? There are other tests which do similar sequences of commands.

You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?

Sorry to push testing btw but the git fetcher is one of our better tested pieces of code so I do want to try and maintain good test coverage!

Cheers,

Richard

#4 lore raw
Quoted 49 lines · Richard Purdie
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +2 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be

unpacked so

lib/bb/fetch2/gitsm.py +2 -0
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and

Git.need_update(self, ud, d):

lib/bb/fetch2/gitsm.py +2 -0
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,

workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,
download_submodule, d)

Quoted 3 lines
Quoted 1 line

def unpack(self, ud, destdir, d):

Does this mean we're missing a test?

Possibly. Maybe a two step test with a recipe using a gitsm:// URL:
1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.

Quoted 2 lines

Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.

Quoted 7 lines

It sounds like it might be reasonable, I'm not sure why we can't do that in lib/bb/tests/fetch.py though? There are other tests which do similar sequences of commands.

You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?

Ok, I've looked more closely at GitMakeShallowTest in lib/bb/tests/fetch.py. We could write a new test_shallow_submodule_mirrors function based on test_shallow_submodules and test_shallow_mirrors.

The case I would want to catch is do_unpack failing. I guess we could check that fetcher.unpack() works as a proxy for that.

#5 lore raw

I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.

Quoted 72 lines · Paul Barker
Quoted 49 lines · Richard Purdie
Quoted 44 lines · Paul Barker
Quoted 30 lines · Richard Purdie
Quoted 29 lines · Paul Barker

When a shallow mirror tarball is used to satisfy a gitsm URI it needs to
be unpacked so that the .gitmodules file can be examined.

---
 lib/bb/fetch2/gitsm.py | 6 ++++++
 1 file changed, 6 insertions(+)
lib/bb/fetch2/gitsm.py +2 -0
index aa121cbe..98f5dae5 100644
@@ -169,6 +169,12 @@ class GitSM(Git):
                 raise
 
         Git.download(self, ud, d)
+
+        # If we're using a shallow mirror tarball it needs to be

unpacked so

lib/bb/fetch2/gitsm.py +2 -0
+        # that we can examine the .gitmodules file
+        if ud.shallow and os.path.exists(ud.fullshallow) and

Git.need_update(self, ud, d):

lib/bb/fetch2/gitsm.py +2 -0
+            bb.utils.mkdirhier(ud.clonedir)
+            runfetchcmd("tar -xzf %s" % ud.fullshallow, d,

workdir=ud.clonedir)
self.process_submodules(ud, ud.clonedir,
download_submodule, d)

Quoted 3 lines
Quoted 1 line

def unpack(self, ud, destdir, d):

Does this mean we're missing a test?

Possibly. Maybe a two step test with a recipe using a gitsm:// URL:
1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR.
2) Using the mirror generated in step 1 try to fetch and unpack the recipe.

Quoted 2 lines

Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way.

Quoted 7 lines

It sounds like it might be reasonable, I'm not sure why we can't do that in lib/bb/tests/fetch.py though? There are other tests which do similar sequences of commands.

You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa?

Ok, I've looked more closely at GitMakeShallowTest in lib/bb/tests/fetch.py. We could write a new test_shallow_submodule_mirrors function based on test_shallow_submodules and test_shallow_mirrors.

The case I would want to catch is do_unpack failing. I guess we could check that fetcher.unpack() works as a proxy for that.
--
_______________________________________________
bitbake-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Signature · 4 lines
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
#6 lore raw
Quoted 2 lines · Christopher Larson

I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.

Any suggestions on where we can unpack the shallow tarball then? We need it to be unpacked somewhere so that we can examine the .gitmodules file.

#7 lore raw
Quoted 8 lines · Paul Barker
Quoted 2 lines · Christopher Larson

I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary.

Any suggestions on where we can unpack the shallow tarball then? We need it to be unpacked somewhere so that we can examine the .gitmodules file.

So my downloads directory blew up and I saw lots of fetch errors for git SRC_URIs on builds overnight after testing this patch.

We could extract to a temporary directory as needed when downloading a gitsm shallow mirror tarball and then throw this away once we've checked .gitmodules. That would mean repeating the tarball extraction if the download method is executed again but that should be fairly rare. How does that sound?

The alternative would be to create a new 'gitsmshallow' directory under DL_DIR for unpacking gitsm shallow mirror tarballs only. The names of directories under 'gitsmshallow' would need to include the commit hash given in the shallow tarball filename so that we don't have clashes when DL_DIR is shared between builds which use different SRCREVs of the same gitsm SRC_URI.

I also have a test case ready to go which I'll include in v2 of this patch once we've decided where to extract the tarballs.

Thanks,
Paul