Skip to contentSkip to conversation

gitsm.py: submodule init and ssh url processing

PATCH Superseded Mark Hatle 2 participants 7 messages Jan 8, 2019 – Jan 9, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

cover gitsm.py: submodule init and ssh url processing

This patch set covers two different items. The first is work that started before Christmas where someone noticed a repository that had a defined gitmodule, but it was never initialized. The first patch deals with that issue. (Thanks Raphael Lisicki for the fix approach...)

The second set works through the issues that Linus Ziegert, and Krystian Garlinski have brought up on the list. This fix takes a different approach then Linus's approach of adding a specific regex. Instead we look for '://', if we find it -- its a URL, otherwise we look for ':' -- finding that it must be 'ssh style', otherwise it has to be a file URL. I think this simplifies the overall approach.

Finally a test case was developed. See:

http://git.yoctoproject.org/cgit/cgit.cgi/git-submodule-test/tree/.gitmodules?h=ssh-gitsm-tests

for the specific .gitmodules file that was tested. I would appreciate it if the people experiencing problems could try out this code and verify it works in their configurations.

RP - in the last patch, I wasn't sure how to switch which URL is used. Either the original simplified testing or the more complex ssh based tests. Any suggestions on a better way to handle this?

Mark Hatle (3):
gitsm.py: Fix when a submodule is defined, but not initialized gitsm.py: Add support for alternative URL formats from submodule files tests/fetch.py: Add alternative gitsm test case

 lib/bb/fetch2/gitsm.py | 39 +++++++++++++++++++++++++++++++++------
 lib/bb/tests/fetch.py  |  6 +++++-
 2 files changed, 38 insertions(+), 7 deletions(-)
Signature · 1 line
1.8.3.1
author
#1 lore raw

PATCH 1/3 gitsm.py: Fix when a submodule is defined, but not initialized

It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value.

Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well.

Thanks to [email protected] for their help is figuring out how to resolve this issue.

Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path.

---
 lib/bb/fetch2/gitsm.py | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)
lib/bb/fetch2/gitsm.py +18 -4
index 35729db..88609f0 100644
@@ -64,6 +64,7 @@ class GitSM(Git):
     def update_submodules(self, ud, d):
         submodules = []
         paths = {}
+        revision = {}
         uris = {}
         local_paths = {}
 
@@ -77,6 +78,7 @@ class GitSM(Git):
             for m, md in self.parse_gitmodules(gitmodules).items():
                 submodules.append(m)
                 paths[m] = md['path']
+                revision[m] = ud.revisions[name]
                 uris[m] = md['url']
                 if uris[m].startswith('..'):
                     newud = copy.copy(ud)
@@ -84,7 +86,12 @@ class GitSM(Git):
                     uris[m] = Git._get_repo_url(self, newud)
 
         for module in submodules:
-            module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir)
+            module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, revision[module], paths[module]), d, quiet=True, workdir=ud.clonedir)
+
+            if not module_hash:
+                logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
+                continue
+
             module_hash = module_hash.split()[2]
 
             # Build new SRC_URI
@@ -143,7 +150,7 @@ class GitSM(Git):
         if not ud.shallow or ud.localpath != ud.fullshallow:
             self.update_submodules(ud, d)
 
-    def copy_submodules(self, submodules, ud, destdir, d):
+    def copy_submodules(self, submodules, ud, name, destdir, d):
         if ud.bareclone:
             repo_conf = destdir
         else:
@@ -156,6 +163,13 @@ class GitSM(Git):
             srcpath = os.path.join(ud.clonedir, 'modules', md['path'])
             modpath = os.path.join(repo_conf, 'modules', md['path'])
 
+            # Check if the module is initialized
+            module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=ud.clonedir)
+
+            if not module_hash:
+                logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
+                continue
+
             if os.path.exists(srcpath):
                 if os.path.exists(os.path.join(srcpath, '.git')):
                     srcpath = os.path.join(srcpath, '.git')
@@ -188,7 +202,7 @@ class GitSM(Git):
                 continue
 
             submodules = self.parse_gitmodules(gitmodules)
-            self.copy_submodules(submodules, ud, dest, d)
+            self.copy_submodules(submodules, ud, name, dest, d)
 
     def unpack(self, ud, destdir, d):
         Git.unpack(self, ud, destdir, d)
@@ -211,7 +225,7 @@ class GitSM(Git):
                 continue
 
             submodules = self.parse_gitmodules(gitmodules)
-            self.copy_submodules(submodules, ud, ud.destdir, d)
+            self.copy_submodules(submodules, ud, name, ud.destdir, d)
 
             submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()]
             while len(submodules_queue) != 0:
Signature · 1 line
1.8.3.1
author
#2 lore raw

PATCH 2/3 gitsm.py: Add support for alternative URL formats from submodule files

In reply to Mark Hatle

The following appear to be the git supported formats:

proto://user:pass@host/path (URI format)
user@host:path (SSH format)
/path or ./path or ../path (local file format)

We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly.

Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well.

---
 lib/bb/fetch2/gitsm.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)
lib/bb/fetch2/gitsm.py +15 -2
index 88609f0..0981bba 100644
@@ -95,8 +95,21 @@ class GitSM(Git):
             module_hash = module_hash.split()[2]
 
             # Build new SRC_URI
-            proto = uris[module].split(':', 1)[0]
-            url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
+            if "://" not in uris[module]:
+                # It's ssh if the format does NOT have "://", but has a ':'
+                if ":" in uris[module]:
+                    proto = "ssh"
+                    if ":/" in uris[module]:
+                        url = "gitsm://" + uris[module].replace(':/', '/', 1)
+                    else:
+                        url = "gitsm://" + uris[module].replace(':', '/', 1)
+                else: # Fall back to 'file' if there is no ':'
+                    proto = "file"
+                    url = "gitsm://" + uris[module]
+            else:
+                proto = uris[module].split(':', 1)[0]
+                url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
+
             url += ';protocol=%s' % proto
             url += ";name=%s" % module
             url += ";bareclone=1;nocheckout=1;nobranch=1"
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [PATCH 2/3] gitsm.py: Add support for alternative URL formats from submodule files

Quoted 3 lines · Mark Hatle

Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well.

...

lib/bb/fetch2/gitsm.py class GitSM(Git): · line 105

Quoted 4 lines
lib/bb/fetch2/gitsm.py +4 -0
+  if ":/" in uris[module]:
+      url = "gitsm://" + uris[module].replace(':/', '/', 1)
+  else:
+      url = "gitsm://" + uris[module].replace(':', '/', 1)

Without a / prefix in the original ssh path, the path becomes relative to whatever the ssh server wants, usually the user's $HOME. I.e., they do not refer to the same path on the remote system.

[email protected]:/foo.git -> ssh://[email protected]/foo.git [email protected]:foo.git -> ssh://[email protected]/home/alice/foo.git (or something else!)

The path in an ssh:// uri is absolute, so the relative case must be handled differently. But it's easy, git supports ~ expansion, like:

[email protected]:foo.git -> ssh://[email protected]/~/foo.git

Signature · 1 line
olofjn
#4 lore raw

Re: [PATCH 2/3] gitsm.py: Add support for alternative URL formats from submodule files

Quoted 5 lines · Olof Johansson

The path in an ssh:// uri is absolute, so the relative case must be handled differently. But it's easy, git supports ~ expansion, like:

[email protected]:foo.git -> ssh://[email protected]/~/foo.git

Tested some more. This works with git over openssh (possibly only against unix systems), but not with github it seems, so this wasn't as general of a solution I hoped. Sorry. I think the issue is real, but my proposed solution wasn't :(.

And, of course, the github ssh remote specifiers all look like [email protected]:alice/foo.

Signature · 1 line
olofjn
author
#5 lore raw

Re: [PATCH 2/3] gitsm.py: Add support for alternative URL formats from submodule files

Quoted 7 lines · Olof Johansson
Quoted 5 lines · Olof Johansson

The path in an ssh:// uri is absolute, so the relative case must be handled differently. But it's easy, git supports ~ expansion, like:

[email protected]:foo.git -> ssh://[email protected]/~/foo.git

Quoted 8 lines

Tested some more. This works with git over openssh (possibly only against unix systems), but not with github it seems, so this wasn't as general of a solution I hoped. Sorry. I think the issue is real, but my proposed solution wasn't :(.

And, of course, the github ssh remote specifiers all look like [email protected]:alice/foo.

Unfortunately, this is what I observed as well. In the specific test cases I had replacing it 'as-is' was a better solution. (test case was specifically github, and gitolite based)

The only alternative I can see in this case, is to try one fetch (try/catch) and if it fails to try the alternative. If they both fail, then it becomes a fetch failure.

--Mark

author
#6 lore raw

PATCH 3/3 tests/fetch.py: Add alternative gitsm test case

In reply to Mark Hatle

In order to test the ssh processing in gitsm, we add an alternative testcase that can be downloaded from git.yoctoproject.org. However, this test case requries (read) access, via ssh, to git.yoctoproject.org.

---
 lib/bb/tests/fetch.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
lib/bb/tests/fetch.py +5 -1
index 6848095..311c701 100644
@@ -893,7 +893,11 @@ class FetcherNetworkTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_git_submodule(self):
-        fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"], self.d)
+        # URL with ssh submodules
+        url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=0d3ffc14bce95e8b3a21a0a67bfe4c4a96ba6350"
+        # Original URL (comment this if you have ssh access to git.yoctoproject.org)
+        url = "gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"
+        fetcher = bb.fetch.Fetch([url], self.d)
         fetcher.download()
         # Previous cwd has been deleted
         os.chdir(os.path.dirname(self.unpackdir))
Signature · 1 line
1.8.3.1