Skip to contentSkip to conversation

fixes and improvements for layers cloning

PATCH New brian avery 1 participant 7 messages Sep 23, 2015

lore
Share & tools
lore
#0 lore raw

cover fixes and improvements for layers cloning

This patchset includes changes for Toaster cloning functionality:
- used more appropriate git commands: git reset --hard instead of git checkout && git rebase.
- changed naming of clone directories, made them unique
- moved clones to subdirectory

The following changes since commit 524ddd86c305af1b029c1773affe5cebffc62b28:

oeqa/utils/qemurunner.py: Remove duplicate message on LoggingThread start (2015-09-23 09:53:21 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/submit/ed/20150923-better-git-layers
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/ed/20150923-better-git-layers

Ed Bartosh (6):
toaster: don't use --single-branch when cloning
toaster: use git reset --hard instead of rebase
toaster: fix bug in resetting git repository
toaster: fix reimporting module
toaster: make clone directory name unique
toaster: move clones into subdirectory

 lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/6 toaster: don't use --single-branch when cloning

From: Ed Bartosh <[email protected]>

git clone --single-branch works only with ref names. It fails if commit sha1 is set in layer configuration with this error:
fatal: Remote branch <commit sha1> not found in upstream origin

Cloning repository without using --single-branch should work for refs and commit sha1.

[YOCTO #7505]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +2 -2
index a9909b8..3d2f8b0 100644
@@ -260,8 +260,8 @@ class LocalhostBEController(BuildEnvironmentController):
                     self._shellcmd("git remote remove origin", localdirname)
                     self._shellcmd("git remote add origin \"%s\"" % giturl, localdirname)
                 else:
-                    logger.debug("localhostbecontroller: cloning %s:%s in %s" % (giturl, commit, localdirname))
-                    self._shellcmd("git clone \"%s\" --single-branch --branch \"%s\" \"%s\"" % (giturl, commit, localdirname))
+                    logger.debug("localhostbecontroller: cloning %s in %s" % (giturl, localdirname))
+                    self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname))
 
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
Signature · 1 line
1.9.1
#2 lore raw

PATCH 2/6 toaster: use git reset --hard instead of rebase

From: Ed Bartosh <[email protected]>

Replaced 'git checkout <ref> && git rebase' with 'git reset --hard' as with git checkout repository ends up with detached HEAD. Rebase makes things even worse as it can cause conflicts. git reset --hard resets repository to the required state in a most straightforward and reliable way.

[YOCTO #7505]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index 3d2f8b0..42be4af 100644
@@ -266,7 +266,7 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd("git fetch --all && git checkout \"%s\" && git rebase \"origin/%s\"" % (commit, commit) , localdirname)
+                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
Signature · 1 line
1.9.1
#3 lore raw

PATCH 3/6 toaster: fix bug in resetting git repository

In reply to brian avery

From: Ed Bartosh <[email protected]>

git reset --hard should be given either commit id or origin/<ref name> to work properly. Without this fix git will complain that origin/<commit id> does not exist.

[YOCTO #7505]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/toaster/bldcontrol/localhostbecontroller.py +2 -1
index 42be4af..0c34957 100644
@@ -266,7 +266,8 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
+                ref = commit if re.match('^[a-fA-F0-9]+$', commit) else 'origin/%s' % commit
+                self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
Signature · 1 line
1.9.1
#4 lore raw

PATCH 4/6 toaster: fix reimporting module

In reply to brian avery

From: Ed Bartosh <[email protected]>

removed second 'import re' from localhostbecontroller.py

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 1 -
 1 file changed, 1 deletion(-)
lib/toaster/bldcontrol/localhostbecontroller.py +0 -1
index 0c34957..c22d41b 100644
@@ -181,7 +181,6 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """ Utility that returns the last component of a git path as directory
         """
-        import re
         components = re.split(r'[:\.\/]', url)
         base = components[-2] if components[-1] == "git" else components[-1]
 
Signature · 1 line
1.9.1
#5 lore raw

PATCH 5/6 toaster: make clone directory name unique

In reply to brian avery

From: Ed Bartosh <[email protected]>

Changed naming scheme for clone directory. Used full git url and branch to make it unique. This should fix the issue with using the same git repository, but different protocols, e.g. git://some.git.repo and
http://some.git.repo.

[YOCTO #8101]

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
lib/toaster/bldcontrol/localhostbecontroller.py +2 -7
index c22d41b..f826ad6 100644
@@ -179,14 +179,9 @@ class LocalhostBEController(BuildEnvironmentController):
         logger.debug("localhostbecontroller: Stopped bitbake server")
 
     def getGitCloneDirectory(self, url, branch):
-        """ Utility that returns the last component of a git path as directory
-        """
-        components = re.split(r'[:\.\/]', url)
-        base = components[-2] if components[-1] == "git" else components[-1]
-
+        """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (base, branch)
-
+            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
Signature · 1 line
1.9.1
#6 lore raw

PATCH 6/6 toaster: move clones into subdirectory

In reply to brian avery

From: Ed Bartosh <[email protected]>

By default Toaster clones layers to the same level where poky clone is. This can look messy if a lot of layers are used for Toaster builds.

Moving them into _toaster_clones/ subdirectory should make directory structure looking cleaner. It also safer to isolate toaster clones from what user might create.

---
 lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index f826ad6..d3528e8 100644
@@ -181,7 +181,7 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
+            return "_toaster_clones/_%s_%s" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
Signature · 1 line
1.9.1