Skip to contentSkip to conversation

fixes and improvements for layers cloning

PATCH In review Ed Bartosh 2 participants 8 messages Sep 23, 2015

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover fixes and improvements for layers cloning

Hi,

This patchset includes changes for Toaster cloning functionality:
- used more appropriate git commands: git reset --hard instead of git checkout && git rebase.
- changed naming of clone directories, made them unique
- moved clones to subdirectory

Please, review.

The following changes since commit 7b86c771c80d0759c2ca0e57c46c4c966f89c49e:

bitbake: bitbake: bb.fetch2.git: Import errno module (2015-09-19 22:38:44 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/clones
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/clones

Ed Bartosh (6):
toaster: don't use --single-branch when cloning
toaster: use git reset --hard instead of rebase
toaster: fix bug in resetting git repository
toaster: fix reimporting module
toaster: make clone directory name unique
toaster: move clones into subdirectory

 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

--
Ed

author
#1 lore raw

PATCH 1/6 toaster: don't use --single-branch when cloning

git clone --single-branch works only with ref names. It fails if commit sha1 is set in layer configuration with this error:
fatal: Remote branch <commit sha1> not found in upstream origin

Cloning repository without using --single-branch should work for refs and commit sha1.

[YOCTO #7505]

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +2 -2
index a9909b8..3d2f8b0 100644
@@ -260,8 +260,8 @@ class LocalhostBEController(BuildEnvironmentController):
                     self._shellcmd("git remote remove origin", localdirname)
                     self._shellcmd("git remote add origin \"%s\"" % giturl, localdirname)
                 else:
-                    logger.debug("localhostbecontroller: cloning %s:%s in %s" % (giturl, commit, localdirname))
-                    self._shellcmd("git clone \"%s\" --single-branch --branch \"%s\" \"%s\"" % (giturl, commit, localdirname))
+                    logger.debug("localhostbecontroller: cloning %s in %s" % (giturl, localdirname))
+                    self._shellcmd('git clone "%s" "%s"' % (giturl, localdirname))
 
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH 2/6 toaster: use git reset --hard instead of rebase

In reply to Ed Bartosh

Replaced 'git checkout <ref> && git rebase' with 'git reset --hard' as with git checkout repository ends up with detached HEAD. Rebase makes things even worse as it can cause conflicts. git reset --hard resets repository to the required state in a most straightforward and reliable way.

[YOCTO #7505]

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index 3d2f8b0..42be4af 100644
@@ -266,7 +266,7 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd("git fetch --all && git checkout \"%s\" && git rebase \"origin/%s\"" % (commit, commit) , localdirname)
+                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH 3/6 toaster: fix bug in resetting git repository

In reply to Ed Bartosh

git reset --hard should be given either commit id or origin/<ref name> to work properly. Without this fix git will complain that origin/<commit id> does not exist.

[YOCTO #7505]

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +2 -1
index 42be4af..0c34957 100644
@@ -266,7 +266,8 @@ class LocalhostBEController(BuildEnvironmentController):
             # branch magic name "HEAD" will inhibit checkout
             if commit != "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %s to %s " % (commit, localdirname))
-                self._shellcmd('git fetch --all && git reset --hard "origin/%s"' % commit, localdirname)
+                ref = commit if re.match('^[a-fA-F0-9]+$', commit) else 'origin/%s' % commit
+                self._shellcmd('git fetch --all && git reset --hard "%s"' % ref, localdirname)
 
             # take the localdirname as poky dir if we can find the oe-init-build-env
             if self.pokydirname is None and os.path.exists(os.path.join(localdirname, "oe-init-build-env")):
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH 4/6 toaster: fix reimporting module

In reply to Ed Bartosh

removed second 'import re' from localhostbecontroller.py

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 1 -
 1 file changed, 1 deletion(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +0 -1
index 0c34957..c22d41b 100644
@@ -181,7 +181,6 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """ Utility that returns the last component of a git path as directory
         """
-        import re
         components = re.split(r'[:\.\/]', url)
         base = components[-2] if components[-1] == "git" else components[-1]
 
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 5/6 toaster: make clone directory name unique

In reply to Ed Bartosh

Changed naming scheme for clone directory. Used full git url and branch to make it unique. This should fix the issue with using the same git repository, but different protocols, e.g. git://some.git.repo and
http://some.git.repo.

[YOCTO #8101]

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +2 -7
index c22d41b..f826ad6 100644
@@ -179,14 +179,9 @@ class LocalhostBEController(BuildEnvironmentController):
         logger.debug("localhostbecontroller: Stopped bitbake server")
 
     def getGitCloneDirectory(self, url, branch):
-        """ Utility that returns the last component of a git path as directory
-        """
-        components = re.split(r'[:\.\/]', url)
-        base = components[-2] if components[-1] == "git" else components[-1]
-
+        """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (base, branch)
-
+            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 6/6 toaster: move clones into subdirectory

In reply to Ed Bartosh

By default Toaster clones layers to the same level where poky clone is. This can look messy if a lot of layers are used for Toaster builds.

Moving them into _toaster_clones/ subdirectory should make directory structure looking cleaner. It also safer to isolate toaster clones from what user might create.

---
 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +1 -1
index f826ad6..d3528e8 100644
@@ -181,7 +181,7 @@ class LocalhostBEController(BuildEnvironmentController):
     def getGitCloneDirectory(self, url, branch):
         """Construct unique clone directory name out of url and branch."""
         if branch != "HEAD":
-            return "_%s_%s.toaster_cloned" % (re.sub('[:/]', '_', url), branch)
+            return "_toaster_clones/_%s_%s" % (re.sub('[:/]', '_', url), branch)
 
         # word of attention; this is a localhost-specific issue; only on the localhost we expect to have "HEAD" releases
         # which _ALWAYS_ means the current poky checkout
Signature · 1 line
2.1.4
#7 lore raw
In reply to Ed Bartosh

upstreamed. better names, in 1 dir, better git commands, what's not to like :)

ty,
b

Quoted 38 lines · Ed Bartosh

Hi,

This patchset includes changes for Toaster cloning functionality:
- used more appropriate git commands: git reset --hard instead of git checkout && git rebase.
- changed naming of clone directories, made them unique
- moved clones to subdirectory

Please, review.

The following changes since commit 7b86c771c80d0759c2ca0e57c46c4c966f89c49e:

bitbake: bitbake: bb.fetch2.git: Import errno module (2015-09-19 22:38:44 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/clones
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/clones

Ed Bartosh (6):
toaster: don't use --single-branch when cloning
toaster: use git reset --hard instead of rebase
toaster: fix bug in resetting git repository
toaster: fix reimporting module
toaster: make clone directory name unique
toaster: move clones into subdirectory

 bitbake/lib/toaster/bldcontrol/localhostbecontroller.py | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

--
Ed

--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster