git fetch support for rebaseable repo (a.k.a crazy git fetcher)

From: Yu Ke <[email protected]>

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/crazygit
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/crazygit

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (1):
git fetcher: add support for rebaseable git repo

From: Yu Ke<[email protected]>

Some upstream git repo may rebase in the future, which means current revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror tar ball is per repo, and may also change in the rebase and lost the current revision info.

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this case, even upstream rebase, the git mirror still has the current revision info.
- for rebaseable repo, generate mirror tar ball by default, since the repo may change in the future.

Signed-off-by Yu Ke<[email protected]>

---
  bitbake/lib/bb/fetch2/git.py |   28 ++++++++++++++++++++--------
  1 files changed, 20 insertions(+), 8 deletions(-)

bitbake/lib/bb/fetch2/git.py +20 -8

index 811acbf..82721c6 100644
@@ -57,6 +57,11 @@ class Git(FetchMethod):
          if 'nocheckout' in ud.parm:
              ud.nocheckout = True
 
+        # rebaseable means the upstream git repo may rebase in the future,
+        # and current revision may disappear from upstream repo
+        # rebaseable is false by default. set rebaseable=1 in SRC_URI if rebaseable.
+        ud.rebaseable = ud.parm.get("rebaseable","0") == "1"
+
          branches = ud.parm.get("branch", "master").split(',')
          if len(branches) != len(ud.names):
              raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)

@@ -65,16 +70,9 @@ class Git(FetchMethod):
              branch = branches[ud.names.index(name)]
              ud.branches[name] = branch
 
-        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
-        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
-        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
-        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
-
          ud.basecmd = data.getVar("FETCHCMD_git", d, True) or "git"
 
-        ud.write_tarballs = (data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0"
-
-        ud.localfile = ud.clonedir
+        ud.write_tarballs = ((data.getVar("BB_GENERATE_MIRROR_TARBALLS", d, True) or "0") != "0") or ud.rebaseable
 
          ud.setup_revisons(d)
 

@@ -84,6 +82,20 @@ class Git(FetchMethod):
                  ud.branches[name] = ud.revisions[name]
                  ud.revisions[name] = self.latest_revision(ud.url, ud, d, name)
 
+        gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
+        # for rebaseable git repo, it is necessary to keep mirror tar ball
+        # per revision, so that even the revision disappears from the
+        # upstream repo in the future, the mirror will remain intact and still
+        # contains the revision
+        if ud.rebaseable:
+            for name in ud.names:
+                gitsrcname = gitsrcname + '_' + ud.revisions[name]
+        ud.mirrortarball = 'git2_%s.tar.gz' % (gitsrcname)
+        ud.fullmirror = os.path.join(data.getVar("DL_DIR", d, True), ud.mirrortarball)
+        ud.clonedir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)
+
+        ud.localfile = ud.clonedir
+
      def localpath(self, url, ud, d):
          return ud.clonedir
 

Quoted 9 lines · Yu Ke

From: Yu Ke<[email protected]>

Some upstream git repo may rebase in the future, which means current revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror tar ball is per repo, and may also change in the rebase and lost the current revision info.

Quoted 7 lines

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this case, even upstream rebase, the git mirror still has the current revision info.
- for rebaseable repo, generate mirror tar ball by default, since the repo may change in the future.

Do we have the options to SRC_URI documented somewhere ? if yes then this should be added there too.

Quoted 18 lines · Khem Raj

Quoted 16 lines · Yu Ke

From: Yu Ke<[email protected]>

Some upstream git repo may rebase in the future, which means current revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror tar ball is per repo, and may also change in the rebase and lost the current revision info.

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this case, even upstream rebase, the git mirror still has the current revision info.
- for rebaseable repo, generate mirror tar ball by default, since the repo may change in the future.

Quoted 2 lines

Do we have the options to SRC_URI documented somewhere ? if yes then this should be added there too.

I was looking and didn't find them. At the very least these should be documented in the fetch2/git.py file for reference when reviewing/maintaining the source.

Quoted 21 lines · Darren Hart

Quoted 18 lines · Khem Raj

Quoted 16 lines · Yu Ke

From: Yu Ke<[email protected]>

Some upstream git repo may rebase in the future, which means current revision may disappear from the upstream repo after the rebase.

current git fetcher can not handle this case, because the git mirror tar ball is per repo, and may also change in the rebase and lost the current revision info.

To fix this issue, this patch
- add rebaseable tag in the SRC_URI
- for rebaseable repo, make git mirror tar ball per revision, in this case, even upstream rebase, the git mirror still has the current revision info.
- for rebaseable repo, generate mirror tar ball by default, since the repo may change in the future.

Quoted 7 lines

Quoted 2 lines

Do we have the options to SRC_URI documented somewhere ? if yes then this should be added there too.

I was looking and didn't find them. At the very least these should be documented in the fetch2/git.py file for reference when reviewing/maintaining the source.

This is good reminder, Thanks.

There are several places that hold this info:

- bitbake manual: http://bitbake.berlios.de/manual/ch03s07.html
- openembedded manual:
http://docs.openembedded.org/usermanual/usermanual.html#id572539
- poky reference manual:
http://www.yoctoproject.org/docs/poky-ref-manual/poky-ref-manual.html#ref-bitbake-fetchers

It looks the most appropriate place would be bitbake manual, which is also refereed by poky reference manual. So does anyone know how can I update this mannual?

Also, it sounds good idea to add the SRC_URI options doc in the source, I will cook a patch to add this info.

Regards
Ke