Skip to contentSkip to conversation

bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

PATCH In review Matt Hoosier 3 participants 5 messages May 9, 2018 – May 10, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake: fetch2/gitsm: avoid live submodule fetching during unpack()

From: Matt Hoosier <[email protected]>

Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task.

Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules.

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

---
 bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/gitsm.py +11 -1
index 0aff1008e5..1f3fc44352 100644
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
Signature · 1 line
2.13.6
#1 lore raw
Quoted 8 lines · Matt Hoosier

From: Matt Hoosier <[email protected]>

Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task.

Quoted 5 lines

Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules.

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

Patches for bitbake should be sent to the bitbake-devel mailing list:

http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Quoted 32 lines
---
 bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/gitsm.py +11 -2
index 0aff1008e5..1f3fc44352 100644
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " + ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init --recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir, 'modules'), repo_conf), d)
+
+            # Careful not to hit the network during unpacking; all history should already
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init --recursive --no-fetch", d, workdir=ud.destdir)
--

2.13.6

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#2 lore raw
Quoted 13 lines · Andre McCurdy
Quoted 11 lines · Matt Hoosier

From: Matt Hoosier <[email protected]>

Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task.

Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules.

Quoted 6 lines
Quoted 2 lines

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

Patches for bitbake should be sent to the bitbake-devel mailing list:

http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Right, okay. How's that meant to work for stuff developed in-tree with poky? Do you manually want the 'bitbake/' leading directory stripped off before emailing the patch?

Quoted 41 lines

Review

Quoted 7 lines

---
 bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/bitbake/lib/bb/fetch2/gitsm.py

bitbake/lib/bb/fetch2/gitsm.py class GitSM(Git): · line 134

Quoted 7 lines

index 0aff1008e5..1f3fc44352 100644

bitbake/lib/bb/fetch2/gitsm.py +0 -0
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " +

ud.revisions[ud.names[0]], d, workdir=ud.destdir)

bitbake/lib/bb/fetch2/gitsm.py class GitSM(Git): · line 135

Quoted 1 line
bitbake/lib/bb/fetch2/gitsm.py +0 -1
-            runfetchcmd(ud.basecmd + " submodule update --init

--recursive", d, workdir=ud.destdir)

bitbake/lib/bb/fetch2/gitsm.py class GitSM(Git):

Quoted 7 lines
bitbake/lib/bb/fetch2/gitsm.py +7 -0
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,

'modules'), repo_conf), d)

bitbake/lib/bb/fetch2/gitsm.py class GitSM(Git): · line 143

Quoted 2 lines
bitbake/lib/bb/fetch2/gitsm.py +2 -0
+
+            # Careful not to hit the network during unpacking; all

history should already

bitbake/lib/bb/fetch2/gitsm.py class GitSM(Git): · line 145

Quoted 2 lines
bitbake/lib/bb/fetch2/gitsm.py +2 -0
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init

--recursive --no-fetch", d, workdir=ud.destdir)

Quoted 8 lines

--
2.13.6

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#3 lore raw
Quoted 18 lines · Matt Hoosier
Quoted 17 lines · Andre McCurdy
Quoted 13 lines · Matt Hoosier

From: Matt Hoosier <[email protected]>

Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task.

Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules.

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

Quoted 7 lines
Quoted 3 lines

Patches for bitbake should be sent to the bitbake-devel mailing list:

http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Right, okay. How's that meant to work for stuff developed in-tree with poky? Do you manually want the 'bitbake/' leading directory stripped off before emailing the patch?

Yes, patches for bitbake should apply directly to a clone of the bitbake repo.

Manually editing a patch created from poky is certainly do-able, but applying the patch to a genuine bikebake repo (with "git am -p2 ...") and re-generating the patch from there may be safer. Whichever you prefer.

Quoted 40 lines
Quoted 38 lines
Quoted 38 lines
---
 bitbake/lib/bb/fetch2/gitsm.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
bitbake/lib/bb/fetch2/gitsm.py +7 -2
index 0aff1008e5..1f3fc44352 100644
@@ -132,4 +132,14 @@ class GitSM(Git):
 
         if self.uses_submodules(ud, d, ud.destdir):
             runfetchcmd(ud.basecmd + " checkout " +
 ud.revisions[ud.names[0]], d, workdir=ud.destdir)
-            runfetchcmd(ud.basecmd + " submodule update --init
--recursive", d, workdir=ud.destdir)
+
+            # Copy over the submodules' fetched histories too.
+            if ud.bareclone:
+                repo_conf = ud.destdir
+            else:
+                repo_conf = os.path.join(ud.destdir, '.git')
+            runfetchcmd("cp -pr %s %s" % (os.path.join(ud.clonedir,

'modules'), repo_conf), d)

bitbake/lib/bb/fetch2/gitsm.py +2 -0
+
+            # Careful not to hit the network during unpacking; all

history should already

bitbake/lib/bb/fetch2/gitsm.py +2 -2
+            # be fetched.
+            runfetchcmd(ud.basecmd + " submodule update --init
--recursive --no-fetch", d, workdir=ud.destdir)
--

2.13.6

--
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

#4 lore raw
In reply to Matt Hoosier
Quoted 20 lines · Matt Hoosier
Quoted 19 lines · Andre McCurdy
Quoted 8 lines · Matt Hoosier

From: Matt Hoosier <[email protected]>

Although the submodules' histories have been fetched during the do_fetch() phase, the mechanics used to clone the workdir copy of the repo haven't been transferring the actual .git/modules directory from the repo fetched into downloads/ during the fetch task.

Quoted 5 lines

Fix that, and for good measure also explicitly tell Git to avoid hitting the network during do_unpack() of the submodules.

Ref:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12739

Patches for bitbake should be sent to the bitbake-devel mailing list:

Quoted 7 lines
Quoted 1 line

http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Right, okay. How's that meant to work for stuff developed in-tree with poky? Do you manually want the 'bitbake/' leading directory stripped off before emailing the patch?

If you're feeling lazy you can just send the patch to bitbake@ from a poky repo and git will figure it out. The same works the other way: a bitbake patch applies fine to a poky repository.

Ross