Skip to contentSkip to conversation

Capture git submodules in mirror archiver

PATCH In review Paul Barker 2 participants 5 messages Jun 4, 2020 – Jun 12, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover Capture git submodules in mirror archiver

Changes in v2:

* Use the git-submodule-test repo on git.yoctoproject.com instead of ovmf so that the test doesn't take forever.

* Fix test_archiver_mode_mirror_gitsm_shallow. As the git-submodule-test recipe isn't likely to change we can be explicit with file names instead of using globs.

Paul Barker (2):
selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
archiver: Capture git submodules in mirror archiver

 .../git-submodule-test/git-submodule-test.bb  |  8 +++
 meta/classes/archiver.bbclass                 | 13 +++--
 meta/lib/oeqa/selftest/cases/archiver.py      | 54 +++++++++++++++++++
 3 files changed, 68 insertions(+), 7 deletions(-)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
Signature · 1 line
2.26.2
#1 lore raw

PATCH v2 1/2 selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

---
 .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8 -0
new file mode 100644
index 0000000000..08089b4186
@@ -0,0 +1,8 @@
+SUMMARY = "Test recipe for fetching git submodules"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
+SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"
Signature · 1 line
2.26.2
#2 lore raw

Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

Hi Paul,

Quoted 12 lines · Paul Barker
---
 .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +0 -0
new file mode 100644
index 0000000000..08089b4186

meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb line 8

Quoted 8 lines
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8 -0
@@ -0,0 +1,8 @@
+SUMMARY = "Test recipe for fetching git submodules"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
+SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"

I didn't spot it at first but unfortunately this recipe is breaking the autobuilder. I think there is a problem with gitsm:// modules not mirroring correctly.

Here you see the a-quick build run:

bitbake universe -c fetch -k
(meta-selftest is included and SOURCE_MIRROR_FETCH = '1' set)

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b

you do see do_fetch called for this recipe. Then later in the build it does:

oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d

The intent is that we check YP can build from its mirrors. The test is say it cannot.

Any ideas on what might be wrong?

Cheers,

Richard

#3 lore raw

Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI

Quoted 39 lines · Richard Purdie

Hi Paul,

Quoted 12 lines · Paul Barker
---
 .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++
 1 file changed, 8 insertions(+)
 create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +0 -0
new file mode 100644
index 0000000000..08089b4186

meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb line 8

Quoted 8 lines
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8 -0
@@ -0,0 +1,8 @@
+SUMMARY = "Test recipe for fetching git submodules"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test"
+SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"

I didn't spot it at first but unfortunately this recipe is breaking the autobuilder. I think there is a problem with gitsm:// modules not mirroring correctly.

Here you see the a-quick build run:

bitbake universe -c fetch -k
(meta-selftest is included and SOURCE_MIRROR_FETCH = '1' set)

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b

you do see do_fetch called for this recipe. Then later in the build it does:

Quoted 8 lines

oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror

https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d

The intent is that we check YP can build from its mirrors. The test is say it cannot.

Any ideas on what might be wrong?

ovmf also uses git submodules and that seems to work. Also, I can see the relevant mirror tarballs are present on downloads.yoctoproject.org:

http://downloads.yoctoproject.org/mirror/sources/git2_git.openembedded.org.bitbake.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test1.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test2.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.git-submodule-test.tar.gz

So it's not immediately obvious what's wrong there. I'll run that selftest case locally and see if I can reproduce the issue.

Thanks,

Signature · 2 lines
Paul Barker
Konsulko Group
#4 lore raw

PATCH v2 2/2 archiver: Capture git submodules in mirror archiver

In reply to Paul Barker

Using the new Fetch.expanded_urldata() function we can get URL data for all git submodules.

---
 meta/classes/archiver.bbclass            | 13 +++---
 meta/lib/oeqa/selftest/cases/archiver.py | 54 ++++++++++++++++++++++++
 2 files changed, 60 insertions(+), 7 deletions(-)
meta/classes/archiver.bbclass +6 -7
index 780c562b68..c2c049c343 100644
@@ -345,13 +345,12 @@ python do_ar_mirror() {
 
     fetcher = bb.fetch2.Fetch(src_uri, d)
 
-    for url in fetcher.urls:
-        if is_excluded(url):
-            bb.note('Skipping excluded url: %s' % (url))
+    for ud in fetcher.expanded_urldata():
+        if is_excluded(ud.url):
+            bb.note('Skipping excluded url: %s' % (ud.url))
             continue
 
-        bb.note('Archiving url: %s' % (url))
-        ud = fetcher.ud[url]
+        bb.note('Archiving url: %s' % (ud.url))
         ud.setup_localpath(d)
         localpath = None
 
@@ -367,7 +366,7 @@ python do_ar_mirror() {
         if len(ud.mirrortarballs) and not localpath:
             bb.warn('Mirror tarballs are listed for a source but none are present. ' \
                     'Falling back to original download.\n' \
-                    'SRC_URI = %s' % (url))
+                    'SRC_URI = %s' % (ud.url))
 
         # Check original download
         if not localpath:
@@ -376,7 +375,7 @@ python do_ar_mirror() {
 
         if not localpath or not os.path.exists(localpath):
             bb.fatal('Original download is missing for a source.\n' \
-                        'SRC_URI = %s' % (url))
+                        'SRC_URI = %s' % (ud.url))
 
         # We now have an appropriate localpath
         bb.note('Copying source mirror')
meta/lib/oeqa/selftest/cases/archiver.py +54 -0

57 lines not on this page · raw message

Signature · 1 line
2.26.2