Capture git submodules in mirror archiver
v2 cover Capture git submodules in mirror archiver
Changes in v2:
* Use the git-submodule-test repo on git.yoctoproject.com instead of ovmf so that the test doesn't take forever.
* Fix test_archiver_mode_mirror_gitsm_shallow. As the git-submodule-test recipe isn't likely to change we can be explicit with file names instead of using globs.
Paul Barker (2):
selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
archiver: Capture git submodules in mirror archiver
.../git-submodule-test/git-submodule-test.bb | 8 +++ meta/classes/archiver.bbclass | 13 +++-- meta/lib/oeqa/selftest/cases/archiver.py | 54 +++++++++++++++++++ 3 files changed, 68 insertions(+), 7 deletions(-) create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
Signature · 1 line
2.26.2
PATCH v2 1/2 selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
--- .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8 -0
@@ -0,0 +1,8 @@ +SUMMARY = "Test recipe for fetching git submodules" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +INHIBIT_DEFAULT_DEPS = "1" + +SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test" +SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"
Signature · 1 line
2.26.2
Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
Hi Paul,
Quoted 12 lines · Paul Barker
Signed-off-by Paul Barker <[email protected]>--- .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bbmeta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +0
-0
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb line 8
Quoted 8 lines
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8
-0@@ -0,0 +1,8 @@ +SUMMARY = "Test recipe for fetching git submodules" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +INHIBIT_DEFAULT_DEPS = "1" + +SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test" +SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"
I didn't spot it at first but unfortunately this recipe is breaking the autobuilder. I think there is a problem with gitsm:// modules not mirroring correctly.
Here you see the a-quick build run:
bitbake universe -c fetch -k
(meta-selftest is included and SOURCE_MIRROR_FETCH = '1' set)
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b
you do see do_fetch called for this recipe. Then later in the build it does:
oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d
The intent is that we check YP can build from its mirrors. The test is say it cannot.
Any ideas on what might be wrong?
Cheers,
Richard
Re: [OE-core] [PATCH v2 1/2] selftest: git-submodule-test: New recipe for testing a gitsm SRC_URI
Quoted 39 lines · Richard Purdie
Hi Paul,
Quoted 12 lines · Paul Barker
Signed-off-by Paul Barker <[email protected]>--- .../recipes-test/git-submodule-test/git-submodule-test.bb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bbmeta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +0
-0meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb line 8
Quoted 8 lines
meta-selftest/recipes-test/git-submodule-test/git-submodule-test.bb new file +8
-0@@ -0,0 +1,8 @@ +SUMMARY = "Test recipe for fetching git submodules" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + +INHIBIT_DEFAULT_DEPS = "1" + +SRC_URI = "gitsm://git.yoctoproject.org/git-submodule-test" +SRCREV = "a2885dd7d25380d23627e7544b7bbb55014b16ee"I didn't spot it at first but unfortunately this recipe is breaking the autobuilder. I think there is a problem with gitsm:// modules not mirroring correctly.
Here you see the a-quick build run:
bitbake universe -c fetch -k
(meta-selftest is included andSOURCE_MIRROR_FETCH= '1' set)https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/10/logs/step1b
you do see
do_fetchcalled for this recipe. Then later in the build it does:
Quoted 8 lines
oe-selftest -r buildoptions.SourceMirroring.test_yocto_source_mirror
https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/1055/steps/12/logs/step1d
The intent is that we check YP can build from its mirrors. The test is say it cannot.
Any ideas on what might be wrong?
ovmf also uses git submodules and that seems to work. Also, I can see the relevant mirror tarballs are present on downloads.yoctoproject.org:
http://downloads.yoctoproject.org/mirror/sources/git2_git.openembedded.org.bitbake.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test1.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.bitbake-gitsm-test2.tar.gz
http://downloads.yoctoproject.org/mirror/sources/git2_git.yoctoproject.org.git-submodule-test.tar.gz
So it's not immediately obvious what's wrong there. I'll run that selftest case locally and see if I can reproduce the issue.
Thanks,
Signature · 2 lines
Paul Barker Konsulko Group
PATCH v2 2/2 archiver: Capture git submodules in mirror archiver
In reply to Paul BarkerUsing the new Fetch.expanded_urldata() function we can get URL data for all git submodules.
--- meta/classes/archiver.bbclass | 13 +++--- meta/lib/oeqa/selftest/cases/archiver.py | 54 ++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 7 deletions(-)
meta/classes/archiver.bbclass +6 -7
@@ -345,13 +345,12 @@ python do_ar_mirror() { fetcher = bb.fetch2.Fetch(src_uri, d) - for url in fetcher.urls: - if is_excluded(url): - bb.note('Skipping excluded url: %s' % (url)) + for ud in fetcher.expanded_urldata(): + if is_excluded(ud.url): + bb.note('Skipping excluded url: %s' % (ud.url)) continue - bb.note('Archiving url: %s' % (url)) - ud = fetcher.ud[url] + bb.note('Archiving url: %s' % (ud.url)) ud.setup_localpath(d) localpath = None
@@ -367,7 +366,7 @@ python do_ar_mirror() { if len(ud.mirrortarballs) and not localpath: bb.warn('Mirror tarballs are listed for a source but none are present. ' \ 'Falling back to original download.\n' \ - 'SRC_URI = %s' % (url)) + 'SRC_URI = %s' % (ud.url)) # Check original download if not localpath:
@@ -376,7 +375,7 @@ python do_ar_mirror() { if not localpath or not os.path.exists(localpath): bb.fatal('Original download is missing for a source.\n' \ - 'SRC_URI = %s' % (url)) + 'SRC_URI = %s' % (ud.url)) # We now have an appropriate localpath bb.note('Copying source mirror')
meta/lib/oeqa/selftest/cases/archiver.py +54 -0
57 lines not on this page · raw message
Signature · 1 line
2.26.2