Create selftest suite for go-vendor.bbclass
v3 cover Create selftest suite for go-vendor.bbclass
This patch series adds a selftest suit for the go-vendor.bbclass and fixes a couple of bugs in the go-vendor.bbclass.
John Ripple (3):
go-vendor.bbclass: Fix test file exclusion and license propagation oeqa/selftest: Add go-vendor.bbclass selftest suite go-vendor.bbclass: Run vendor_unlink task under fakeroot
.../recipes-test/govendor/files/modules.txt | 7 + .../govendor/go-vendor-src-uri_1.0.bb | 17 ++ .../govendor/govendor-real-test_1.0.bb | 29 +++ meta/classes/go-vendor.bbclass | 9 +- meta/lib/oeqa/selftest/cases/govendor.py | 179 ++++++++++++++++++ 5 files changed, 237 insertions(+), 4 deletions(-) create mode 100644 meta-selftest/recipes-test/govendor/files/modules.txt create mode 100644 meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb create mode 100644 meta-selftest/recipes-test/govendor/govendor-real-test_1.0.bb create mode 100644 meta/lib/oeqa/selftest/cases/govendor.py
PATCH v3 1/3 go-vendor.bbclass: Fix test file exclusion and license propagation
Two bugs in do_go_vendor:
1. The shutil.ignore_patterns glob was "*._test.go" (with an extra dot), which never matches standard Go test files (*_test.go). Remove the extra dot.
2. The license-propagation block used os.path.join(src, "LICENSE") as the copy destination, writing into the already-processed vendor.fetch source tree rather than into the vendor destination directory. Switch to os.path.join(dst, "LICENSE") so the LICENSE lands in vendor/ where go tools and licence scanners expect it. Also fix the guard condition from not os.path.exists(subdir) (checking an unrelated relative name) to not os.path.exists(subdirLicense) (checking the actual target).
--- meta/classes/go-vendor.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/go-vendor.bbclass +3 -3
@@ -141,7 +141,7 @@ python do_go_vendor() { shutil.copytree(src, dst, symlinks=True, dirs_exist_ok=True, \ ignore=shutil.ignore_patterns(".git", \ "vendor", \ - "*._test.go")) + "*_test.go")) # If the root directory has a LICENSE file but not the subdir # we copy the root license to the sub module since the license
@@ -149,9 +149,9 @@ python do_go_vendor() { # see https://go.dev/ref/mod#vcs-license if subdir: rootdirLicese = os.path.join(rootdir, "LICENSE") - subdirLicense = os.path.join(src, "LICENSE") + subdirLicense = os.path.join(dst, "LICENSE") - if not os.path.exists(subdir) and \ + if not os.path.exists(subdirLicense) and \ os.path.exists(rootdirLicese): shutil.copy2(rootdirLicese, subdirLicense)
Re: [OE-core] [meta][PATCH v3 1/3] go-vendor.bbclass: Fix test file exclusion and license propagation
Hi John,
Thanks for your contribution.
Let me know if any changes are needed in the test repository as well.
Slava
Quoted 45 lines · John Ripple via lists.openembedded.org
Two bugs in
do_go_vendor:1. The shutil.ignore_patterns glob was "*._test.go" (with an extra dot), which never matches standard Go test files (*_test.go). Remove the extra dot.
2. The license-propagation block used
os.path.join(src, "LICENSE")as the copy destination, writing into the already-processed vendor.fetch source tree rather than into the vendor destination directory. Switch toos.path.join(dst, "LICENSE")so the LICENSE lands in vendor/ where go tools and licence scanners expect it. Also fix the guard condition from notos.path.exists(subdir)(checking an unrelated relative name) to notos.path.exists(subdirLicense)(checking the actual target).Signed-off-by John Ripple <[email protected]>--- meta/classes/go-vendor.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/classes/go-vendor.bbclass +3
-3@@ -141,7 +141,7 @@ python do_go_vendor() { shutil.copytree(src, dst, symlinks=True, dirs_exist_ok=True, \ ignore=shutil.ignore_patterns(".git", \ "vendor", \ - "*._test.go")) + "*_test.go")) # If the root directory has a LICENSE file but not the subdir # we copy the root license to the sub module since the license@@ -149,9 +149,9 @@ python do_go_vendor() { # see https://go.dev/ref/mod#vcs-license if subdir: rootdirLicese = os.path.join(rootdir, "LICENSE") - subdirLicense = os.path.join(src, "LICENSE") + subdirLicense = os.path.join(dst, "LICENSE") - if not os.path.exists(subdir) and \ + if not os.path.exists(subdirLicense) and \ os.path.exists(rootdirLicese): shutil.copy2(rootdirLicese, subdirLicense)
PATCH v3 2/3 oeqa/selftest: Add go-vendor.bbclass selftest suite
In reply to John RippleAdd govendor.py with two test classes that cover the behaviors fixed and defined in go-vendor.bbclass:
GoVendorSrcUriTests (offline, ~10 s): unit tests for go_src_uri() parsed via bb.tinfoil; verifies the basic URI shape, path/pathmajor/subdir/replaces encoding, and that non-git VCS omits the git-specific nobranch/protocol flags.
GoVendorTaskTests: runs do_go_vendor once against the real recipetool-go-test project and checks that external deps land in vendor/, *_test.go files are excluded, pathmajor=/v5 falls back to the repo root when no v5/ subdir exists, modules.txt is reproduced in vendor/, a local replacement (./is) appears as a symlink, and the S/src/GO_IMPORT/vendor -> S/src/import/vendor symlink is created.
Exclude the govendor test recipes from world builds since they are only meant to be built directly by the govendor test suite.
--- .../recipes-test/govendor/files/modules.txt | 7 + .../govendor/go-vendor-src-uri_1.0.bb | 17 ++ .../govendor/govendor-real-test_1.0.bb | 29 +++ meta/lib/oeqa/selftest/cases/govendor.py | 179 ++++++++++++++++++ 4 files changed, 232 insertions(+) create mode 100644 meta-selftest/recipes-test/govendor/files/modules.txt create mode 100644 meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb create mode 100644 meta-selftest/recipes-test/govendor/govendor-real-test_1.0.bb create mode 100644 meta/lib/oeqa/selftest/cases/govendor.py
meta-selftest/recipes-test/govendor/files/modules.txt new file +7 -0
@@ -0,0 +1,7 @@ +# github.com/godbus/dbus/v5 v5.1.0 +## explicit; go 1.12 +github.com/godbus/dbus/v5 +# github.com/matryer/is v1.4.1 => ./is +## explicit; go 1.14 +github.com/matryer/is +# github.com/matryer/is => ./is
meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb new file +17 -0
@@ -0,0 +1,17 @@ +SUMMARY = "Selftest fixture: go_src_uri parameter variants" +LICENSE = "CLOSED" + +# Parsed via tinfoil for its GO_SRC_URI_* variables only; never actually built +# (it has no real SRC_URI, so do_go_vendor would fail). +EXCLUDE_FROM_WORLD = "1" + +inherit go-vendor + +GO_IMPORT = "example.com/testpkg" + +GO_SRC_URI_BASIC = "${@go_src_uri('github.com/foo/bar', 'v1.0.0')}" +GO_SRC_URI_PATH = "${@go_src_uri('github.com/foo/bar', 'v2.1.0', path='github.com/foo/bar/v2')}" +GO_SRC_URI_PATHMAJOR = "${@go_src_uri('github.com/foo/bar', 'v2.0.0', pathmajor='/v2')}" +GO_SRC_URI_SUBDIR = "${@go_src_uri('github.com/foo/bar', 'v1.0.0', subdir='sub/pkg')}" +GO_SRC_URI_REPLACES = "${@go_src_uri('github.com/baz/qux', 'v1.0.0', replaces='../local/baz')}" +GO_SRC_URI_SVN = "${@go_src_uri('example.com/svnmod', 'r42', vcs='svn')}"
meta-selftest/recipes-test/govendor/govendor-real-test_1.0.bb new file +29 -0
@@ -0,0 +1,29 @@ +SUMMARY = "go-vendor integration test: recipetool-go-test" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://src/${GO_IMPORT}/LICENSE;md5=4e3933dd47afbf115e484d11385fb3bd" + +# Only meant to be built directly by GoVendorTaskTests (bitbake('%s -c go_vendor')), +# not as part of a general world build. +EXCLUDE_FROM_WORLD = "1" + +# Native variant lets GoVendorTaskTests.test_do_compile_succeeds exercise the +# real go toolchain without requiring a full target Go cross-toolchain. +BBCLASSEXTEND = "native" + +inherit go-vendor + +GO_IMPORT = "git.yoctoproject.org/recipetool-go-test" + +# modules.txt for recipetool-go-test at SRCREV: +# - github.com/godbus/dbus/v5 v5.1.0 (external, fetched via go_src_uri) +# - github.com/matryer/is v1.4.1 (local replacement => ./is) +# Use go mod vendor to generate the modules.txt +SRC_URI = "\ + git://${GO_IMPORT};name=recipetool-go-test;branch=main;protocol=https;nobranch=1;destsuffix=${GO_SRCURI_DESTSUFFIX} \ + ${@go_src_uri('github.com/godbus/dbus', 'v5.1.0', path='github.com/godbus/dbus/v5', pathmajor='/v5')} \ + file://modules.txt \ + " +SRCREV_FORMAT = "recipetool-go-test" +# git.yoctoproject.org/recipetool-go-test main branch HEAD +SRCREV_recipetool-go-test = "c3e213c01b6c1406b430df03ef0d1ae77de5d2f7" +SRCREV_github.com.godbus.dbus.v5 = "e523abc905595cf17fb0001a7d77eaaddfaa216d"
meta/lib/oeqa/selftest/cases/govendor.py new file +179 -0
179 lines not on this page · raw message
Re: [OE-core] [meta][PATCH v3 2/3] oeqa/selftest: Add go-vendor.bbclass selftest suite
Quoted 14 lines · John Ripple via lists.openembedded.org
Add
govendor.pywith two test classes that cover the behaviors fixed and defined in go-vendor.bbclass:GoVendorSrcUriTests (offline, ~10 s): unit tests for
go_src_uri()parsed via bb.tinfoil; verifies the basic URI shape, path/pathmajor/subdir/replaces encoding, and that non-git VCS omits the git-specific nobranch/protocol flags.GoVendorTaskTests: runs
do_go_vendoronce against the real recipetool-go-test project and checks that external deps land in vendor/, *_test.go files are excluded, pathmajor=/v5 falls back to the repo root when no v5/ subdir exists, modules.txt is reproduced in vendor/, a local replacement (./is) appears as a symlink, and the S/src/GO_IMPORT/vendor -> S/src/import/vendor symlink is created.
Quoted 5 lines
Exclude the govendor test recipes from world builds since they are only meant to be built directly by the govendor test suite.
Signed-off-by John Ripple <[email protected]>---
Hi John,
Thanks for your patch.
meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb line 2
Quoted 5 lines
meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb new file +2
-0@@ -0,0 +1,17 @@ +SUMMARY = "Selftest fixture: go_src_uri parameter variants" +LICENSE = "CLOSED"
Using "CLOSED" is now deprecated, if you rebase on today master you will see an error:
WARNING: /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-1874317/meta-selftest/recipes-test/govendor/go-vendor-src-uri_1.0.bb: go-vendor-src-uri: LICENSE is using "CLOSED", which is deprecated. Convert to using a license ref pointing to an actual license file, e.g. LICENSE = "LicenseRef-go-vendor-src-uri-CLOSED"
Can you send a new version of this series, with the licence fixed?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v3 3/3 go-vendor.bbclass: Run vendor_unlink task under fakeroot
In reply to John Rippledo_install and do_package both run under fakeroot (pseudo), which keeps pseudo's ownership/inode database in sync with what is actually on disk under D. do_vendor_unlink runs between them but was never marked fakeroot, so its os.unlink() of the vendor symlink happened outside of pseudo's view: the file disappeared from disk but pseudo's database still held a stale record for that path/inode.
The next fakeroot task to touch that path (do_package) then finds the on-disk state and pseudo's database disagreeing, and pseudo aborts with "path mismatch" / "inode mismatch" errors, failing do_package.
Mark do_vendor_unlink fakeroot, matching do_install and do_package, so its filesystem operations are tracked by pseudo consistently.
--- meta/classes/go-vendor.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/go-vendor.bbclass +2 -1
@@ -40,13 +40,14 @@ def go_src_uri(repo, version, path=None, subdir=None, \ return src_uri -python do_vendor_unlink() { +fakeroot python do_vendor_unlink() { go_import = d.getVar('GO_IMPORT') linkname = os.path.join(d.getVar('D') + d.getVar('libdir'), 'go', 'src', go_import, 'vendor') if os.path.islink(linkname): os.unlink(linkname) } +do_vendor_unlink[depends] += "virtual/fakeroot-native:do_populate_sysroot" addtask vendor_unlink before do_package after do_install python do_go_vendor() {