Create selftest suite for go-vendor.bbclass
cover Create selftest suite for go-vendor.bbclass
This patch series adds a selftest suit for the go-vendor.bbclass and fixes a couple of bugs in the go-vendor.bbclass and the selftest suit itself.
John Ripple (3):
go-vendor.bbclass: Fix test file exclusion and license propagation oeqa/selftest: Add go-vendor.bbclass selftest suite oeqa/utils/metadata: Handle empty git repos in git_rev_info
meta/classes/go-vendor.bbclass | 6 +- meta/lib/oeqa/selftest/cases/govendor.py | 264 +++++++++++++++++++++++ meta/lib/oeqa/utils/metadata.py | 11 +- 3 files changed, 274 insertions(+), 7 deletions(-) create mode 100644 meta/lib/oeqa/selftest/cases/govendor.py
PATCH 1/3 go-vendor.bbclass: Fix test file exclusion and license propagation
Two bugs in do_go_vendor:
1. The shutil.ignore_patterns glob was "*._test.go" (with an extra dot), which never matches standard Go test files (*_test.go). Remove the extra dot.
2. The license-propagation block used os.path.join(src, "LICENSE") as the copy destination, writing into the already-processed vendor.fetch source tree rather than into the vendor destination directory. Switch to os.path.join(dst, "LICENSE") so the LICENSE lands in vendor/ where go tools and licence scanners expect it. Also fix the guard condition from not os.path.exists(subdir) (checking an unrelated relative name) to not os.path.exists(subdirLicense) (checking the actual target).
--- meta/classes/go-vendor.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes/go-vendor.bbclass +3 -3
@@ -141,7 +141,7 @@ python do_go_vendor() { shutil.copytree(src, dst, symlinks=True, dirs_exist_ok=True, \ ignore=shutil.ignore_patterns(".git", \ "vendor", \ - "*._test.go")) + "*_test.go")) # If the root directory has a LICENSE file but not the subdir # we copy the root license to the sub module since the license
@@ -149,9 +149,9 @@ python do_go_vendor() { # see https://go.dev/ref/mod#vcs-license if subdir: rootdirLicese = os.path.join(rootdir, "LICENSE") - subdirLicense = os.path.join(src, "LICENSE") + subdirLicense = os.path.join(dst, "LICENSE") - if not os.path.exists(subdir) and \ + if not os.path.exists(subdirLicense) and \ os.path.exists(rootdirLicese): shutil.copy2(rootdirLicese, subdirLicense)
PATCH 2/3 oeqa/selftest: Add go-vendor.bbclass selftest suite
In reply to John RippleAdd govendor.py with two test classes that cover the behaviors fixed and defined in go-vendor.bbclass:
GoVendorSrcUriTests (offline, ~10 s): unit tests for go_src_uri() parsed via bb.tinfoil; verifies the basic URI shape, path/pathmajor/subdir/replaces encoding, and that non-git VCS omits the git-specific nobranch/protocol flags.
GoVendorTaskTests: runs do_go_vendor once against the real recipetool-go-test project and checks that external deps land in vendor/, *_test.go files are excluded, pathmajor=/v5 falls back to the repo root when no v5/ subdir exists, modules.txt is reproduced in vendor/, a local replacement (./is) appears as a symlink, and the S/src/GO_IMPORT/vendor -> S/src/import/vendor symlink is created.
--- meta/lib/oeqa/selftest/cases/govendor.py | 264 +++++++++++++++++++++++ 1 file changed, 264 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/govendor.py
meta/lib/oeqa/selftest/cases/govendor.py new file +264 -0
@@ -0,0 +1,264 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# +# Comprehensive tests for go-vendor.bbclass: +# GoVendorSrcUriTests -- unit tests for the go_src_uri() helper (offline, ~10 s) +# GoVendorTaskTests -- integration tests using the real recipetool-go-test project +# (network fetch of git.yoctoproject.org + github.com/godbus/dbus) + +import os +import shutil +import tempfile + +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer + +# Module-level temp layer shared by both test classes +templayerdir = None + + +def setUpModule(): + global templayerdir + templayerdir = tempfile.mkdtemp(prefix='govendorqa') + create_temp_layer(templayerdir, 'selftestgovendor') + runCmd('bitbake-layers add-layer %s' % templayerdir) + + +def tearDownModule(): + runCmd('bitbake-layers remove-layer %s' % templayerdir, ignore_status=True) + runCmd('rm -rf %s' % templayerdir) + + +# --------------------------------------------------------------------------- +# Class 1: go_src_uri() helper function +# --------------------------------------------------------------------------- + +class GoVendorSrcUriTests(OESelftestTestCase): + """Tests for the go_src_uri() Python helper in go-vendor.bbclass. + + Parses a single fixture recipe via bb.tinfoil (config_only=True) once per + class so the function runs inside the real BitBake class-loading machinery. + All tests are offline and complete in ~10 s total. + """ + + @classmethod + def setUpClass(cls): + import bb.tinfoil + super().setUpClass() + + recipe_dir = os.path.join(templayerdir, 'recipes-test-src-uri', + 'go-vendor-src-uri') + os.makedirs(recipe_dir, exist_ok=True) + recipe_path = os.path.join(recipe_dir, 'go-vendor-src-uri_1.0.bb') + + with open(recipe_path, 'w') as f: + f.write( + 'SUMMARY = "Selftest fixture: go_src_uri parameter variants"\n' + 'LICENSE = "CLOSED"\n' + '\n' + 'inherit go-vendor\n' + '\n' + 'GO_IMPORT = "example.com/testpkg"\n' + '\n' + 'GO_SRC_URI_BASIC = "${@go_src_uri(\'github.com/foo/bar\', \'v1.0.0\')}"\n' + 'GO_SRC_URI_PATH = "${@go_src_uri(\'github.com/foo/bar\', \'v2.1.0\', path=\'github.com/foo/bar/v2\')}"\n' + 'GO_SRC_URI_PATHMAJOR = "${@go_src_uri(\'github.com/foo/bar\', \'v2.0.0\', pathmajor=\'/v2\')}"\n' + 'GO_SRC_URI_SUBDIR = "${@go_src_uri(\'github.com/foo/bar\', \'v1.0.0\', subdir=\'sub/pkg\')}"\n' + 'GO_SRC_URI_REPLACES = "${@go_src_uri(\'github.com/baz/qux\', \'v1.0.0\', replaces=\'../local/baz\')}"\n' + 'GO_SRC_URI_SVN = "${@go_src_uri(\'example.com/svnmod\', \'r42\', vcs=\'svn\')}"\n' + ) + + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=True, quiet=2) + rd = tinfoil.parse_recipe_file(recipe_path) + cls.bp = rd.getVar('BP') + cls.uris = {k: rd.getVar('GO_SRC_URI_' + k) + for k in ('BASIC', 'PATH', 'PATHMAJOR', 'SUBDIR', + 'REPLACES', 'SVN')} + + @classmethod + def tearDownClass(cls): + runCmd('rm -rf %s/recipes-test-src-uri' % templayerdir, + ignore_status=True) + super().tearDownClass() + + # --- individual test methods ------------------------------------------- + + def test_basic(self): + """destsuffix uses expanded BP""" + uri = self.uris['BASIC'] + expected_ds = 'destsuffix=%s/src/import/vendor.fetch/github.com/foo/[email protected]' % self.bp + self.assertIn(expected_ds, uri) + self.assertIn('go_module_path=github.com/foo/bar', uri) + self.assertIn('nobranch=1', uri) + self.assertIn('protocol=https', uri) + self.assertIn('is_go_dependency=1', uri) + + def test_path_override(self): + """Explicit path= sets go_module_path independently of the repo URL.""" + uri = self.uris['PATH'] + self.assertIn('go_module_path=github.com/foo/bar/v2', uri) + expected_ds = 'destsuffix=%s/src/import/vendor.fetch/github.com/foo/[email protected]' % self.bp + self.assertIn(expected_ds, uri) + + def test_pathmajor(self): + """pathmajor is encoded as go_pathmajor in the URI.""" + self.assertIn('go_pathmajor=/v2', self.uris['PATHMAJOR']) + + def test_subdir(self): + """subdir is encoded as go_subdir in the URI.""" + self.assertIn('go_subdir=sub/pkg', self.uris['SUBDIR']) + + def test_replaces(self): + """replaces is encoded as go_module_replacement in the URI.""" + self.assertIn('go_module_replacement=../local/baz', self.uris['REPLACES']) + + def test_non_git_vcs(self): + """Non-git VCS omits the git-specific nobranch and protocol=https flags.""" + uri = self.uris['SVN'] + self.assertIn('svn://example.com/svnmod', uri) + self.assertNotIn('nobranch', uri) + self.assertNotIn('protocol=https', uri) + self.assertIn('is_go_dependency=1', uri) + + +# --------------------------------------------------------------------------- +# Class 2: do_go_vendor task integration — real project +# --------------------------------------------------------------------------- + +class GoVendorTaskTests(OESelftestTestCase): + """Integration tests for do_go_vendor using the real recipetool-go-test project. + + The recipe fetches git.yoctoproject.org/recipetool-go-test (commit + c3e213c) as its main source and github.com/godbus/dbus/v5 v5.1.0 as an + external vendor dependency. The project also has a local module + replacement (github.com/matryer/is => ./is) that lives inside the main + source tree. + + do_go_vendor runs ONCE in setUpClass; individual test methods inspect the + resulting vendor directory without re-running bitbake. + + Verified behaviors: + - External dependency files are copied to vendor/ + - *_test.go files are excluded from the vendor copy + - pathmajor fallback: when the /v5 subdir is absent, repo root is used + - modules.txt is reproduced inside vendor/ + - Local module replacement creates a relative symlink in vendor/ + - S/src/GO_IMPORT/vendor symlink is created pointing at S/src/import/vendor + """ + + # git.yoctoproject.org/recipetool-go-test main branch HEAD + SRCREV = 'c3e213c01b6c1406b430df03ef0d1ae77de5d2f7' + GO_IMPORT = 'git.yoctoproject.org/recipetool-go-test' + RECIPE_NAME = 'govendor-real-test' + + @classmethod + def setUpClass(cls): + from oeqa.utils import ftools + super().setUpClass() + + # overlayfs-user.bb in meta-selftest requires a machine-specific mount + # point that this test environment does not provide. Mask it so that + # bitbake can parse the full recipe universe without halting. + ftools.append_file(cls.testinc_path, + 'BBMASK += "meta-selftest/recipes-test/overlayfs-user"') + + recipe_dir = os.path.join(templayerdir, 'recipes-test', cls.RECIPE_NAME) + files_dir = os.path.join(recipe_dir, 'files') + os.makedirs(files_dir, exist_ok=True) + + # modules.txt for recipetool-go-test at SRCREV: + # - github.com/godbus/dbus/v5 v5.1.0 (external, fetched via go_src_uri) + # - github.com/matryer/is v1.4.1 (local replacement => ./is) + with open(os.path.join(files_dir, 'modules.txt'), 'w') as fh: + fh.write( + '# github.com/godbus/dbus/v5 v5.1.0\n' + '## explicit; go 1.18\n' + 'github.com/godbus/dbus/v5\n' + '# github.com/matryer/is v1.4.1 => ./is\n' + '## explicit; go 1.14\n' + 'github.com/matryer/is\n' + ) + + with open(os.path.join(recipe_dir, '%s_1.0.bb' % cls.RECIPE_NAME), 'w') as fh: + fh.write( + 'SUMMARY = "go-vendor integration test: recipetool-go-test"\n' + 'LICENSE = "MIT"\n' + 'LIC_FILES_CHKSUM = "file://src/${GO_IMPORT}/LICENSE;md5=4e3933dd47afbf115e484d11385fb3bd"\n' + '\n' + 'inherit go-vendor\n' + '\n' + 'GO_IMPORT = "%s"\n' + '\n' + 'SRC_URI = "\\\n' + ' git://${GO_IMPORT};name=recipetool-go-test;branch=main;protocol=https' + ';nobranch=1;destsuffix=${GO_SRCURI_DESTSUFFIX} \\\n' + ' ${@go_src_uri(\'github.com/godbus/dbus\', \'v5.1.0\', \\\n' + ' path=\'github.com/godbus/dbus/v5\', pathmajor=\'/v5\')} \\\n' + ' file://modules.txt \\\n' + ' "\n' + 'SRCREV_FORMAT = "recipetool-go-test"\n' + 'SRCREV_recipetool-go-test = "%s"\n' + 'SRCREV_github.com.godbus.dbus.v5 = "e523abc905595cf17fb0001a7d77eaaddfaa216d"\n' + % (cls.GO_IMPORT, cls.SRCREV) + ) + + bitbake('%s -c go_vendor' % cls.RECIPE_NAME) + cls.s_dir = get_bb_var('S', cls.RECIPE_NAME) + cls.vendor_dir = os.path.join(cls.s_dir, 'src', 'import', 'vendor') + + @classmethod + def tearDownClass(cls): + runCmd('rm -rf %s/recipes-test' % templayerdir, ignore_status=True) + super().tearDownClass() + + # --- test methods (all inspect cls.vendor_dir, no bitbake) --------------- + + def test_external_dep_copied_to_vendor(self): + """godbus/dbus/v5 source files are copied into vendor/.""" + dbus_dir = os.path.join(self.vendor_dir, 'github.com', 'godbus', 'dbus', 'v5') + self.assertTrue(os.path.isdir(dbus_dir), + 'vendor/github.com/godbus/dbus/v5/ not created') + self.assertTrue(os.path.isfile(os.path.join(dbus_dir, 'conn.go')), + 'conn.go not found in vendor/github.com/godbus/dbus/v5/') + + def test_test_files_excluded_from_vendor(self): + """*_test.go files from godbus/dbus are not copied into vendor/.""" + dbus_dir = os.path.join(self.vendor_dir, 'github.com', 'godbus', 'dbus', 'v5') + self.assertFalse( + os.path.isfile(os.path.join(dbus_dir, 'conn_test.go')), + 'conn_test.go should have been excluded from vendor by *_test.go pattern') + + def test_pathmajor_falls_back_to_root(self): + """godbus/dbus v5.1.0 has no v5/ subdir; pathmajor=/v5 falls back to root.""" + dbus_dir = os.path.join(self.vendor_dir, 'github.com', 'godbus', 'dbus', 'v5') + # auth.go is at the repo root of godbus/dbus, not inside a v5/ subdir. + self.assertTrue(os.path.isfile(os.path.join(dbus_dir, 'auth.go')), + 'auth.go not found; pathmajor=/v5 fallback to repo root did not work') + + def test_modules_txt_in_vendor(self): + """modules.txt is reproduced inside vendor/ and lists the external dep.""" + modules_txt = os.path.join(self.vendor_dir, 'modules.txt') + self.assertTrue(os.path.isfile(modules_txt)) + with open(modules_txt) as fh: + content = fh.read() + self.assertIn('github.com/godbus/dbus/v5 v5.1.0', content) + + def test_local_replacement_symlink(self): + """go.mod local replacement (./is) creates a symlink in vendor/ to S/src/GO_IMPORT/is.""" + symlink = os.path.join(self.vendor_dir, 'github.com', 'matryer', 'is') + self.assertTrue(os.path.islink(symlink), + 'vendor/github.com/matryer/is should be a symlink for the ./is local replacement') + expected_target = os.path.join(self.s_dir, 'src', self.GO_IMPORT, 'is') + self.assertEqual(os.path.realpath(symlink), os.path.realpath(expected_target), + 'Local replacement symlink does not point to S/src/GO_IMPORT/is') + + def test_go_import_vendor_symlink(self): + """do_go_vendor creates S/src/GO_IMPORT/vendor -> S/src/import/vendor.""" + link = os.path.join(self.s_dir, 'src', self.GO_IMPORT, 'vendor') + self.assertTrue(os.path.islink(link), + 'S/src/GO_IMPORT/vendor symlink not created') + self.assertEqual(os.path.realpath(link), os.path.realpath(self.vendor_dir), + 'S/src/GO_IMPORT/vendor does not point to S/src/import/vendor')
PATCH 3/3 oeqa/utils/metadata: Handle empty git repos in git_rev_info
In reply to John Ripplegit_rev_info() calls repo.head.commit without guarding against ValueError, which gitpython raises when HEAD points to an unborn branch (e.g. a freshly-initialised layer directory with no commits). Wrap the commit-info block in try/except so metadata collection continues rather than crashing with a non-zero exit code.
--- meta/lib/oeqa/utils/metadata.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
meta/lib/oeqa/utils/metadata.py +7 -4
@@ -85,12 +85,15 @@ def git_rev_info(path): repo = Repo(path, search_parent_directories=True) except (InvalidGitRepositoryError, NoSuchPathError): return info - info['commit'] = repo.head.commit.hexsha - info['commit_count'] = repo.head.commit.count() - info['commit_time'] = repo.head.commit.committed_date + try: + info['commit'] = repo.head.commit.hexsha + info['commit_count'] = repo.head.commit.count() + info['commit_time'] = repo.head.commit.committed_date + except (ValueError, TypeError): + pass try: info['branch'] = repo.active_branch.name - except TypeError: + except (TypeError, ValueError): info['branch'] = '(nobranch)' return info
Re: [OE-core] [meta][PATCH 2/3] oeqa/selftest: Add go-vendor.bbclass selftest suite
In reply to John Ripplemeta/lib/oeqa/selftest/cases/govendor.py line 31
Quoted 10 lines · John Ripple via lists.openembedded.org
meta/lib/oeqa/selftest/cases/govendor.py new file +10
-0+def setUpModule(): + global templayerdir + templayerdir = tempfile.mkdtemp(prefix='govendorqa') + create_temp_layer(templayerdir, 'selftestgovendor') + runCmd('bitbake-layers add-layer %s' % templayerdir) + + +def tearDownModule(): + runCmd('bitbake-layers remove-layer %s' % templayerdir, ignore_status=True) + runCmd('rm -rf %s' % templayerdir)
There is no need to create a new layer, and place files into it in the test. There's meta-selftest in oe-core, and it exists exactly for this purpose. As far as I see they're all static files, and do not get modified inside the test.
Alex
Re: [OE-core] [meta][PATCH 3/3] oeqa/utils/metadata: Handle empty git repos in git_rev_info
In reply to John Ripplemeta/lib/oeqa/utils/metadata.py def git_rev_info(path): · line 93
Quoted 6 lines · John Ripple via lists.openembedded.org
meta/lib/oeqa/utils/metadata.py +6
-0+ try: + info['commit'] = repo.head.commit.hexsha + info['commit_count'] = repo.head.commit.count() + info['commit_time'] = repo.head.commit.committed_date + except (ValueError, TypeError): + pass
Suppressing generic exceptions like this to avoid a specific problem is problematic, as it can quietly suppress different, real problems. I'd say any layers handled by this function must have a commit, and if they don't it's a user error. How did this come up?
Alex