Skip to contentSkip to conversation

recipetool: Don't fail on local go modules

PATCH Superseded Vyacheslav Yurkov 3 participants 7 messages Jan 10, 2024 – Jan 16, 2024

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 recipetool: Don't fail on local go modules

Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI.

---
 scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------
 1 file changed, 19 insertions(+), 13 deletions(-)
scripts/lib/recipetool/create_go.py +19 -13
index 21dcb41271..bd20b07f91 100644
@@ -504,7 +504,7 @@ class GoRecipeHandler(RecipeHandler):
 
         return inline_fcn, commit
 
-    def __go_handle_dependencies(self, go_mod, localfilesdir, extravalues, d):
+    def __go_handle_dependencies(self, go_mod, srctree, localfilesdir, extravalues, d):
 
         src_uris = []
         src_revs = []
@@ -525,6 +525,23 @@ class GoRecipeHandler(RecipeHandler):
 
             return src_rev
 
+        # we first go over replacement list, because we are essentialy
+        # interested only in the replaced path
+        if go_mod['Replace']:
+            for replacement in go_mod['Replace']:
+                oldpath = replacement['Old']['Path']
+                path = replacement['New']['Path']
+                if 'Version' in replacement['New']:
+                    version = replacement['New']['Version']
+
+                if os.path.exists(os.path.join(srctree, path)):
+                    # the module refers to the local path, remove it from requirement list
+                    # because it's a local module
+                    go_mod['Require'][:] = [v for v in go_mod['Require'] if v.get('Path') != oldpath]
+                else:
+                    # Replace the path and the version, so we don't iterate replacement list anymore
+                    go_mod['Require']['Path'] = path
+
         for require in go_mod['Require']:
             path = require['Path']
             version = require['Version']
@@ -534,17 +551,6 @@ class GoRecipeHandler(RecipeHandler):
             src_uris.append(inline_fcn)
             src_revs.append(generate_src_rev(path, version, commithash))
 
-        if go_mod['Replace']:
-            for replacement in go_mod['Replace']:
-                oldpath = replacement['Old']['Path']
-                path = replacement['New']['Path']
-                version = replacement['New']['Version']
-
-                inline_fcn, commithash = self.__generate_srcuri_inline_fcn(
-                    path, version, oldpath)
-                src_uris.append(inline_fcn)
-                src_revs.append(generate_src_rev(path, version, commithash))
-
         pn, _ = determine_from_url(go_mod['Module']['Path'])
         go_mods_basename = "%s-modules.inc" % pn
 
@@ -693,7 +699,7 @@ class GoRecipeHandler(RecipeHandler):
 
             self.__rewrite_src_uri(lines_before, ["file://modules.txt"])
 
-            self.__go_handle_dependencies(go_mod, localfilesdir, extravalues, d)
+            self.__go_handle_dependencies(go_mod, srctree, localfilesdir, extravalues, d)
             lines_before.append("require ${BPN}-modules.inc")
 
         # Do generic license handling
Signature · 1 line
2.35.1
#1 lore raw

PATCH 2/3 classes: go-vendor: Reference local modules

Create symlinks for local modules, which are usually not referenced in the SRC_URI, but still expected to be found in the vendor directory during the build.

---
 meta/classes/go-vendor.bbclass | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
meta/classes/go-vendor.bbclass +17 -1
index 2426bddfba..b965428dd1 100644
@@ -169,6 +169,7 @@ python do_go_vendor() {
     fetched_paths.remove('.')
 
     vendored_paths = set()
+    replaced_paths = dict()
     with open(modules_txt_src) as f:
         for line in f:
             if not line.startswith("#"):
@@ -182,6 +183,15 @@ python do_go_vendor() {
                         vendored_paths.add(topdir)
 
                     topdir = os.path.dirname(topdir)
+            else:
+                replaced_module = line.split("=>")
+                if len(replaced_module) > 1:
+                    # This module has been replaced, use a local path
+                    # we parse the line that has a pattern "# module-name [module-version] => local-path
+                    actual_path = replaced_module[1].strip()
+                    vendored_name = replaced_module[0].split()[1]
+                    bb.debug(1, "added vendored name %s for actual path %s" % (vendored_name, actual_path))
+                    replaced_paths[vendored_name] = actual_path
 
     for path in fetched_paths:
         if path not in vendored_paths:
@@ -189,7 +199,13 @@ python do_go_vendor() {
             if os.path.exists(realpath):
                 shutil.rmtree(realpath)
 
-    # Create a symlink the the actual directory
+    for vendored_name, replaced_path in replaced_paths.items():
+        symlink_target = os.path.join(source_dir, *['src', go_import, replaced_path])
+        symlink_name = os.path.join(vendor_dir, vendored_name)
+        bb.debug(1, "vendored name %s, symlink name %s" % (vendored_name, symlink_name))
+        os.symlink(symlink_target, symlink_name)
+
+    # Create a symlink to the actual directory
     os.symlink(vendor_dir, linkname)
 }
 
Signature · 1 line
2.35.1
#2 lore raw

PATCH 3/3 classes: go-vendor: Handle modules from the same repo

In reply to Vyacheslav Yurkov

Take into account module version when populating vendor directory, because a module with the same URL but with a different version tag could be used as an indirect dependency.

---
 meta/classes/go-vendor.bbclass | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
meta/classes/go-vendor.bbclass +4 -5
index b965428dd1..48f4d1041c 100644
@@ -103,18 +103,17 @@ python do_go_vendor() {
         pathMajor = fetcher.ud[url].parm.get('go_pathmajor')
         pathMajor = None if not pathMajor else pathMajor.strip('/')
 
-        if not repo in modules:
-            modules[repo] =   { "version": version,
+        if not (repo, version) in modules:
+            modules[(repo, version)] =   {
                                 "repo_path": os.path.join(import_dir, p),
                                 "module_path": module_path,
                                 "subdir": subdir,
                                 "pathMajor": pathMajor }
 
-    for module_key in sorted(modules):
+    for module_key, module in modules.items():
 
         # only take the version which is explicitly listed
         # as a dependency in the go.mod
-        module = modules[module_key]
         module_path = module['module_path']
         rootdir = module['repo_path']
         subdir = module['subdir']
@@ -139,7 +138,7 @@ python do_go_vendor() {
         dst = os.path.join(vendor_dir, module_path)
 
         bb.debug(1, "cp %s --> %s" % (src, dst))
-        shutil.copytree(src, dst, symlinks=True, \
+        shutil.copytree(src, dst, symlinks=True, dirs_exist_ok=True, \
             ignore=shutil.ignore_patterns(".git", \
                                             "vendor", \
                                             "*._test.go"))
Signature · 1 line
2.35.1
#3 lore raw
In reply to Vyacheslav Yurkov
Quoted 7 lines · Vyacheslav Yurkov

Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI.

---
 scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------
 1 file changed, 19 insertions(+), 13 deletions(-)

Do these changes mean we need to improve the test coverage in oe- selftest for recipetool?

Cheers,

Richard

#4 lore raw
Quoted 11 lines · Richard Purdie
Quoted 7 lines · Vyacheslav Yurkov

Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI.

---
  scripts/lib/recipetool/create_go.py | 32 +++++++++++++++++------------
  1 file changed, 19 insertions(+), 13 deletions(-)

Do these changes mean we need to improve the test coverage in oe- selftest for recipetool?

Test converage needs improvement, yes. I haven't considered the case for local modules since this seems to be rare(?). Are there public projects which use this mechanism?

Best regards
Lukas

Quoted 4 lines

Cheers,

Richard

#5 lore raw
Quoted 11 lines · Lukas Funke
Quoted 10 lines · Richard Purdie
Quoted 8 lines · Vyacheslav Yurkov

Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI.

---
  scripts/lib/recipetool/create_go.py | 32
+++++++++++++++++------------

  1 file changed, 19 insertions(+), 13 deletions(-)
Quoted 8 lines
Quoted 2 lines

Do these changes mean we need to improve the test coverage in oe- selftest for recipetool?

Test converage needs improvement, yes. I haven't considered the case for local modules since this seems to be rare(?). Are there public projects which use this mechanism?

Best regards
Lukas

I added coverage for local modules in my v2.

I'm not a golang expert, but it seems that local modules in golang is a means to "place a patched version into the project space until upstream fixes it" more or less like Yocto does with Upstream patches. It's used a lot by in-house projects, but you could also get a glimpse on github with this query https://github.com/search?q=path%3A%2Fgo.mod+NOT+is%3Afork+content%3A%2F%5Ereplace+.*%2F&type=code&query=content%3Ago.mod+replace&p=3 . Note, Github is able to display only first 100 search results (Github API restriction).

One thing I don't like in my v2 is that I need to look up a public project to add required coverage. I would rather improve tests further by having everything in one repository and just one test, i.e. it has to be a small test project that could be used to test all go-related functionality. The question is though where it should be placed? I could create it on my Github account, but would that be a good option?

Richard, any suggestions on where these projects for selftest coverage should be hosted?

Slava

#6 lore raw
Quoted 41 lines · Vyacheslav Yurkov
Quoted 14 lines · Lukas Funke
Quoted 4 lines · Richard Purdie
Quoted 3 lines · Vyacheslav Yurkov

Local modules are usually referenced with a 'replace' directive in go.mod file. If that's the case, remove them from populating SRC_URI.

Quoted 8 lines
Quoted 5 lines

---
  scripts/lib/recipetool/create_go.py | 32
+++++++++++++++++------------

  1 file changed, 19 insertions(+), 13 deletions(-)

Do these changes mean we need to improve the test coverage in oe- selftest for recipetool?

Quoted 7 lines

Test converage needs improvement, yes. I haven't considered the case for local modules since this seems to be rare(?). Are there public projects which use this mechanism?

Best regards
Lukas

I added coverage for local modules in my v2.

I'm not a golang expert, but it seems that local modules in golang is a means to "place a patched version into the project space until upstream fixes it" more or less like Yocto does with Upstream patches. It's used a lot by in-house projects, but you could also get a glimpse on github with this query
https://github.com/search?q=path%3A%2Fgo.mod+NOT+is%3Afork+content%3A%2F%5Ereplace+.*%2F&type=code&query=content%3Ago.mod+replace&p=3
. Note, Github is able to display only first 100 search results (Github API restriction).

One thing I don't like in my v2 is that I need to look up a public project to add required coverage. I would rather improve tests further by having everything in one repository and just one test, i.e. it has to be a small test project that could be used to test all go-related functionality. The question is though where it should be placed? I could create it on my Github account, but would that be a good option?

Quoted 2 lines

Richard, any suggestions on where these projects for selftest coverage should be hosted?

Thanks for the tests, they are massively helpful in maintaining this code.

We could always add something to git.yoctoproject.org, we do have a few test repositories there already for various purposes like this.

If you want to do that, please send an email to [email protected] with an ssh key (or I think a pointer to your github account), requesting setup for the repo and a brief description of what it is for.

Cheers,

Richard