go-mod.bbclass: Allow network in do_compile
PATCH 1/1 go-mod.bbclass: Allow network in do_compile
This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:
https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883
--- meta/classes-recipe/go-mod.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes-recipe/go-mod.bbclass +5 -0
@@ -24,3 +24,8 @@ inherit go GO_WORKDIR ?= "${GO_IMPORT}" do_compile[dirs] += "${B}/src/${GO_WORKDIR}" +do_compile[network] = "1" + +do_compile:prepend() { + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" +}
Signature · 1 line
2.35.1
Quoted 17 lines · Vyacheslav Yurkov
This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:
https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/classes-recipe/go-mod.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/go-mod.bbclass +0
-0
meta/classes-recipe/go-mod.bbclass inherit go · line 31
Quoted 8 lines
meta/classes-recipe/go-mod.bbclass +5
-0@@ -24,3 +24,8 @@ inherit go GO_WORKDIR ?= "${GO_IMPORT}" do_compile[dirs] += "${B}/src/${GO_WORKDIR}" +do_compile[network] = "1" + +do_compile:prepend() { + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" +}
This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.
Cheers,
Richard
Quoted 20 lines · Richard Purdie
Quoted 18 lines · Vyacheslav Yurkov
This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:
https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883Signed-off-by Vyacheslav Yurkov <[email protected]>--- meta/classes-recipe/go-mod.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes-recipe/go-mod.bbclass +0
-0@@ -24,3 +24,8 @@ inherit go
Quoted 11 lines
meta/classes-recipe/go-mod.bbclass inherit go · line 31
Quoted 7 lines
meta/classes-recipe/go-mod.bbclass +5
-0GO_WORKDIR ?= "${GO_IMPORT}" do_compile[dirs] += "${B}/src/${GO_WORKDIR}" +do_compile[network] = "1" + +do_compile:prepend() { + bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!" +}This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.
'nor should it go to master.
If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.
Bruce
Quoted 13 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#174968): https://lists.openembedded.org/g/openembedded-core/message/174968
Mute This Topic: https://lists.openembedded.org/mt/95827446/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
Quoted 4 lines · Bruce Ashfield
Quoted 2 lines · Richard Purdie
This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.
My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
Quoted 6 lines
'nor should it go to master.
If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.
Bruce
Probably I misunderstood the outcome of the discussion in my first link. Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad practice in general (I'm aware of a few more build systems with the same shortcoming).
My point is, until a proper solution is in place, it should be at least documented somewhere that this should be done in own layer/recipe. Any suggestions where this can be documented if not in OE-Core?
Vyacheslav
Quoted 22 lines · Vyacheslav Yurkov
Quoted 4 lines · Bruce Ashfield
Quoted 2 lines · Richard Purdie
This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.
My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
Quoted 6 lines
'nor should it go to master.
If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.
Bruce
Probably I misunderstood the outcome of the discussion in my first link. Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad practice in general (I'm aware of a few more build systems with the same shortcoming).
Quoted 4 lines
My point is, until a proper solution is in place, it should be at least documented somewhere that this should be done in own layer/recipe. Any suggestions where this can be documented if not in OE-Core?
Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?
It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.
But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum', 'Version': 'v0.12.0'} due to 404 "ERROR:
Error while fetching redirect page: HTTP Error 404: Not Found"
{'Path': 'code.cloudfoundry.org/clock', 'Version': 'v1.0.0', 'Indirect':
True} due to this being redirect to github.com and failing with "ERROR:
Error while fetching redirect page: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org'. (_ssl.c:996)>" parse
2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111,
in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100,
in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py",
line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 349, in process
self._rewrite_lic_uri(lines_before)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadata
if handle_var_end():
^^^^^^^^^^^^^^^^ File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, inhandle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op,
newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 372, in varfunc
uri, chksum = license.split(';', 1)
^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH
ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?
Regards,
Slava
Quoted 60 lines · Martin Jansa
Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?It's not perfect, I was testing it on https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.
But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"{'Path': 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com <http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>" parse2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipehandler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in processself._rewrite_lic_uri(lines_before)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines =bb.utils.edit_metadata(lines_before, ['LIC_FILES_CHKSUM'], varfunc)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadata
ifhandle_var_end():
^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, inhandle_var_end
(newvalue, newop, indent, minbreak) =varfunc(in_var, full_value, op, newlines)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc
uri, chksum =license.split(';', 1)
^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go <http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
Quoted 5 lines · Vyacheslav Yurkov
Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?
That's not my understanding.
The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet).
Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete).
go fetcher in bitbake is another possibility for future, but generating the long deps in SRC_URI is much better solution then enabling network in every go-mod recipe.
Please try recipetool changes on your go-mod recipes and provide feedback.
Regards,
Quoted 3 lines · Martin Jansa
Please try recipetool changes on your go-mod recipes and provide feedback.
Coincidentally, I also needed grpc-web recipe, which original patchset was tested on. I applied the series on my kirkstone branch of poky and tried to run the command:
% recipetool create -S c1971f71a28950d212b7ddc8df07b5563723031d -B master https://github.com/improbable-eng/grpc-web.git
Unfortunately this failed with the following log:
WARNING: go.mod files generated by Go < 1.17 might have incomplete indirect dependencies.
Traceback (most recent call last):
File "/home/uvv/projects/poky/scripts/recipetool", line 111, in <module>
ret = main()
File "/home/uvv/projects/poky/scripts/recipetool", line 100, in main
ret = args.func(args)
File "/home/uvv/projects/poky/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 332, in process
go_mod = self._go_mod_patch(patchfile, go_import, srctree, localfilesdir, extravalues, d)
File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 291, in _go_mod_patch bb.process.run("go mod tidy", stderr=subprocess.STDOUT, env=runenv, shell=True, cwd=srctree)
File "/home/uvv/projects/poky/bitbake/lib/bb/process.py", line 189, in run
raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'go mod tidy' failed with exit code 1:
go: finding module for package google.golang.org/grpc/metadata
go: finding module for package google.golang.org/grpc
go: finding module for package github.com/golang/protobuf/protoc-gen-go
go: finding module for package golang.org/x/net/context
go: finding module for package google.golang.org/grpc/codes
go: finding module for package google.golang.org/grpc/grpclog
go: downloading golang.org/x/net v0.4.0
go: downloading google.golang.org/grpc v1.51.0
go: downloading github.com/golang/protobuf v1.5.2
go: finding module for package github.com/cenkalti/backoff/v4
go: downloading github.com/cenkalti/backoff/v4 v4.2.0
go: finding module for package github.com/desertbit/timer
go: downloading github.com/cenkalti/backoff v2.2.1+incompatible
go: finding module for package github.com/rs/cors
go: finding module for package golang.org/x/net/http2
go: finding module for package google.golang.org/grpc/health/grpc_health_v1
go: finding module for package google.golang.org/grpc/status
go: finding module for package nhooyr.io/websocket
go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus
go: downloading github.com/rs/cors v1.8.3
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package github.com/mwitkow/go-conntrack
go: downloading nhooyr.io/websocket v1.8.7
go: finding module for package github.com/mwitkow/go-conntrack/connhelpers
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: finding module for package github.com/mwitkow/grpc-proxy/proxy
go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp
go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: finding module for package github.com/sirupsen/logrus
go: finding module for package github.com/spf13/pflag
go: finding module for package golang.org/x/net/trace
go: finding module for package google.golang.org/grpc/credentials
go: finding module for package github.com/golang/protobuf/ptypes/empty
go: downloading github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32
go: finding module for package github.com/golang/protobuf/proto
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/sirupsen/logrus v1.9.0
go: finding module for package github.com/stretchr/testify/assert
go: finding module for package github.com/stretchr/testify/require
go: finding module for package github.com/stretchr/testify/suite
go: finding module for package google.golang.org/grpc/health
go: finding module for package google.golang.org/protobuf/reflect/protoreflect
go: downloading github.com/prometheus/client_golang v1.14.0
go: downloading github.com/stretchr/testify v1.8.1
go: finding module for package google.golang.org/protobuf/runtime/protoimpl
go: finding module for package google.golang.org/protobuf/types/known/emptypb
go: downloading google.golang.org/protobuf v1.28.1
go: found github.com/golang/protobuf/protoc-gen-go in github.com/golang/protobuf v1.5.2
go: found golang.org/x/net/context in golang.org/x/net v0.4.0
go: found google.golang.org/grpc in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/codes in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/grpclog in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/metadata in google.golang.org/grpc v1.51.0
go: found github.com/cenkalti/backoff/v4 in github.com/cenkalti/backoff/v4 v4.2.0
go: found github.com/desertbit/timer in github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
go: found github.com/rs/cors in github.com/rs/cors v1.8.3
go: found golang.org/x/net/http2 in golang.org/x/net v0.4.0
go: found google.golang.org/grpc/health/grpc_health_v1 in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/status in google.golang.org/grpc v1.51.0
go: found nhooyr.io/websocket in nhooyr.io/websocket v1.8.7
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: found github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/mwitkow/go-conntrack in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: found github.com/mwitkow/go-conntrack/connhelpers in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: found github.com/mwitkow/grpc-proxy/proxy in github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32
go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.14.0
go: found github.com/sirupsen/logrus in github.com/sirupsen/logrus v1.9.0
go: found github.com/spf13/pflag in github.com/spf13/pflag v1.0.5
go: found golang.org/x/net/trace in golang.org/x/net v0.4.0
go: found google.golang.org/grpc/credentials in google.golang.org/grpc v1.51.0
go: found github.com/golang/protobuf/ptypes/empty in github.com/golang/protobuf v1.5.2
go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.5.2
go: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.8.1
go: found github.com/stretchr/testify/require in github.com/stretchr/testify v1.8.1
go: found github.com/stretchr/testify/suite in github.com/stretchr/testify v1.8.1
go: found google.golang.org/grpc/health in google.golang.org/grpc v1.51.0
go: found google.golang.org/protobuf/reflect/protoreflect in google.golang.org/protobuf v1.28.1
go: found google.golang.org/protobuf/runtime/protoimpl in google.golang.org/protobuf v1.28.1
go: found google.golang.org/protobuf/types/known/emptypb in google.golang.org/protobuf v1.28.1
go: downloading google.golang.org/genproto v0.0.0-20210401141331-865547bb08e2
go: downloading github.com/google/go-cmp v0.5.6
go: downloading github.com/klauspost/compress v1.10.3
go: downloading github.com/gin-gonic/gin v1.6.3
go: downloading github.com/gobwas/ws v1.0.2
go: downloading github.com/gorilla/websocket v1.4.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/jpillora/backoff v1.0.0
go: downloading github.com/prometheus/client_model v0.3.0
go: downloading github.com/prometheus/common v0.37.0
go: downloading golang.org/x/sys v0.3.0
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee
go: downloading github.com/gobwas/pool v0.2.0
go: downloading golang.org/x/text v0.5.0
go: downloading gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/prometheus/procfs v0.8.0
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/go-playground/validator/v10 v10.2.0
go: downloading github.com/ugorji/go/codec v1.1.7
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/ugorji/go v1.1.7
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/go-playground/universal-translator v0.17.0
go: downloading github.com/leodido/go-urn v1.2.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/go-playground/locales v0.13.0
go: downloading github.com/google/go-cmp v0.5.8
go: downloading golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
go: downloading gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 github.com/improbable-eng/grpc-web/go/grpcwebproxy imports
github.com/mwitkow/grpc-proxy/proxy tested by
github.com/mwitkow/grpc-proxy/proxy.test imports
github.com/mwitkow/grpc-proxy/testservice imports
golang.org/x/sync/errgroup loaded from golang.org/x/[email protected],
but go 1.16 would select v0.0.0-20220722155255-886fb9371eb4
To upgrade to the versions selected by go 1.16:
go mod tidy -go=1.16 && go mod tidy -go=1.17 If reproducibility with go 1.16 is not needed:
go mod tidy -compat=1.17
For other options, see:
https://golang.org/doc/modules/pruning
Kirkstone uses go1.17, but for some reason it's not detected properly. Perhaps I'm doing something wrong?
Thanks,
Slava
Hi Martin,
Am 26.12.2022 um 10:59 schrieb Martin Jansa via lists.openembedded.org:
Quoted 9 lines · Vyacheslav Yurkov
Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?
Quoted 7 lines
That's not my understanding.
The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet).
Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete).
go fetcher in bitbake is another possibility for future, but generating the long deps in
SRC_URIis much better solution then enabling network in every go-mod recipe.
Does an agreement exist if we should use a package manager class which creates a wget fetcher URI (ex. pipi) or a specific fetcher (ex. crate and npm)? At the moment both specific fetchers have different URI styles, create their specific tar commands and doesn't reuse the wget fetcher. Furthermore, the crate fetcher doesn't check the integrity during fetch. The fetcher inside bitbake have the advantage that you can fetch the latest version (ex. AUTOREV), but it brings the risk of code duplication and missing features. If we have an agreement and a common style its easy to replace the function with a specific fetcher.
Regards
Stefan
Martin,
first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported issues.
I've looked into the issues and would like to give some explanation.
1) the first module you mention ('gonum.org/v1/gonum <http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.
I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?
2) yes, my bad :)
3) The problem is that some repositories have a prefix in their tags. The tags usually only contain the semantic version string. I'm curretly working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
Quoted 103 lines · Vyacheslav Yurkov
On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> BruceProbably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?It's not perfect, I was testing it on https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.
But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"{'Path': 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com <http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>" parse2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipehandler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in processself._rewrite_lic_uri(lines_before)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines =bb.utils.edit_metadata(lines_before, ['LIC_FILES_CHKSUM'], varfunc)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadata
ifhandle_var_end():
^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, inhandle_var_end
(newvalue, newop, indent, minbreak) =varfunc(in_var, full_value, op, newlines)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc
uri, chksum =license.split(';', 1)
^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go <http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#174971): https://lists.openembedded.org/g/openembedded-core/message/174971
Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 8 lines · Lukas Funke
Martin,
first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported issues.
Hi Lukas,
and update on this series?
It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.
Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.
Kind Regards,
I've looked into the issues and would like to give some explanation.
Quoted 14 lines
1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.
Quoted 4 lines
I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?
Yes, that's what I was starting to do in:
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00
Quoted 2 lines
2) yes, my bad :)
as work around I was using
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547
3) The problem is that some repositories have a prefix in their tags.
Quoted 161 lines
The tags usually only contain the semantic version string. I'm curretly working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
Quoted 61 lines
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:
On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> BruceProbably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"Quoted 7 lines
{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.(_ssl.c:996)>"
Quoted 69 lines
parse
2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipe
handler.process(srctree_use, classes,lines_before,lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadataif handle_var_end(): ^^^^^^^^^^^^^^^^File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var,full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfuncuri, chksum = license.split(';', 1) ^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.
Quoted 7 lines
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#174971):https://lists.openembedded.org/g/openembedded-core/message/174971
Quoted 3 lines
Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Quoted 2 lines
-=-=-=-=-=-=-=-=-=-=-=-
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 22 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke
Martin,
first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported issues.
Hi Lukas, Stefan,
and update on this series?
It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.
Quoted 4 lines
Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.
Hello Lukas,
any progress on updated patch series? Nanbield release is near and I don't remember seeing the updated patch series in last couple months.
Is there anything I can do to help? The last version looked reasonably well and IMHO could be used as starting point for everybody to help fixing those corner cases where it didn't work well.
Regards,
Quoted 205 lines
Kind Regards,
I've looked into the issues and would like to give some explanation.
Quoted 14 lines
1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.Quoted 4 lines
I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?
Yes, that's what I was starting to do in:
Quoted 2 lines
2) yes, my bad :)
as work around I was using
3) The problem is that some repositories have a prefix in their tags.
Quoted 141 lines
The tags usually only contain the semantic version string. I'm curretly working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
Quoted 61 lines
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:
On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> BruceProbably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"Quoted 7 lines
{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.(_ssl.c:996)>"
Quoted 64 lines
parse
2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipe
handler.process(srctree_use, classes,lines_before,lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadataif handle_var_end(): ^^^^^^^^^^^^^^^^File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var,full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfuncuri, chksum = license.split(';', 1) ^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHQuoted 13 lines
Quoted 2 lines
ERROR: Parsing newly created recipe failed, moving recipe to
/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed
Quoted 9 lines
for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 32 lines · Martin Jansa
Quoted 22 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke
Martin,
first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported issues.
Hi Lukas, Stefan,
and update on this series?
It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.
Quoted 4 lines
Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.
Hello Lukas,
Quoted 8 lines
any progress on updated patch series? Nanbield release is near and I don't remember seeing the updated patch series in last couple months.
Is there anything I can do to help? The last version looked reasonably well and IMHO could be used as starting point for everybody to help fixing those corner cases where it didn't work well.
Regards,
Hello Martin,
I'm currently working on an update, but can't make any promises regarding the timeline.
Thanks for offering help! Reviewing and testing is always helpful when the updated series is on it's way.
Regards
Lukas
Quoted 208 lines
Quoted 207 lines
Kind Regards,
I've looked into the issues and would like to give some explanation.
Quoted 14 lines
1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.Quoted 4 lines
I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?
Yes, that's what I was starting to do in:
Quoted 2 lines
2) yes, my bad :)
as work around I was using
3) The problem is that some repositories have a prefix in their tags.
Quoted 141 lines
The tags usually only contain the semantic version string. I'm curretly working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:
Quoted 61 lines
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:
On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.
> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> BruceProbably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"Quoted 7 lines
{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.(_ssl.c:996)>"
Quoted 64 lines
parse
2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipe
handler.process(srctree_use, classes,lines_before,lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadataif handle_var_end(): ^^^^^^^^^^^^^^^^File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var,full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfuncuri, chksum = license.split(';', 1) ^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHQuoted 13 lines
Quoted 2 lines
ERROR: Parsing newly created recipe failed, moving recipe to
/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed
Quoted 9 lines
for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 44 lines · Lukas Funke
Quoted 15 lines · Martin Jansa
Quoted 12 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke
Martin,
first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.
tl;dr: I'm working on an updated patch series to address the reported issues.
Quoted 5 lines
Quoted 5 lines
Hi Lukas, Stefan,
and update on this series?
It doesn't have to be perfect, other people can help with remaining
corner
Quoted 2 lines
Quoted 2 lines
cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to
submit
Quoted 1 line
Quoted 1 line
incremental improvements - I have some as well, but haven't sent them
as I
Quoted 10 lines
Quoted 6 lines
don't know what you've already updated locally, so I'm waiting for v2.
Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.
Hello Lukas,
any progress on updated patch series? Nanbield release is near and I
don't
Quoted 3 lines
remember seeing the updated patch series in last couple months.
Is there anything I can do to help? The last version looked reasonably
well
Quoted 1 line
and IMHO could be used as starting point for everybody to help fixing
those
Quoted 9 lines
Quoted 3 lines
corner cases where it didn't work well.
Regards,
Hello Martin,
I'm currently working on an update, but can't make any promises regarding the timeline.
Just FYI: on meta-virt master there is a reworked script that is used to update the .inc files for go projects in the layer.
It might be helpful, it might not be!
Bruce
Quoted 234 lines
Thanks for offering help! Reviewing and testing is always helpful when the updated series is on it's way.
Regards
LukasQuoted 7 lines
Quoted 7 lines
Kind Regards,
I've looked into the issues and would like to give some explanation.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a404-page.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback.
Would
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
do you think about this solution?
Yes, that's what I was starting to do in:
Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
2) yes, my bad :)
as work around I was using
Quoted 11 lines
Quoted 11 lines
3) The problem is that some repositories have a prefix in their tags.
Quoted 8 lines
The tags usually only contain the semantic version string. I'm curretly working on this issue.
Best regards
Lukas
Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org
:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]
wrote:
On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport thereforenot
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
>> eligible for kirkstone/langdale.
My bad, the intention was to have it in all three branches:
master,
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
kirkstone, and langdale.
> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> BruceProbably I misunderstood the outcome of the discussion in my
first
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
link.
Bruce and Richard, you both suggested to have it in go-mod withthe
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
warning. I agree that it breaks reproducibility, and is
considered
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in ownlayer/recipe.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Any
suggestions where this can be documented if not in OE-Core?Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
Quoted 78 lines
Quoted 78 lines
Quoted 78 lines
Quoted 23 lines
?
It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).
Lukas/Stefan: is there something we can do to help with v2 of your patches?
I wanted to submit better review on submitted patches after more debugging, but here it is:
1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"Quoted 7 lines
{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL:CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.(_ssl.c:996)>"
Quoted 47 lines
parse
2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replaceUnknownwith the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret =main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret =args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, increate_recipe
handler.process(srctree_use, classes,lines_before,lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in_rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, inedit_metadataif handle_var_end(): ^^^^^^^^^^^^^^^^File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var,full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfuncuri, chksum = license.split(';', 1) ^^^^^^^^^^^ValueError: not enough values to unpack (expected 2, got 1)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
(the license was just '/')
3) some SRCREVs set to None leading to parsing failure:
File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstreamgit
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATHQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
ERROR: Parsing newly created recipe failed, moving recipe to
/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed
Quoted 23 lines
Quoted 22 lines
Quoted 10 lines
Quoted 9 lines
for reference. If this looks to be caused by the recipe itself, please report this error.
It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end - "Use the force Harry" - Gandalf, Star Trek II
I also tested using telegraf v1.21.4.
Kirkstone uses go1.17, so I tested this version.
Unfortunately, it failed with the following log.
Did I have something mistakes?
~$ recipetool create -S 583ee20a093acaa0781508b7840deb97a0fda622 -B release-1.21 https://github.com/influxdata/telegraf.git
Initialising tasks: 100% |################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)
NOTE: No setscene tasks
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 4 seconds
INFO: Resolved download path for import 'cloud.google.com/go/bigquery' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/monitoring' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/pubsub' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'collectd.org' => https://github.com/collectd/go-collectd
INFO: Resolved download path for import 'github.com/Azure/azure-event-hubs-go/v3' => https://github.com/Azure/azure-event-hubs-go
INFO: Resolved download path for import 'github.com/Azure/azure-kusto-go' => https://github.com/Azure/azure-kusto-go
INFO: Resolved download path for import 'github.com/Azure/azure-storage-queue-go' => https://github.com/Azure/azure-storage-queue-go
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/adal' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/azure/auth' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/BurntSushi/toml' => https://github.com/BurntSushi/toml
INFO: Resolved download path for import 'github.com/Mellanox/rdmamap' => https://github.com/Mellanox/rdmamap
INFO: Resolved download path for import 'github.com/Shopify/sarama' => https://github.com/Shopify/sarama
INFO: Resolved download path for import 'github.com/aerospike/aerospike-client-go' => https://github.com/aerospike/aerospike-client-go
INFO: Resolved download path for import 'github.com/alecthomas/units' => https://github.com/alecthomas/units
INFO: Resolved download path for import 'github.com/aliyun/alibaba-cloud-sdk-go' => https://github.com/aliyun/alibaba-cloud-sdk-go
INFO: Resolved download path for import 'github.com/amir/raidman' => https://github.com/amir/raidman
INFO: Resolved download path for import 'github.com/antchfx/jsonquery' => https://github.com/antchfx/jsonquery
INFO: Resolved download path for import 'github.com/antchfx/xmlquery' => https://github.com/antchfx/xmlquery
INFO: Resolved download path for import 'github.com/antchfx/xpath' => https://github.com/antchfx/xpath
INFO: Resolved download path for import 'github.com/apache/thrift' => https://github.com/apache/thrift
INFO: Resolved download path for import 'github.com/aristanetworks/goarista' => https://github.com/aristanetworks/goarista
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/config' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/credentials' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/feature/ec2/imds' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatch' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/dynamodb' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/ec2' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/kinesis' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/sts' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/timestreamwrite' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/smithy-go' => https://github.com/aws/smithy-go
INFO: Resolved download path for import 'github.com/benbjohnson/clock' => https://github.com/benbjohnson/clock
INFO: Resolved download path for import 'github.com/bmatcuk/doublestar/v3' => https://github.com/bmatcuk/doublestar
INFO: Resolved download path for import 'github.com/caio/go-tdigest' => https://github.com/caio/go-tdigest
INFO: Resolved download path for import 'github.com/cisco-ie/nx-telemetry-proto' => https://github.com/cisco-ie/nx-telemetry-proto
INFO: Resolved download path for import 'github.com/coreos/go-semver' => https://github.com/coreos/go-semver
INFO: Resolved download path for import 'github.com/couchbase/go-couchbase' => https://github.com/couchbase/go-couchbase
INFO: Resolved download path for import 'github.com/denisenkom/go-mssqldb' => https://github.com/denisenkom/go-mssqldb
INFO: Resolved download path for import 'github.com/dimchansky/utfbom' => https://github.com/dimchansky/utfbom
INFO: Resolved download path for import 'github.com/djherbis/times' => https://github.com/djherbis/times
INFO: Resolved download path for import 'github.com/docker/docker' => https://github.com/docker/docker
INFO: Resolved download path for import 'github.com/doclambda/protobufquery' => https://github.com/doclambda/protobufquery
INFO: Resolved download path for import 'github.com/dynatrace-oss/dynatrace-metric-utils-go' => https://github.com/dynatrace-oss/dynatrace-metric-utils-go
INFO: Resolved download path for import 'github.com/eclipse/paho.mqtt.golang' => https://github.com/eclipse/paho.mqtt.golang
INFO: Resolved download path for import 'github.com/fatih/color' => https://github.com/fatih/color
INFO: Resolved download path for import 'github.com/ghodss/yaml' => https://github.com/ghodss/yaml
INFO: Resolved download path for import 'github.com/go-ldap/ldap/v3' => https://github.com/go-ldap/ldap
INFO: Resolved download path for import 'github.com/go-logfmt/logfmt' => https://github.com/go-logfmt/logfmt
INFO: Resolved download path for import 'github.com/go-ping/ping' => https://github.com/go-ping/ping
INFO: Resolved download path for import 'github.com/go-redis/redis' => https://github.com/go-redis/redis
INFO: Resolved download path for import 'github.com/go-sql-driver/mysql' => https://github.com/go-sql-driver/mysql
INFO: Resolved download path for import 'github.com/gobwas/glob' => https://github.com/gobwas/glob
INFO: Resolved download path for import 'github.com/gofrs/uuid' => https://github.com/gofrs/uuid
INFO: Resolved download path for import 'github.com/golang-jwt/jwt/v4' => https://github.com/golang-jwt/jwt
INFO: Resolved download path for import 'github.com/golang/geo' => https://github.com/golang/geo
INFO: Resolved download path for import 'github.com/golang/snappy' => https://github.com/golang/snappy
INFO: Resolved download path for import 'github.com/google/go-cmp' => https://github.com/google/go-cmp
INFO: Resolved download path for import 'github.com/google/go-github/v32' => https://github.com/google/go-github
INFO: Resolved download path for import 'github.com/gopcua/opcua' => https://github.com/gopcua/opcua
INFO: Resolved download path for import 'github.com/gophercloud/gophercloud' => https://github.com/gophercloud/gophercloud
INFO: Resolved download path for import 'github.com/gorilla/mux' => https://github.com/gorilla/mux
INFO: Resolved download path for import 'github.com/gorilla/websocket' => https://github.com/gorilla/websocket
INFO: Resolved download path for import 'github.com/gosnmp/gosnmp' => https://github.com/gosnmp/gosnmp
INFO: Resolved download path for import 'github.com/grid-x/modbus' => https://github.com/grid-x/modbus
INFO: Resolved download path for import 'github.com/gwos/tcg/sdk' => https://github.com/gwos/tcg
INFO: Resolved download path for import 'github.com/harlow/kinesis-consumer' => https://github.com/harlow/kinesis-consumer
INFO: Resolved download path for import 'github.com/hashicorp/consul/api' => https://github.com/hashicorp/consul
INFO: Resolved download path for import 'github.com/hashicorp/go-uuid' => https://github.com/hashicorp/go-uuid
INFO: Resolved download path for import 'github.com/influxdata/go-syslog/v3' => https://github.com/influxdata/go-syslog
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/common' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/influx2otel' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/otel2influx' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/tail' => https://github.com/influxdata/tail
INFO: Resolved download path for import 'github.com/influxdata/toml' => https://github.com/influxdata/toml
INFO: Resolved download path for import 'github.com/influxdata/wlog' => https://github.com/influxdata/wlog
INFO: Resolved download path for import 'github.com/intel/iaevents' => https://github.com/intel/iaevents
INFO: Resolved download path for import 'github.com/jackc/pgx/v4' => https://github.com/jackc/pgx
INFO: Resolved download path for import 'github.com/james4k/rcon' => https://github.com/james4k/rcon
INFO: Resolved download path for import 'github.com/jhump/protoreflect' => https://github.com/jhump/protoreflect
INFO: Resolved download path for import 'github.com/jmespath/go-jmespath' => https://github.com/jmespath/go-jmespath
INFO: Resolved download path for import 'github.com/kardianos/service' => https://github.com/kardianos/service
INFO: Resolved download path for import 'github.com/karrick/godirwalk' => https://github.com/karrick/godirwalk
INFO: Resolved download path for import 'github.com/kballard/go-shellquote' => https://github.com/kballard/go-shellquote
INFO: Resolved download path for import 'github.com/matttproud/golang_protobuf_extensions' => https://github.com/matttproud/golang_protobuf_extensions
INFO: Resolved download path for import 'github.com/mdlayher/apcupsd' => https://github.com/mdlayher/apcupsd
INFO: Resolved download path for import 'github.com/microsoft/ApplicationInsights-Go' => https://github.com/microsoft/ApplicationInsights-Go
INFO: Resolved download path for import 'github.com/miekg/dns' => https://github.com/miekg/dns
INFO: Resolved download path for import 'github.com/moby/ipvs' => https://github.com/moby/ipvs
INFO: Resolved download path for import 'github.com/multiplay/go-ts3' => https://github.com/multiplay/go-ts3
INFO: Resolved download path for import 'github.com/nats-io/nats-server/v2' => https://github.com/nats-io/nats-server
INFO: Resolved download path for import 'github.com/nats-io/nats.go' => https://github.com/nats-io/nats.go
INFO: Resolved download path for import 'github.com/newrelic/newrelic-telemetry-sdk-go' => https://github.com/newrelic/newrelic-telemetry-sdk-go
INFO: Resolved download path for import 'github.com/nsqio/go-nsq' => https://github.com/nsqio/go-nsq
INFO: Resolved download path for import 'github.com/olivere/elastic' => https://github.com/olivere/elastic
INFO: Resolved download path for import 'github.com/openconfig/gnmi' => https://github.com/openconfig/gnmi
INFO: Resolved download path for import 'github.com/opentracing/opentracing-go' => https://github.com/opentracing/opentracing-go
INFO: Resolved download path for import 'github.com/openzipkin-contrib/zipkin-go-opentracing' => https://github.com/openzipkin-contrib/zipkin-go-opentracing
INFO: Resolved download path for import 'github.com/openzipkin/zipkin-go' => https://github.com/openzipkin/zipkin-go
INFO: Resolved download path for import 'github.com/pion/dtls/v2' => https://github.com/pion/dtls
INFO: Resolved download path for import 'github.com/pkg/errors' => https://github.com/pkg/errors
INFO: Resolved download path for import 'github.com/prometheus/client_golang' => https://github.com/prometheus/client_golang
INFO: Resolved download path for import 'github.com/prometheus/client_model' => https://github.com/prometheus/client_model
INFO: Resolved download path for import 'github.com/prometheus/common' => https://github.com/prometheus/common
INFO: Resolved download path for import 'github.com/prometheus/procfs' => https://github.com/prometheus/procfs
INFO: Resolved download path for import 'github.com/prometheus/prometheus' => https://github.com/prometheus/prometheus
INFO: Resolved download path for import 'github.com/riemann/riemann-go-client' => https://github.com/riemann/riemann-go-client
INFO: Resolved download path for import 'github.com/safchain/ethtool' => https://github.com/safchain/ethtool
INFO: Resolved download path for import 'github.com/sensu/sensu-go/api/core/v2' => https://github.com/sensu/sensu-go
INFO: Resolved download path for import 'github.com/shirou/gopsutil/v3' => https://github.com/shirou/gopsutil
INFO: Resolved download path for import 'github.com/showwin/speedtest-go' => https://github.com/showwin/speedtest-go
INFO: Resolved download path for import 'github.com/signalfx/golib/v3' => https://github.com/signalfx/golib
INFO: Resolved download path for import 'github.com/sirupsen/logrus' => https://github.com/sirupsen/logrus
INFO: Resolved download path for import 'github.com/sleepinggenius2/gosmi' => https://github.com/sleepinggenius2/gosmi
INFO: Resolved download path for import 'github.com/snowflakedb/gosnowflake' => https://github.com/snowflakedb/gosnowflake
INFO: Resolved download path for import 'github.com/streadway/amqp' => https://github.com/streadway/amqp
INFO: Resolved download path for import 'github.com/stretchr/testify' => https://github.com/stretchr/testify
INFO: Resolved download path for import 'github.com/tbrandon/mbserver' => https://github.com/tbrandon/mbserver
INFO: Resolved download path for import 'github.com/testcontainers/testcontainers-go' => https://github.com/testcontainers/testcontainers-go
INFO: Resolved download path for import 'github.com/tidwall/gjson' => https://github.com/tidwall/gjson
INFO: Resolved download path for import 'github.com/tinylib/msgp' => https://github.com/tinylib/msgp
INFO: Resolved download path for import 'github.com/vapourismo/knx-go' => https://github.com/vapourismo/knx-go
INFO: Resolved download path for import 'github.com/vjeantet/grok' => https://github.com/vjeantet/grok
INFO: Resolved download path for import 'github.com/vmware/govmomi' => https://github.com/vmware/govmomi
INFO: Resolved download path for import 'github.com/wavefronthq/wavefront-sdk-go' => https://github.com/wavefronthq/wavefront-sdk-go
INFO: Resolved download path for import 'github.com/wvanbergen/kafka' => https://github.com/wvanbergen/kafka
INFO: Resolved download path for import 'github.com/xdg/scram' => https://github.com/xdg/scram
INFO: Resolved download path for import 'go.mongodb.org/mongo-driver' => https://github.com/mongodb/mongo-go-driver.git
INFO: Resolved download path for import 'go.opentelemetry.io/collector/model' => https://github.com/open-telemetry/opentelemetry-collector
INFO: Resolved download path for import 'go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.opentelemetry.io/otel/metric' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.opentelemetry.io/otel/sdk/metric' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.starlark.net' => https://github.com/google/starlark-go
INFO: Resolved download path for import 'golang.org/x/net' => https://go.googlesource.com/net
INFO: Resolved download path for import 'golang.org/x/oauth2' => https://go.googlesource.com/oauth2
INFO: Resolved download path for import 'golang.org/x/sync' => https://go.googlesource.com/sync
INFO: Resolved download path for import 'golang.org/x/sys' => https://go.googlesource.com/sys
INFO: Resolved download path for import 'golang.org/x/text' => https://go.googlesource.com/text
INFO: Resolved download path for import 'golang.zx2c4.com/wireguard/wgctrl' => https://github.com/WireGuard/wgctrl-go
INFO: Resolved download path for import 'google.golang.org/api' => https://github.com/googleapis/google-api-go-client
INFO: Resolved download path for import 'google.golang.org/genproto' => https://github.com/googleapis/go-genproto
INFO: Resolved download path for import 'google.golang.org/grpc' => https://github.com/grpc/grpc-go
INFO: Resolved download path for import 'google.golang.org/protobuf' => https://go.googlesource.com/protobuf
INFO: Resolved download path for import 'gopkg.in/gorethink/gorethink.v3' => https://gopkg.in/gorethink/gorethink.v3
INFO: Resolved download path for import 'gopkg.in/olivere/elastic.v5' => https://gopkg.in/olivere/elastic.v5
INFO: Resolved download path for import 'gopkg.in/tomb.v1' => https://gopkg.in/tomb.v1
INFO: Resolved download path for import 'gopkg.in/yaml.v2' => https://gopkg.in/yaml.v2
INFO: Resolved download path for import 'k8s.io/api' => https://github.com/kubernetes/api
INFO: Resolved download path for import 'k8s.io/apimachinery' => https://github.com/kubernetes/apimachinery
INFO: Resolved download path for import 'k8s.io/client-go' => https://github.com/kubernetes/client-go
INFO: Resolved download path for import 'modernc.org/sqlite' => https://gitlab.com/cznic/sqlite
INFO: Resolved download path for import 'cloud.google.com/go' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/compute' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/iam' => https://github.com/googleapis/google-cloud-go
Traceback (most recent call last):
File "/oe-core/scripts/recipetool", line 111, in <module>
ret = main()
File "/oe-core/scripts/recipetool", line 100, in main
ret = args.func(args)
File "/oe-core/scripts/lib/recipetool/create.py", line 746, in create_recipehandler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 336, in processdep_src_uris, src_revs = self._handle_dependencies(d, srctree, go_mod)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 261, in _handle_dependencies
repodata = self._resolve_repository(module_path)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 184, in _resolve_repository
repodata = self._resolve_repository_dynamic(modulepath)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 176, in _resolve_repository_dynamic
return GoImport(parser.rootpath, parser.vcs, parser.repourl, None)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 151, in rootpath
return self.__srv[0]
IndexError: list index out of range