Skip to contentSkip to conversation

go-mod.bbclass: Allow network in do_compile

PATCH In review Vyacheslav Yurkov 7 participants 15 messages Dec 22, 2022 – Sep 27, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 go-mod.bbclass: Allow network in do_compile

This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:

https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883

---
 meta/classes-recipe/go-mod.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes-recipe/go-mod.bbclass +5 -0
index 61571596bc..da9a583288 100644
@@ -24,3 +24,8 @@ inherit go
 
 GO_WORKDIR ?= "${GO_IMPORT}"
 do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
+do_compile[network] = "1"
+
+do_compile:prepend() {
+    bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!"
+}
Signature · 1 line
2.35.1
#1 lore raw
Quoted 17 lines · Vyacheslav Yurkov

This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:

https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883

---
 meta/classes-recipe/go-mod.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes-recipe/go-mod.bbclass +0 -0
index 61571596bc..da9a583288 100644

meta/classes-recipe/go-mod.bbclass inherit go · line 31

Quoted 8 lines
meta/classes-recipe/go-mod.bbclass +5 -0
@@ -24,3 +24,8 @@ inherit go
 
 GO_WORKDIR ?= "${GO_IMPORT}"
 do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
+do_compile[network] = "1"
+
+do_compile:prepend() {
+    bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!"
+}

This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.

Cheers,

Richard

#2 lore raw
Quoted 20 lines · Richard Purdie
Quoted 18 lines · Vyacheslav Yurkov

This is a workaround to allow using go-mod until a proper solution is in place. Possible solutions were discussued ealier this year in these threads:

https://lists.openembedded.org/g/openembedded-core/topic/89464905
https://lists.openembedded.org/g/openembedded-architecture/topic/90782883

---
 meta/classes-recipe/go-mod.bbclass | 5 +++++
 1 file changed, 5 insertions(+)
meta/classes-recipe/go-mod.bbclass +0 -0
index 61571596bc..da9a583288 100644
@@ -24,3 +24,8 @@ inherit go
 
Quoted 11 lines

meta/classes-recipe/go-mod.bbclass inherit go · line 31

Quoted 7 lines
meta/classes-recipe/go-mod.bbclass +5 -0
 GO_WORKDIR ?= "${GO_IMPORT}"
 do_compile[dirs] += "${B}/src/${GO_WORKDIR}"
+do_compile[network] = "1"
+
+do_compile:prepend() {
+    bbwarn "Using go-mod enables network during compilation to be able to fetch dependencies. This breaks reproducibility of the build!"
+}

This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.

'nor should it go to master.

If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.

Bruce

Quoted 13 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#174968): https://lists.openembedded.org/g/openembedded-core/message/174968
Mute This Topic: https://lists.openembedded.org/mt/95827446/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#3 lore raw
Quoted 4 lines · Bruce Ashfield
Quoted 2 lines · Richard Purdie

This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

Quoted 6 lines

'nor should it go to master.

If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.

Bruce

Probably I misunderstood the outcome of the discussion in my first link. Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad practice in general (I'm aware of a few more build systems with the same shortcoming).

My point is, until a proper solution is in place, it should be at least documented somewhere that this should be done in own layer/recipe. Any suggestions where this can be documented if not in OE-Core?

Vyacheslav

#4 lore raw
Quoted 22 lines · Vyacheslav Yurkov
Quoted 4 lines · Bruce Ashfield
Quoted 2 lines · Richard Purdie

This patch is not in master and is not a backport therefore not eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

Quoted 6 lines

'nor should it go to master.

If someone wants to allow go to fetch sources during builds, it needs to be done in their own layers.

Bruce

Probably I misunderstood the outcome of the discussion in my first link. Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad practice in general (I'm aware of a few more build systems with the same shortcoming).

Quoted 4 lines

My point is, until a proper solution is in place, it should be at least documented somewhere that this should be done in own layer/recipe. Any suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum', 'Version': 'v0.12.0'} due to 404 "ERROR:
Error while fetching redirect page: HTTP Error 404: Not Found"

{'Path': 'code.cloudfoundry.org/clock', 'Version': 'v1.0.0', 'Indirect':
True} due to this being redirect to github.com and failing with "ERROR:
Error while fetching redirect page: <urlopen error [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org'. (_ssl.c:996)>" parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111,
in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100,
in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py",
line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 349, in process
self._rewrite_lic_uri(lines_before)
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
edit_metadata

    if handle_var_end():
       ^^^^^^^^^^^^^^^^

File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op,
newlines)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py",
line 372, in varfunc

    uri, chksum = license.split(';', 1)
    ^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

#5 lore raw

Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?

Regards,
Slava

Quoted 60 lines · Martin Jansa

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

{'Path': 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com <http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>" parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111, in <module>
    ret = main()
          ^^^^^^
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100, in main
    ret = args.func(args)
          ^^^^^^^^^^^^^^^
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process self._rewrite_lic_uri(lines_before)
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
    updated, newlines = bb.utils.edit_metadata(lines_before, ['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in edit_metadata
    if handle_var_end():
       ^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in handle_var_end
    (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc
    uri, chksum = license.split(';', 1)
    ^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

  File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in _latest_revision
    raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go <http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

#6 lore raw
Quoted 5 lines · Vyacheslav Yurkov

Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?

That's not my understanding.

The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet).

Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete).

go fetcher in bitbake is another possibility for future, but generating the long deps in SRC_URI is much better solution then enabling network in every go-mod recipe.

Please try recipetool changes on your go-mod recipes and provide feedback.

Regards,

#7 lore raw
Quoted 3 lines · Martin Jansa

Please try recipetool changes on your go-mod recipes and provide feedback.

Coincidentally, I also needed grpc-web recipe, which original patchset was tested on. I applied the series on my kirkstone branch of poky and tried to run the command:

% recipetool create -S c1971f71a28950d212b7ddc8df07b5563723031d -B master https://github.com/improbable-eng/grpc-web.git

Unfortunately this failed with the following log:

WARNING: go.mod files generated by Go < 1.17 might have incomplete indirect dependencies.
Traceback (most recent call last):
  File "/home/uvv/projects/poky/scripts/recipetool", line 111, in <module>
    ret = main()
  File "/home/uvv/projects/poky/scripts/recipetool", line 100, in main
    ret = args.func(args)
  File "/home/uvv/projects/poky/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 332, in process
    go_mod = self._go_mod_patch(patchfile, go_import, srctree, localfilesdir, extravalues, d)
  File "/home/uvv/projects/poky/scripts/lib/recipetool/create_go.py", line 291, in _go_mod_patch bb.process.run("go mod tidy", stderr=subprocess.STDOUT, env=runenv, shell=True, cwd=srctree)
  File "/home/uvv/projects/poky/bitbake/lib/bb/process.py", line 189, in run
    raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'go mod tidy' failed with exit code 1:
go: finding module for package google.golang.org/grpc/metadata
go: finding module for package google.golang.org/grpc
go: finding module for package github.com/golang/protobuf/protoc-gen-go
go: finding module for package golang.org/x/net/context
go: finding module for package google.golang.org/grpc/codes
go: finding module for package google.golang.org/grpc/grpclog
go: downloading golang.org/x/net v0.4.0
go: downloading google.golang.org/grpc v1.51.0
go: downloading github.com/golang/protobuf v1.5.2
go: finding module for package github.com/cenkalti/backoff/v4
go: downloading github.com/cenkalti/backoff/v4 v4.2.0
go: finding module for package github.com/desertbit/timer
go: downloading github.com/cenkalti/backoff v2.2.1+incompatible
go: finding module for package github.com/rs/cors
go: finding module for package golang.org/x/net/http2
go: finding module for package google.golang.org/grpc/health/grpc_health_v1
go: finding module for package google.golang.org/grpc/status
go: finding module for package nhooyr.io/websocket
go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus
go: downloading github.com/rs/cors v1.8.3
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package github.com/mwitkow/go-conntrack
go: downloading nhooyr.io/websocket v1.8.7
go: finding module for package github.com/mwitkow/go-conntrack/connhelpers
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: finding module for package github.com/mwitkow/grpc-proxy/proxy
go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp
go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: finding module for package github.com/sirupsen/logrus
go: finding module for package github.com/spf13/pflag
go: finding module for package golang.org/x/net/trace
go: finding module for package google.golang.org/grpc/credentials
go: finding module for package github.com/golang/protobuf/ptypes/empty
go: downloading github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32
go: finding module for package github.com/golang/protobuf/proto
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/sirupsen/logrus v1.9.0
go: finding module for package github.com/stretchr/testify/assert
go: finding module for package github.com/stretchr/testify/require
go: finding module for package github.com/stretchr/testify/suite
go: finding module for package google.golang.org/grpc/health
go: finding module for package google.golang.org/protobuf/reflect/protoreflect
go: downloading github.com/prometheus/client_golang v1.14.0
go: downloading github.com/stretchr/testify v1.8.1
go: finding module for package google.golang.org/protobuf/runtime/protoimpl
go: finding module for package google.golang.org/protobuf/types/known/emptypb
go: downloading google.golang.org/protobuf v1.28.1
go: found github.com/golang/protobuf/protoc-gen-go in github.com/golang/protobuf v1.5.2
go: found golang.org/x/net/context in golang.org/x/net v0.4.0
go: found google.golang.org/grpc in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/codes in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/grpclog in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/metadata in google.golang.org/grpc v1.51.0
go: found github.com/cenkalti/backoff/v4 in github.com/cenkalti/backoff/v4 v4.2.0
go: found github.com/desertbit/timer in github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
go: found github.com/rs/cors in github.com/rs/cors v1.8.3
go: found golang.org/x/net/http2 in golang.org/x/net v0.4.0
go: found google.golang.org/grpc/health/grpc_health_v1 in google.golang.org/grpc v1.51.0
go: found google.golang.org/grpc/status in google.golang.org/grpc v1.51.0
go: found nhooyr.io/websocket in nhooyr.io/websocket v1.8.7
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: found github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus in github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/mwitkow/go-conntrack in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: found github.com/mwitkow/go-conntrack/connhelpers in github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
go: found github.com/mwitkow/grpc-proxy/proxy in github.com/mwitkow/grpc-proxy v0.0.0-20220126150247-db34e7bfee32
go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.14.0
go: found github.com/sirupsen/logrus in github.com/sirupsen/logrus v1.9.0
go: found github.com/spf13/pflag in github.com/spf13/pflag v1.0.5
go: found golang.org/x/net/trace in golang.org/x/net v0.4.0
go: found google.golang.org/grpc/credentials in google.golang.org/grpc v1.51.0
go: found github.com/golang/protobuf/ptypes/empty in github.com/golang/protobuf v1.5.2
go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.5.2
go: found github.com/stretchr/testify/assert in github.com/stretchr/testify v1.8.1
go: found github.com/stretchr/testify/require in github.com/stretchr/testify v1.8.1
go: found github.com/stretchr/testify/suite in github.com/stretchr/testify v1.8.1
go: found google.golang.org/grpc/health in google.golang.org/grpc v1.51.0
go: found google.golang.org/protobuf/reflect/protoreflect in google.golang.org/protobuf v1.28.1
go: found google.golang.org/protobuf/runtime/protoimpl in google.golang.org/protobuf v1.28.1
go: found google.golang.org/protobuf/types/known/emptypb in google.golang.org/protobuf v1.28.1
go: downloading google.golang.org/genproto v0.0.0-20210401141331-865547bb08e2
go: downloading github.com/google/go-cmp v0.5.6
go: downloading github.com/klauspost/compress v1.10.3
go: downloading github.com/gin-gonic/gin v1.6.3
go: downloading github.com/gobwas/ws v1.0.2
go: downloading github.com/gorilla/websocket v1.4.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/jpillora/backoff v1.0.0
go: downloading github.com/prometheus/client_model v0.3.0
go: downloading github.com/prometheus/common v0.37.0
go: downloading golang.org/x/sys v0.3.0
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee
go: downloading github.com/gobwas/pool v0.2.0
go: downloading golang.org/x/text v0.5.0
go: downloading gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/prometheus/procfs v0.8.0
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/go-playground/validator/v10 v10.2.0
go: downloading github.com/ugorji/go/codec v1.1.7
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/ugorji/go v1.1.7
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/go-playground/universal-translator v0.17.0
go: downloading github.com/leodido/go-urn v1.2.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading github.com/go-playground/locales v0.13.0
go: downloading github.com/google/go-cmp v0.5.8
go: downloading golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f
go: downloading gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 github.com/improbable-eng/grpc-web/go/grpcwebproxy imports
        github.com/mwitkow/grpc-proxy/proxy tested by
        github.com/mwitkow/grpc-proxy/proxy.test imports
        github.com/mwitkow/grpc-proxy/testservice imports
        golang.org/x/sync/errgroup loaded from golang.org/x/[email protected],
        but go 1.16 would select v0.0.0-20220722155255-886fb9371eb4

To upgrade to the versions selected by go 1.16:
        go mod tidy -go=1.16 && go mod tidy -go=1.17 If reproducibility with go 1.16 is not needed:
        go mod tidy -compat=1.17
For other options, see:
https://golang.org/doc/modules/pruning

Kirkstone uses go1.17, but for some reason it's not detected properly. Perhaps I'm doing something wrong?

Thanks,
Slava

#8 lore raw
In reply to Martin Jansa

Hi Martin,

Am 26.12.2022 um 10:59 schrieb Martin Jansa via lists.openembedded.org:

Quoted 9 lines · Vyacheslav Yurkov

Martin, I did see that patch series, but I was not sure about its state, therefore I didn't try it. If I understood correctly, that partially solves the issue, but the proper solution should involve the fetcher, that's why it's not yet merged to master. Am I missing something?

Quoted 7 lines

That's not my understanding.

The proposed changes for recipetool had some cosmetic and functional review comments and it doesn't work correctly for every possible go dependency (e.g. the 3 issues I've mentioned from telegram recipe), but it's very good starting point and we should be able to get it merged in master (and then everybody can more easily contribute fixes for the corner cases where it might not work well yet).

Even missing documentation IMHO shouldn't block merging it at this point, because there is a clear need this (as there are already 3 partial implementations to fix this issue, neither 100% complete and nothing merged in master - the recipetool changes are IMHO the best of 3 as it integrates with devtool as well and seems to be most complete).

go fetcher in bitbake is another possibility for future, but generating the long deps in SRC_URI is much better solution then enabling network in every go-mod recipe.

Does an agreement exist if we should use a package manager class which creates a wget fetcher URI (ex. pipi) or a specific fetcher (ex. crate and npm)? At the moment both specific fetchers have different URI styles, create their specific tar commands and doesn't reuse the wget fetcher. Furthermore, the crate fetcher doesn't check the integrity during fetch. The fetcher inside bitbake have the advantage that you can fetch the latest version (ex. AUTOREV), but it brings the risk of code duplication and missing features. If we have an agreement and a common style its easy to replace the function with a specific fetcher.

Regards
  Stefan

#9 lore raw
In reply to Martin Jansa

Martin,

first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported issues.

I've looked into the issues and would like to give some explanation.

1) the first module you mention ('gonum.org/v1/gonum <http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.

I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?

2) yes, my bad :)

3) The problem is that some repositories have a prefix in their tags. The tags usually only contain the semantic version string. I'm curretly working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:

Quoted 103 lines · Vyacheslav Yurkov

On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).

My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version': 'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

{'Path': 'code.cloudfoundry.org/clock <http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com <http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'. (_ssl.c:996)>" parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 111, in <module>
    ret = main()
          ^^^^^^
  File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line 100, in main
    ret = args.func(args)
          ^^^^^^^^^^^^^^^
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process self._rewrite_lic_uri(lines_before)
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
    updated, newlines = bb.utils.edit_metadata(lines_before, ['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in edit_metadata
    if handle_var_end():
       ^^^^^^^^^^^^^^^^
  File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in handle_var_end
    (newvalue, newop, indent, minbreak) = varfunc(in_var, full_value, op, newlines) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc
    uri, chksum = license.split(';', 1)
    ^^^^^^^^^^^
ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

  File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in _latest_revision
    raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go <http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#174971): https://lists.openembedded.org/g/openembedded-core/message/174971
Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#10 lore raw
Quoted 8 lines · Lukas Funke

Martin,

first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported issues.

Hi Lukas,

and update on this series?

It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.

Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.

Kind Regards,

I've looked into the issues and would like to give some explanation.

Quoted 14 lines

1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.

Quoted 4 lines

I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?

Yes, that's what I was starting to do in:
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00

Quoted 2 lines

2) yes, my bad :)

as work around I was using
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547

3) The problem is that some repositories have a prefix in their tags.

Quoted 161 lines

The tags usually only contain the semantic version string. I'm curretly working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:

Quoted 61 lines

On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:

On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).

My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

Quoted 7 lines

{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.

(_ssl.c:996)>"

Quoted 69 lines

parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
edit_metadata

    if handle_var_end():
       ^^^^^^^^^^^^^^^^

File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc

    uri, chksum = license.split(';', 1)
    ^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

ERROR: Parsing newly created recipe failed, moving recipe to /OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed for reference. If this looks to be caused by the recipe itself, please report this error.

Quoted 7 lines

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#174971):

https://lists.openembedded.org/g/openembedded-core/message/174971

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/95827446/7344993
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [

[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 22 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke

Martin,

first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported issues.

Hi Lukas, Stefan,

and update on this series?

It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.

Quoted 4 lines

Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.

Hello Lukas,

any progress on updated patch series? Nanbield release is near and I don't remember seeing the updated patch series in last couple months.

Is there anything I can do to help? The last version looked reasonably well and IMHO could be used as starting point for everybody to help fixing those corner cases where it didn't work well.

Regards,

Quoted 205 lines

Kind Regards,

I've looked into the issues and would like to give some explanation.

Quoted 14 lines

1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.

Quoted 4 lines

I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?

Yes, that's what I was starting to do in:

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00

Quoted 2 lines

2) yes, my bad :)

as work around I was using

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547

3) The problem is that some repositories have a prefix in their tags.

Quoted 141 lines

The tags usually only contain the semantic version string. I'm curretly working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:

Quoted 61 lines

On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:

On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).

My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

Quoted 7 lines

{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.

(_ssl.c:996)>"

Quoted 64 lines

parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
edit_metadata

    if handle_var_end():
       ^^^^^^^^^^^^^^^^

File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc

    uri, chksum = license.split(';', 1)
    ^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in
_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

Quoted 13 lines
Quoted 2 lines

ERROR: Parsing newly created recipe failed, moving recipe to

/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed

Quoted 9 lines

for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#13 lore raw
Quoted 32 lines · Martin Jansa
Quoted 22 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke

Martin,

first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported issues.

Hi Lukas, Stefan,

and update on this series?

It doesn't have to be perfect, other people can help with remaining corner cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to submit incremental improvements - I have some as well, but haven't sent them as I don't know what you've already updated locally, so I'm waiting for v2.

Quoted 4 lines

Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.

Hello Lukas,

Quoted 8 lines

any progress on updated patch series? Nanbield release is near and I don't remember seeing the updated patch series in last couple months.

Is there anything I can do to help? The last version looked reasonably well and IMHO could be used as starting point for everybody to help fixing those corner cases where it didn't work well.

Regards,

Hello Martin,

I'm currently working on an update, but can't make any promises regarding the timeline.

Thanks for offering help! Reviewing and testing is always helpful when the updated series is on it's way.

Regards
Lukas

Quoted 208 lines
Quoted 207 lines

Kind Regards,

I've looked into the issues and would like to give some explanation.

Quoted 14 lines

1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a 404-page. This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.

Quoted 4 lines

I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback. Would do you think about this solution?

Yes, that's what I was starting to do in:

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00

Quoted 2 lines

2) yes, my bad :)

as work around I was using

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547

3) The problem is that some repositories have a prefix in their tags.

Quoted 141 lines

The tags usually only contain the semantic version string. I'm curretly working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org:

Quoted 61 lines

On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]> wrote:

On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore not >> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches: master, kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my first link.
Bruce and Richard, you both suggested to have it in go-mod with the warning. I agree that it breaks reproducibility, and is considered a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).

My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own layer/recipe. Any
suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:
https://lists.openembedded.org/g/openembedded-architecture/message/1539
?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

Quoted 7 lines

{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.

(_ssl.c:996)>"

Quoted 64 lines

parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
edit_metadata

     if handle_var_end():
        ^^^^^^^^^^^^^^^^

File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc

     uri, chksum = license.split(';', 1)
     ^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

   File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in

_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream git
repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

Quoted 13 lines
Quoted 2 lines

ERROR: Parsing newly created recipe failed, moving recipe to

/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed

Quoted 9 lines

for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#14 lore raw
Quoted 44 lines · Lukas Funke
Quoted 15 lines · Martin Jansa
Quoted 12 lines · Martin Jansa via lists.openembedded.org
Quoted 8 lines · Lukas Funke

Martin,

first of all: thank you for testing the patches. As usual the corner cases are the most tricky ones.

tl;dr: I'm working on an updated patch series to address the reported issues.

Quoted 5 lines
Quoted 5 lines

Hi Lukas, Stefan,

and update on this series?

It doesn't have to be perfect, other people can help with remaining

corner

Quoted 2 lines
Quoted 2 lines

cases and right now there is nothing for this in oe-core, so any version will be big improvement :) and it will make it easier for others to

submit

Quoted 1 line
Quoted 1 line

incremental improvements - I have some as well, but haven't sent them

as I

Quoted 10 lines
Quoted 6 lines

don't know what you've already updated locally, so I'm waiting for v2.

Mickledore is already in feature freeze, but maybe RP will make an exception as this is new important functionality which isn't likely to break other existing code.

Hello Lukas,

any progress on updated patch series? Nanbield release is near and I

don't

Quoted 3 lines

remember seeing the updated patch series in last couple months.

Is there anything I can do to help? The last version looked reasonably

well

Quoted 1 line

and IMHO could be used as starting point for everybody to help fixing

those

Quoted 9 lines
Quoted 3 lines

corner cases where it didn't work well.

Regards,

Hello Martin,

I'm currently working on an update, but can't make any promises regarding the timeline.

Just FYI: on meta-virt master there is a reworked script that is used to update the .inc files for go projects in the layer.

It might be helpful, it might not be!

Bruce

Quoted 234 lines

Thanks for offering help! Reviewing and testing is always helpful when the updated series is on it's way.

Regards
Lukas

Quoted 7 lines
Quoted 7 lines

Kind Regards,

I've looked into the issues and would like to give some explanation.

Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

1) the first module you mention ('gonum.org/v1/gonum
<http://gonum.org/v1/gonum>') does not obey the go documentation for resolving the modules source-code repository (see
https://go.dev/ref/mod#go-mod-download , Section 'Version control systems'). The documentation states that there should be a html page, queried by "?go-get=1", with a <meta>-tag which contains the original URL to the source-code repository. For 'gonum' there is only a

404-page.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

This page contains javascript-code that redirects to the actual gonum package page. Unnecessary hard to process. I'll open an issue for that. The second module 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>' has a broken certificate. I already opend an issue on github for this.

I addressed this issue above by maintaining a list of modules and their actual repo URLs inside the go-recipetool as an absolute fallback.

Would

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

do you think about this solution?

Yes, that's what I was starting to do in:

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=b973c7f17c8a613233a1a18de0bf6daa352c47d8

Quoted 2 lines
Quoted 2 lines

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=61c6c9962b3b7d4c1cf6e6823de6c32b09998b00

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines

2) yes, my bad :)

as work around I was using

https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/go&id=5767819bfe96de29ab751d4a37a431a324f7e547

Quoted 11 lines
Quoted 11 lines

3) The problem is that some repositories have a prefix in their tags.

Quoted 8 lines

The tags usually only contain the semantic version string. I'm curretly working on this issue.

Best regards

Lukas

Am 22.12.2022 um 20:08 schrieb Martin Jansa via lists.openembedded.org

:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

On Thu, Dec 22, 2022 at 7:39 PM Vyacheslav Yurkov <[email protected]

wrote:

On 22.12.2022 17:48, Bruce Ashfield wrote:
> On Thu, Dec 22, 2022 at 11:20 AM Richard Purdie
> <[email protected]> wrote:
>> This patch is not in master and is not a backport therefore

not

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

>> eligible for kirkstone/langdale.

My bad, the intention was to have it in all three branches:

master,

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

kirkstone, and langdale.

> 'nor should it go to master.
>
> If someone wants to allow go to fetch sources during builds, it needs
> to be done in their own layers.
>
> Bruce

Probably I misunderstood the outcome of the discussion in my

first

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

link.
Bruce and Richard, you both suggested to have it in go-mod with

the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

warning. I agree that it breaks reproducibility, and is

considered

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

a bad
practice in general (I'm aware of a few more build systems with the same
shortcoming).

My point is, until a proper solution is in place, it should be at least
documented somewhere that this should be done in own

layer/recipe.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Any
suggestions where this can be documented if not in OE-Core?

Have you tried the changes submitted by Lukas/Stefan in:

https://lists.openembedded.org/g/openembedded-architecture/message/1539

Quoted 78 lines
Quoted 78 lines
Quoted 78 lines
Quoted 23 lines

?

It's not perfect, I was testing it on
https://github.com/influxdata/telegraf/blob/master/go.mod and I've found some corner cases where it failed.

But it seems like very good start and we should work with Lukas/Stefan to get it merged in master. Then all branches could consume recipes created in master and only the exceptions would need to have network access (instead of the bbclass enabling it for every go-mod user - go-vendor inherits go-mod as well).

Lukas/Stefan: is there something we can do to help with v2 of your patches?

I wanted to submit better review on submitted patches after more debugging, but here it is:

1) to resolve 2 deps:
{'Path': 'gonum.org/v1/gonum <http://gonum.org/v1/gonum>', 'Version':
'v0.12.0'} due to 404 "ERROR: Error while fetching redirect page: HTTP Error 404: Not Found"

Quoted 7 lines

{'Path': 'code.cloudfoundry.org/clock
<http://code.cloudfoundry.org/clock>', 'Version': 'v1.0.0', 'Indirect': True} due to this being redirect to github.com
<http://github.com> and failing with "ERROR: Error while fetching redirect page: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'code.cloudfoundry.org <http://code.cloudfoundry.org>'.

(_ssl.c:996)>"

Quoted 47 lines

parse

2) some issues with LICENSEs:
INFO: Please add the following line for 'docs/LICENSE_OF_DEPENDENCIES.md' to a 'lib/recipetool/licenses.csv' and replace Unknown with the license:
c4807c5b058a56c02f592207b4e056b1,Unknown
Traceback (most recent call last):
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
111, in <module>
ret = main()
^^^^^^
File "/OE/build/oe-core/openembedded-core/scripts/recipetool", line
100, in main
ret = args.func(args)
^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create.py", line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after,
handled, extravalues)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 349, in process
self._rewrite_lic_uri(lines_before)
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 380, in _rewrite_lic_uri
updated, newlines = bb.utils.edit_metadata(lines_before,
['LIC_FILES_CHKSUM'], varfunc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1382, in
edit_metadata

     if handle_var_end():
        ^^^^^^^^^^^^^^^^

File "/OE/build/oe-core/bitbake/lib/bb/utils.py", line 1280, in
handle_var_end
(newvalue, newop, indent, minbreak) = varfunc(in_var, full_value,
op, newlines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/OE/layers/openembedded-core/scripts/lib/recipetool/create_go.py", line 372, in varfunc

     uri, chksum = license.split(';', 1)
     ^^^^^^^^^^^

ValueError: not enough values to unpack (expected 2, got 1)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

(the license was just '/')

3) some SRCREVs set to None leading to parsing failure:

   File "/OE/layers/bitbake/lib/bb/fetch2/git.py", line 762, in

_latest_revision
raise bb.fetch2.FetchError("Unable to resolve '%s' in upstream

git

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

repository in git ls-remote output for %s" % \
bb.data_smart.ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
FetchError: Fetcher failure: Unable to resolve 'None' in upstream git repository in git ls-remote output for github.com/googleapis/google-cloud-go
<http://github.com/googleapis/google-cloud-go>
The variable dependency chain for the failure is: SRCPV -> PV -> BP -> FILESPATH

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

ERROR: Parsing newly created recipe failed, moving recipe to

/OE/build/oe-core/workspace/recipes/telegraf/telegraf_git.bb.parsefailed

Quoted 23 lines
Quoted 22 lines
Quoted 10 lines
Quoted 9 lines

for reference. If this looks to be caused by the recipe itself, please report this error.

It takes long time to fetch all telegraf dependencies, so I'll create some test recipe to more easily debug these 3 issues in isolation.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178380):
https://lists.openembedded.org/g/openembedded-core/message/178380
Mute This Topic: https://lists.openembedded.org/mt/95827446/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#15 lore raw
In reply to Martin Jansa

I also tested using telegraf v1.21.4.
Kirkstone uses go1.17, so I tested this version.
Unfortunately, it failed with the following log.
Did I have something mistakes?

~$ recipetool create -S 583ee20a093acaa0781508b7840deb97a0fda622 -B release-1.21 https://github.com/influxdata/telegraf.git

Initialising tasks: 100% |################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)
NOTE: No setscene tasks
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 4 seconds
INFO: Resolved download path for import 'cloud.google.com/go/bigquery' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/monitoring' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/pubsub' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'collectd.org' => https://github.com/collectd/go-collectd
INFO: Resolved download path for import 'github.com/Azure/azure-event-hubs-go/v3' => https://github.com/Azure/azure-event-hubs-go
INFO: Resolved download path for import 'github.com/Azure/azure-kusto-go' => https://github.com/Azure/azure-kusto-go
INFO: Resolved download path for import 'github.com/Azure/azure-storage-queue-go' => https://github.com/Azure/azure-storage-queue-go
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/adal' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/Azure/go-autorest/autorest/azure/auth' => https://github.com/Azure/go-autorest
INFO: Resolved download path for import 'github.com/BurntSushi/toml' => https://github.com/BurntSushi/toml
INFO: Resolved download path for import 'github.com/Mellanox/rdmamap' => https://github.com/Mellanox/rdmamap
INFO: Resolved download path for import 'github.com/Shopify/sarama' => https://github.com/Shopify/sarama
INFO: Resolved download path for import 'github.com/aerospike/aerospike-client-go' => https://github.com/aerospike/aerospike-client-go
INFO: Resolved download path for import 'github.com/alecthomas/units' => https://github.com/alecthomas/units
INFO: Resolved download path for import 'github.com/aliyun/alibaba-cloud-sdk-go' => https://github.com/aliyun/alibaba-cloud-sdk-go
INFO: Resolved download path for import 'github.com/amir/raidman' => https://github.com/amir/raidman
INFO: Resolved download path for import 'github.com/antchfx/jsonquery' => https://github.com/antchfx/jsonquery
INFO: Resolved download path for import 'github.com/antchfx/xmlquery' => https://github.com/antchfx/xmlquery
INFO: Resolved download path for import 'github.com/antchfx/xpath' => https://github.com/antchfx/xpath
INFO: Resolved download path for import 'github.com/apache/thrift' => https://github.com/apache/thrift
INFO: Resolved download path for import 'github.com/aristanetworks/goarista' => https://github.com/aristanetworks/goarista
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/config' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/credentials' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/feature/ec2/imds' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatch' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/dynamodb' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/ec2' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/kinesis' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/sts' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/aws-sdk-go-v2/service/timestreamwrite' => https://github.com/aws/aws-sdk-go-v2
INFO: Resolved download path for import 'github.com/aws/smithy-go' => https://github.com/aws/smithy-go
INFO: Resolved download path for import 'github.com/benbjohnson/clock' => https://github.com/benbjohnson/clock
INFO: Resolved download path for import 'github.com/bmatcuk/doublestar/v3' => https://github.com/bmatcuk/doublestar
INFO: Resolved download path for import 'github.com/caio/go-tdigest' => https://github.com/caio/go-tdigest
INFO: Resolved download path for import 'github.com/cisco-ie/nx-telemetry-proto' => https://github.com/cisco-ie/nx-telemetry-proto
INFO: Resolved download path for import 'github.com/coreos/go-semver' => https://github.com/coreos/go-semver
INFO: Resolved download path for import 'github.com/couchbase/go-couchbase' => https://github.com/couchbase/go-couchbase
INFO: Resolved download path for import 'github.com/denisenkom/go-mssqldb' => https://github.com/denisenkom/go-mssqldb
INFO: Resolved download path for import 'github.com/dimchansky/utfbom' => https://github.com/dimchansky/utfbom
INFO: Resolved download path for import 'github.com/djherbis/times' => https://github.com/djherbis/times
INFO: Resolved download path for import 'github.com/docker/docker' => https://github.com/docker/docker
INFO: Resolved download path for import 'github.com/doclambda/protobufquery' => https://github.com/doclambda/protobufquery
INFO: Resolved download path for import 'github.com/dynatrace-oss/dynatrace-metric-utils-go' => https://github.com/dynatrace-oss/dynatrace-metric-utils-go
INFO: Resolved download path for import 'github.com/eclipse/paho.mqtt.golang' => https://github.com/eclipse/paho.mqtt.golang
INFO: Resolved download path for import 'github.com/fatih/color' => https://github.com/fatih/color
INFO: Resolved download path for import 'github.com/ghodss/yaml' => https://github.com/ghodss/yaml
INFO: Resolved download path for import 'github.com/go-ldap/ldap/v3' => https://github.com/go-ldap/ldap
INFO: Resolved download path for import 'github.com/go-logfmt/logfmt' => https://github.com/go-logfmt/logfmt
INFO: Resolved download path for import 'github.com/go-ping/ping' => https://github.com/go-ping/ping
INFO: Resolved download path for import 'github.com/go-redis/redis' => https://github.com/go-redis/redis
INFO: Resolved download path for import 'github.com/go-sql-driver/mysql' => https://github.com/go-sql-driver/mysql
INFO: Resolved download path for import 'github.com/gobwas/glob' => https://github.com/gobwas/glob
INFO: Resolved download path for import 'github.com/gofrs/uuid' => https://github.com/gofrs/uuid
INFO: Resolved download path for import 'github.com/golang-jwt/jwt/v4' => https://github.com/golang-jwt/jwt
INFO: Resolved download path for import 'github.com/golang/geo' => https://github.com/golang/geo
INFO: Resolved download path for import 'github.com/golang/snappy' => https://github.com/golang/snappy
INFO: Resolved download path for import 'github.com/google/go-cmp' => https://github.com/google/go-cmp
INFO: Resolved download path for import 'github.com/google/go-github/v32' => https://github.com/google/go-github
INFO: Resolved download path for import 'github.com/gopcua/opcua' => https://github.com/gopcua/opcua
INFO: Resolved download path for import 'github.com/gophercloud/gophercloud' => https://github.com/gophercloud/gophercloud
INFO: Resolved download path for import 'github.com/gorilla/mux' => https://github.com/gorilla/mux
INFO: Resolved download path for import 'github.com/gorilla/websocket' => https://github.com/gorilla/websocket
INFO: Resolved download path for import 'github.com/gosnmp/gosnmp' => https://github.com/gosnmp/gosnmp
INFO: Resolved download path for import 'github.com/grid-x/modbus' => https://github.com/grid-x/modbus
INFO: Resolved download path for import 'github.com/gwos/tcg/sdk' => https://github.com/gwos/tcg
INFO: Resolved download path for import 'github.com/harlow/kinesis-consumer' => https://github.com/harlow/kinesis-consumer
INFO: Resolved download path for import 'github.com/hashicorp/consul/api' => https://github.com/hashicorp/consul
INFO: Resolved download path for import 'github.com/hashicorp/go-uuid' => https://github.com/hashicorp/go-uuid
INFO: Resolved download path for import 'github.com/influxdata/go-syslog/v3' => https://github.com/influxdata/go-syslog
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/common' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/influx2otel' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/influxdb-observability/otel2influx' => https://github.com/influxdata/influxdb-observability
INFO: Resolved download path for import 'github.com/influxdata/tail' => https://github.com/influxdata/tail
INFO: Resolved download path for import 'github.com/influxdata/toml' => https://github.com/influxdata/toml
INFO: Resolved download path for import 'github.com/influxdata/wlog' => https://github.com/influxdata/wlog
INFO: Resolved download path for import 'github.com/intel/iaevents' => https://github.com/intel/iaevents
INFO: Resolved download path for import 'github.com/jackc/pgx/v4' => https://github.com/jackc/pgx
INFO: Resolved download path for import 'github.com/james4k/rcon' => https://github.com/james4k/rcon
INFO: Resolved download path for import 'github.com/jhump/protoreflect' => https://github.com/jhump/protoreflect
INFO: Resolved download path for import 'github.com/jmespath/go-jmespath' => https://github.com/jmespath/go-jmespath
INFO: Resolved download path for import 'github.com/kardianos/service' => https://github.com/kardianos/service
INFO: Resolved download path for import 'github.com/karrick/godirwalk' => https://github.com/karrick/godirwalk
INFO: Resolved download path for import 'github.com/kballard/go-shellquote' => https://github.com/kballard/go-shellquote
INFO: Resolved download path for import 'github.com/matttproud/golang_protobuf_extensions' => https://github.com/matttproud/golang_protobuf_extensions
INFO: Resolved download path for import 'github.com/mdlayher/apcupsd' => https://github.com/mdlayher/apcupsd
INFO: Resolved download path for import 'github.com/microsoft/ApplicationInsights-Go' => https://github.com/microsoft/ApplicationInsights-Go
INFO: Resolved download path for import 'github.com/miekg/dns' => https://github.com/miekg/dns
INFO: Resolved download path for import 'github.com/moby/ipvs' => https://github.com/moby/ipvs
INFO: Resolved download path for import 'github.com/multiplay/go-ts3' => https://github.com/multiplay/go-ts3
INFO: Resolved download path for import 'github.com/nats-io/nats-server/v2' => https://github.com/nats-io/nats-server
INFO: Resolved download path for import 'github.com/nats-io/nats.go' => https://github.com/nats-io/nats.go
INFO: Resolved download path for import 'github.com/newrelic/newrelic-telemetry-sdk-go' => https://github.com/newrelic/newrelic-telemetry-sdk-go
INFO: Resolved download path for import 'github.com/nsqio/go-nsq' => https://github.com/nsqio/go-nsq
INFO: Resolved download path for import 'github.com/olivere/elastic' => https://github.com/olivere/elastic
INFO: Resolved download path for import 'github.com/openconfig/gnmi' => https://github.com/openconfig/gnmi
INFO: Resolved download path for import 'github.com/opentracing/opentracing-go' => https://github.com/opentracing/opentracing-go
INFO: Resolved download path for import 'github.com/openzipkin-contrib/zipkin-go-opentracing' => https://github.com/openzipkin-contrib/zipkin-go-opentracing
INFO: Resolved download path for import 'github.com/openzipkin/zipkin-go' => https://github.com/openzipkin/zipkin-go
INFO: Resolved download path for import 'github.com/pion/dtls/v2' => https://github.com/pion/dtls
INFO: Resolved download path for import 'github.com/pkg/errors' => https://github.com/pkg/errors
INFO: Resolved download path for import 'github.com/prometheus/client_golang' => https://github.com/prometheus/client_golang
INFO: Resolved download path for import 'github.com/prometheus/client_model' => https://github.com/prometheus/client_model
INFO: Resolved download path for import 'github.com/prometheus/common' => https://github.com/prometheus/common
INFO: Resolved download path for import 'github.com/prometheus/procfs' => https://github.com/prometheus/procfs
INFO: Resolved download path for import 'github.com/prometheus/prometheus' => https://github.com/prometheus/prometheus
INFO: Resolved download path for import 'github.com/riemann/riemann-go-client' => https://github.com/riemann/riemann-go-client
INFO: Resolved download path for import 'github.com/safchain/ethtool' => https://github.com/safchain/ethtool
INFO: Resolved download path for import 'github.com/sensu/sensu-go/api/core/v2' => https://github.com/sensu/sensu-go
INFO: Resolved download path for import 'github.com/shirou/gopsutil/v3' => https://github.com/shirou/gopsutil
INFO: Resolved download path for import 'github.com/showwin/speedtest-go' => https://github.com/showwin/speedtest-go
INFO: Resolved download path for import 'github.com/signalfx/golib/v3' => https://github.com/signalfx/golib
INFO: Resolved download path for import 'github.com/sirupsen/logrus' => https://github.com/sirupsen/logrus
INFO: Resolved download path for import 'github.com/sleepinggenius2/gosmi' => https://github.com/sleepinggenius2/gosmi
INFO: Resolved download path for import 'github.com/snowflakedb/gosnowflake' => https://github.com/snowflakedb/gosnowflake
INFO: Resolved download path for import 'github.com/streadway/amqp' => https://github.com/streadway/amqp
INFO: Resolved download path for import 'github.com/stretchr/testify' => https://github.com/stretchr/testify
INFO: Resolved download path for import 'github.com/tbrandon/mbserver' => https://github.com/tbrandon/mbserver
INFO: Resolved download path for import 'github.com/testcontainers/testcontainers-go' => https://github.com/testcontainers/testcontainers-go
INFO: Resolved download path for import 'github.com/tidwall/gjson' => https://github.com/tidwall/gjson
INFO: Resolved download path for import 'github.com/tinylib/msgp' => https://github.com/tinylib/msgp
INFO: Resolved download path for import 'github.com/vapourismo/knx-go' => https://github.com/vapourismo/knx-go
INFO: Resolved download path for import 'github.com/vjeantet/grok' => https://github.com/vjeantet/grok
INFO: Resolved download path for import 'github.com/vmware/govmomi' => https://github.com/vmware/govmomi
INFO: Resolved download path for import 'github.com/wavefronthq/wavefront-sdk-go' => https://github.com/wavefronthq/wavefront-sdk-go
INFO: Resolved download path for import 'github.com/wvanbergen/kafka' => https://github.com/wvanbergen/kafka
INFO: Resolved download path for import 'github.com/xdg/scram' => https://github.com/xdg/scram
INFO: Resolved download path for import 'go.mongodb.org/mongo-driver' => https://github.com/mongodb/mongo-go-driver.git
INFO: Resolved download path for import 'go.opentelemetry.io/collector/model' => https://github.com/open-telemetry/opentelemetry-collector
INFO: Resolved download path for import 'go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.opentelemetry.io/otel/metric' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.opentelemetry.io/otel/sdk/metric' => https://github.com/open-telemetry/opentelemetry-go
INFO: Resolved download path for import 'go.starlark.net' => https://github.com/google/starlark-go
INFO: Resolved download path for import 'golang.org/x/net' => https://go.googlesource.com/net
INFO: Resolved download path for import 'golang.org/x/oauth2' => https://go.googlesource.com/oauth2
INFO: Resolved download path for import 'golang.org/x/sync' => https://go.googlesource.com/sync
INFO: Resolved download path for import 'golang.org/x/sys' => https://go.googlesource.com/sys
INFO: Resolved download path for import 'golang.org/x/text' => https://go.googlesource.com/text
INFO: Resolved download path for import 'golang.zx2c4.com/wireguard/wgctrl' => https://github.com/WireGuard/wgctrl-go
INFO: Resolved download path for import 'google.golang.org/api' => https://github.com/googleapis/google-api-go-client
INFO: Resolved download path for import 'google.golang.org/genproto' => https://github.com/googleapis/go-genproto
INFO: Resolved download path for import 'google.golang.org/grpc' => https://github.com/grpc/grpc-go
INFO: Resolved download path for import 'google.golang.org/protobuf' => https://go.googlesource.com/protobuf
INFO: Resolved download path for import 'gopkg.in/gorethink/gorethink.v3' => https://gopkg.in/gorethink/gorethink.v3
INFO: Resolved download path for import 'gopkg.in/olivere/elastic.v5' => https://gopkg.in/olivere/elastic.v5
INFO: Resolved download path for import 'gopkg.in/tomb.v1' => https://gopkg.in/tomb.v1
INFO: Resolved download path for import 'gopkg.in/yaml.v2' => https://gopkg.in/yaml.v2
INFO: Resolved download path for import 'k8s.io/api' => https://github.com/kubernetes/api
INFO: Resolved download path for import 'k8s.io/apimachinery' => https://github.com/kubernetes/apimachinery
INFO: Resolved download path for import 'k8s.io/client-go' => https://github.com/kubernetes/client-go
INFO: Resolved download path for import 'modernc.org/sqlite' => https://gitlab.com/cznic/sqlite
INFO: Resolved download path for import 'cloud.google.com/go' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/compute' => https://github.com/googleapis/google-cloud-go
INFO: Resolved download path for import 'cloud.google.com/go/iam' => https://github.com/googleapis/google-cloud-go
Traceback (most recent call last):
File "/oe-core/scripts/recipetool", line 111, in <module>
ret = main()
File "/oe-core/scripts/recipetool", line 100, in main
ret = args.func(args)
File "/oe-core/scripts/lib/recipetool/create.py", line 746, in create_recipe
handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 336, in process
dep_src_uris, src_revs = self._handle_dependencies(d, srctree, go_mod)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 261, in _handle_dependencies
repodata = self._resolve_repository(module_path)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 184, in _resolve_repository
repodata = self._resolve_repository_dynamic(modulepath)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 176, in _resolve_repository_dynamic
return GoImport(parser.rootpath, parser.vcs, parser.repourl, None)
File "/oe-core/scripts/lib/recipetool/create_go.py", line 151, in rootpath
return self.__srv[0]
IndexError: list index out of range