Skip to contentSkip to conversation

cargo_common.bbclass: do not use buit-in git to fetch crates

PATCH In review [email protected] 3 participants 28 messages Mar 12, 2023 – Mar 14, 2023 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cargo_common.bbclass: do not use buit-in git to fetch crates

From: Frederic Martinsons <[email protected]>

When a project has a dependencies from a git repository and ssh access, this kind of thing in Cargo.toml for example:

[dependencies]
my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }

do_compile step failed with authentication error:

|     Updating git repository `ssh://git@myserver:7999/myrepo.git`
| error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
|
| Caused by:
|   failed to load source for dependency `my-crate`
|
| Caused by:
|   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
|
| Caused by:
|   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
|
| Caused by:
|   failed to authenticate when downloading repository
|
|   * attempted ssh-agent authentication, but no usernames succeeded: `git`
|
|   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
|   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|   error authenticating: no auth sock variable; class=Ssh (23)

No problem if I run "cargo build" manually.

As the compiler suggested, it is corrected by adding git-fetch-with-cli in cargo config file.

---
 meta/classes-recipe/cargo_common.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
meta/classes-recipe/cargo_common.bbclass +6 -0
index f503a001dd..c16aa4934b 100644
@@ -114,6 +114,12 @@ cargo_common_do_configure () {
 	progress.when = 'always'
 	progress.width = 80
 	EOF
+
+	cat <<- EOF >> ${CARGO_HOME}/config
+
+	[net]
+	git-fetch-with-cli = true
+	EOF
 }
 
 oe_cargo_fix_env () {
Signature · 1 line
2.34.1
#1 lore raw

Not sure if I understand this fully, but network access in do_compile is not allowed. You need to place all crates into SRC_URI.

Alex

Quoted 73 lines · Frederic Martinsons

From: Frederic Martinsons <[email protected]>

When a project has a dependencies from a git repository and ssh access, this kind of thing in Cargo.toml for example:

[dependencies]
my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }

do_compile step failed with authentication error:

|     Updating git repository `ssh://git@myserver:7999/myrepo.git`
| error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
|
| Caused by:
|   failed to load source for dependency `my-crate`
|
| Caused by:
|   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
|
| Caused by:
|   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
|
| Caused by:
|   failed to authenticate when downloading repository
|
|   * attempted ssh-agent authentication, but no usernames succeeded: `git`
|
|   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
|   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|   error authenticating: no auth sock variable; class=Ssh (23)

No problem if I run "cargo build" manually.

As the compiler suggested, it is corrected by adding git-fetch-with-cli in cargo config file.

---
 meta/classes-recipe/cargo_common.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
meta/classes-recipe/cargo_common.bbclass +6 -1
index f503a001dd..c16aa4934b 100644
@@ -114,6 +114,12 @@ cargo_common_do_configure () {
        progress.when = 'always'
        progress.width = 80
        EOF
+
+       cat <<- EOF >> ${CARGO_HOME}/config
+
+       [net]
+       git-fetch-with-cli = true
+       EOF
 }
 
 oe_cargo_fix_env () {
--

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178375): https://lists.openembedded.org/g/openembedded-core/message/178375
Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Yes I did do that in a first place (because this is what cargo bitbake give to me in a first place), but for git dependencies this doesn't work.
So , to take the example again I had that in SRC_URI (after correcting malformed url as tracked by
https://github.com/meta-rust/cargo-bitbake/issues/41):

"git://git@myserver:7999/myrepo.git;protocol=ssh;nobranch=1;name=my-crate;destsuffix=my-crate"

And also the SRCREV:
SRCREV_FORMAT .= "_my-crate"
SRCREV_my-crate = "main"
EXTRA_OECARGO_PATHS += "${WORKDIR}/my-crate"

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

Quoted 81 lines · Alexander Kanavin

Not sure if I understand this fully, but network access in do_compile is not allowed. You need to place all crates into SRC_URI.

Alex

Quoted 73 lines · Frederic Martinsons

From: Frederic Martinsons <[email protected]>

When a project has a dependencies from a git repository and ssh access, this kind of thing in Cargo.toml for example:

[dependencies]
my-crate = { git = "ssh://git@smyserver:7999/myrepo.git, branch="main" }

do_compile step failed with authentication error:

|     Updating git repository `ssh://git@myserver:7999/myrepo.git`
| error: failed to get `my-crate` as a dependency of package `bscli v0.0.1 (/home/fmartinsons/bscli)`
|
| Caused by:
|   failed to load source for dependency `my-crate`
|
| Caused by:
|   Unable to update ssh://git@myserver:7999/myrepo.git?branch=main
|
| Caused by:
|   failed to fetch into: /home/fmartinsons/TAPOS_build/build-tapos/tmp/work/corei7-64-tapos-linux/bscli/v1.0.0+gitAUTOINC+5b5ba2aa3e-r0/cargo_home/git/db/cra-d8e163876777fda6
|
| Caused by:
|   failed to authenticate when downloading repository
|
|   * attempted ssh-agent authentication, but no usernames succeeded: `git`
|
|   if the git CLI succeeds then `net.git-fetch-with-cli` may help here
|   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|   error authenticating: no auth sock variable; class=Ssh (23)

No problem if I run "cargo build" manually.

As the compiler suggested, it is corrected by adding git-fetch-with-cli in cargo config file.

---
 meta/classes-recipe/cargo_common.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
meta/classes-recipe/cargo_common.bbclass +6 -1
index f503a001dd..c16aa4934b 100644
@@ -114,6 +114,12 @@ cargo_common_do_configure () {
        progress.when = 'always'
        progress.width = 80
        EOF
+
+       cat <<- EOF >> ${CARGO_HOME}/config
+
+       [net]
+       git-fetch-with-cli = true
+       EOF
 }
 
 oe_cargo_fix_env () {
--

2.34.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178375): https://lists.openembedded.org/g/openembedded-core/message/178375
Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

#4 lore raw

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

#5 lore raw

Try with this patch:

https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/

Quoted 35 lines · Frederic Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 14 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

Quoted 5 lines

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178417): https://lists.openembedded.org/g/openembedded-core/message/178417
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 1 line
Alex Kiernan
#6 lore raw
In reply to Frédéric Martinsons

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

#7 lore raw

ISTR I tried this and ran into a different set of different problems.

Quoted 45 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 1 line
Alex Kiernan
#8 lore raw

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

#9 lore raw

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

Signature · 1 line
Alex Kiernan
#10 lore raw

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

-

Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

#11 lore raw

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Alex

Quoted 87 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -
Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

#12 lore raw

I understand, I'll pursue all my tests with the current poky master. Thanks

Quoted 99 lines · Alexander Kanavin

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Alex

Quoted 87 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -
Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

#13 lore raw

Hello, I finally found why my setup didn't work. That was not related to https versus ssh.
For the patch provided in
https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/
to work, the repository should have a Cargo.lock file. WIthout the Cargo.lock file, the cargo build will try to generate it (and so needs network access).

Note that I made a little change to the patch to support user in url (the patch process in cargo only work if url is exact match):

@@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
name = ud.parm.get('name')
destsuffix = ud.parm.get('destsuffix') if name is not None and destsuffix is not None:
- repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+ if ud.user:
+ repo = '%s://%s@%s%s' % (ud.proto, ud.user,
ud.host, ud.path)
+ else:
+ repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)

                 path = '%s = { path = "%s" }' % (name,
os.path.join(workdir, destsuffix))
                 patches.setdefault(repo, []).append(path)

Anyway, I would like to know how to you feel with this need for Cargo.lock problem ?
Because for it to be generated, the project needs to be built once and if this is expected for binary type to have Cargo.lock under version control:
https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
It is not for "library" project.

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
| NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
--release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml | error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  can't checkout from

'ssh://[email protected]/fmartinsons/zbus-git-dep-test.git': you are in the offline mode (--offline)
| WARNING: exit code 101 from a shell command.

Instead of

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/ | NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
| Updating git repository
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git
| warning: spurious network error (2 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| warning: spurious network error (1 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  failed to clone into:
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
|
| Caused by:
|  network failure seems to have happened
|  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
|  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|  failed to resolve address for gitlab.com: Temporary failure in name

resolution; class=Net (12)|
| WARNING: exit code 101 from a shell command.

In any case, I'll be glad to work on this patch if you tell me what strategy to adopt.

Quoted 103 lines · Frédéric Martinsons

I understand, I'll pursue all my tests with the current poky master. Thanks

Quoted 99 lines · Alexander Kanavin

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Alex

Quoted 87 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -
Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

#14 lore raw

--offline seems like the right thing to add, if it produces better errors.

Generating Cargo.lock on the other hand is not right. We rely on rust checking the source tree against the checksums in Cargo.lock (to prevent supply chain attacks), and this would completely subvert that. There could be better diagnostics around missing Cargo.lock, and steps to address the issue, but such steps must be manually taken (even though they could be codified in a special task, similar to how generating list of crates in SRC_URI is a task).

Alex

Quoted 215 lines · Frédéric Martinsons

Hello, I finally found why my setup didn't work. That was not related to https versus ssh.
For the patch provided in
https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/
to work, the repository should have a Cargo.lock file. WIthout the Cargo.lock file, the cargo build will try to generate it (and so needs network access).

Note that I made a little change to the patch to support user in url (the patch process in cargo only work if url is exact match):

(quoted diff) +4 -1
@@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
             name = ud.parm.get('name')
             destsuffix = ud.parm.get('destsuffix')
             if name is not None and destsuffix is not None:
-                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                if ud.user:
+                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
 ud.host, ud.path)
+                else:
+                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
                 path = '%s = { path = "%s" }' % (name,
 os.path.join(workdir, destsuffix))
                 patches.setdefault(repo, []).append(path)

Anyway, I would like to know how to you feel with this need for Cargo.lock problem ?
Because for it to be generated, the project needs to be built once and if this is expected for binary type to have Cargo.lock under version control:
https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
It is not for "library" project.

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
| NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
--release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml | error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  can't checkout from

'ssh://[email protected]/fmartinsons/zbus-git-dep-test.git': you are in the offline mode (--offline)
| WARNING: exit code 101 from a shell command.

Instead of

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/ | NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
| Updating git repository
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git
| warning: spurious network error (2 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| warning: spurious network error (1 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  failed to clone into:
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
|
| Caused by:
|  network failure seems to have happened
|  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
|  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|  failed to resolve address for gitlab.com: Temporary failure in name

resolution; class=Net (12)|
| WARNING: exit code 101 from a shell command.

In any case, I'll be glad to work on this patch if you tell me what strategy to adopt.

Quoted 103 lines · Frédéric Martinsons

I understand, I'll pursue all my tests with the current poky master. Thanks

Quoted 99 lines · Alexander Kanavin

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Alex

Quoted 87 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -
Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

#15 lore raw
Quoted 11 lines · Alexander Kanavin

--offline seems like the right thing to add, if it produces better errors.

Generating Cargo.lock on the other hand is not right. We rely on rust checking the source tree against the checksums in Cargo.lock (to prevent supply chain attacks), and this would completely subvert that. There could be better diagnostics around missing Cargo.lock, and steps to address the issue, but such steps must be manually taken (even though they could be codified in a special task, similar to how generating list of crates in SRC_URI is a task).

Quoted 2 lines

Alex

Ok I see what you mean, generating the Cargo.lock in a dedicated task inside meta/classes-recipe/cargo-update-recipe-crates.bbclass or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass ?

#16 lore raw
Quoted 13 lines · Frédéric Martinsons
Quoted 11 lines · Alexander Kanavin

--offline seems like the right thing to add, if it produces better errors.

Generating Cargo.lock on the other hand is not right. We rely on rust checking the source tree against the checksums in Cargo.lock (to prevent supply chain attacks), and this would completely subvert that. There could be better diagnostics around missing Cargo.lock, and steps to address the issue, but such steps must be manually taken (even though they could be codified in a special task, similar to how generating list of crates in SRC_URI is a task).

Quoted 6 lines
Quoted 2 lines

Alex

Ok I see what you mean, generating the Cargo.lock in a dedicated task inside meta/classes-recipe/cargo-update-recipe-crates.bbclass or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass ?

I'm really not sure we want Cargo.lock generation as part of the OE workflow - Cargo.lock should be part of your checked in sources for non-library projects.

--
Alex Kiernan

#17 lore raw

Understood, I searched how to add checksum support in crate fetcher but I don't know enough of bitbake to be effective.

I'll check in the Cargo.lock in my project, backport your patch and continue my exploration of the wonderful rust world on embedded device ^^

Thank you very much for the help on the subject guys and like I said earlier, if you need me to do some more tests to help the patch being merged in the master, I'm available.

Quoted 33 lines · Alex Kiernan
Quoted 2 lines · Frédéric Martinsons

On Tue, 14 Mar 2023 at 11:35, Alexander Kanavin <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines

--offline seems like the right thing to add, if it produces better

errors.

Quoted 9 lines
Quoted 9 lines

Generating Cargo.lock on the other hand is not right. We rely on rust checking the source tree against the checksums in Cargo.lock (to prevent supply chain attacks), and this would completely subvert that. There could be better diagnostics around missing Cargo.lock, and steps to address the issue, but such steps must be manually taken (even though they could be codified in a special task, similar to how generating list of crates in SRC_URI is a task).

Quoted 6 lines
Quoted 2 lines

Alex

Ok I see what you mean, generating the Cargo.lock in a dedicated task inside meta/classes-recipe/cargo-update-recipe-crates.bbclass or maybe a dedicated meta/classes-recipe/cargo-generate-lockfile.bbclass

?

I'm really not sure we want Cargo.lock generation as part of the OE workflow - Cargo.lock should be part of your checked in sources for non-library projects.

--
Alex Kiernan

#18 lore raw
Quoted 1 line · Frédéric Martinsons

Understood, I searched how to add checksum support in crate fetcher but I don't know enough of bitbake to be effective.

You can check how for example http:// fetcher does it by utilizing verify_checksum() from lib/bb/fetch2/__init__.py in lib/bb/fetch2/wget.py.

Something like that should happen in crate.py as well.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#20 lore raw

The other option is to add checksumming support to crate fetcher in bitbake. If all items in src_uri are verified by fetchers directly for integrity from checksums in the recipe, the cargo can generate cargo.lock anytime it wants to.

Alex

Quoted 274 lines · Alexander Kanavin via lists.openembedded.org

--offline seems like the right thing to add, if it produces better errors.

Generating Cargo.lock on the other hand is not right. We rely on rust checking the source tree against the checksums in Cargo.lock (to prevent supply chain attacks), and this would completely subvert that. There could be better diagnostics around missing Cargo.lock, and steps to address the issue, but such steps must be manually taken (even though they could be codified in a special task, similar to how generating list of crates in SRC_URI is a task).

Alex

Quoted 5 lines · Frédéric Martinsons

Hello, I finally found why my setup didn't work. That was not related to https versus ssh.
For the patch provided in

https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/

Quoted 21 lines

to work, the repository should have a Cargo.lock file. WIthout the Cargo.lock file, the cargo build will try to generate it (and so needs network access).

Note that I made a little change to the patch to support user in url (the patch process in cargo only work if url is exact match):

(quoted diff) +4 -1
@@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
             name = ud.parm.get('name')
             destsuffix = ud.parm.get('destsuffix')
             if name is not None and destsuffix is not None:
-                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                if ud.user:
+                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
 ud.host, ud.path)
+                else:
+                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
                 path = '%s = { path = "%s" }' % (name,
 os.path.join(workdir, destsuffix))
                 patches.setdefault(repo, []).append(path)
Quoted 6 lines

Anyway, I would like to know how to you feel with this need for Cargo.lock problem ?
Because for it to be generated, the project needs to be built once and if this is expected for binary type to have Cargo.lock under version control:

https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries

Quoted 11 lines

It is not for "library" project.

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from

/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/

Quoted 2 lines

| NOTE: cargo =

/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo

Quoted 2 lines

| NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu --release

--manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml

Quoted 3 lines

| error: failed to get zbus-git-dep-test as a dependency of package `zbus-test v0.0.1

(/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`

Quoted 10 lines
|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  can't checkout from
Quoted 9 lines

'ssh://[email protected]/fmartinsons/zbus-git-dep-test.git': you are in the offline mode (--offline)
| WARNING: exit code 101 from a shell command.

Instead of

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from

/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/

Quoted 2 lines

| NOTE: cargo =

/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo

Quoted 2 lines

| NOTE: cargo build -v --target x86_64-poky-linux-gnu --release

--manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml

Quoted 2 lines

| Updating git repository
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git

<http://[email protected]/fmartinsons/zbus-git-dep-test.git>

Quoted 9 lines

| warning: spurious network error (2 tries remaining): failed to resolve address for gitlab.com: Temporary failure in name resolution; class=Net (12)
| warning: spurious network error (1 tries remaining): failed to resolve address for gitlab.com: Temporary failure in name resolution; class=Net (12)
| error: failed to get zbus-git-dep-test as a dependency of package `zbus-test v0.0.1

(/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)`

Quoted 11 lines
|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  failed to clone into:

/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5

Quoted 4 lines
|
| Caused by:
|  network failure seems to have happened
|  if a proxy or similar is necessary `net.git-fetch-with-cli` may help

here

Quoted 1 line

|

https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Quoted 7 lines
|
| Caused by:
|  failed to resolve address for gitlab.com: Temporary failure in name

resolution; class=Net (12)|
| WARNING: exit code 101 from a shell command.

Quoted 7 lines

In any case, I'll be glad to work on this patch if you tell me what strategy to adopt.

Quoted 2 lines · Frédéric Martinsons

I understand, I'll pursue all my tests with the current poky master.

Thanks

Quoted 2 lines
Quoted 2 lines

On Mon, 13 Mar 2023 at 11:42, Alexander Kanavin <

Quoted 8 lines · [email protected]> wrote:
Quoted 8 lines
Quoted 8 lines

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Alex

Quoted 4 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is

successful.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

One major thing I forget to mention is that I use dunfell branch

where

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I backport all the cargo/rust relatives from master. The backport was functional and straightforward, I just have to

keep

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -

On Mon, 13 Mar 2023 at 10:46, Alex Kiernan <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Are you listing your ssh repos in your SRC_URI? We run with

network

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

disabled everywhere other than fetch and with that patch we

don't see

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

fetches from local ssh repos in compile.

Quoted 2 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened

(git

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way

of

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

supporting git dependencies in Cargo.toml

On Mon, 13 Mar 2023 at 10:23, Alex Kiernan <

Quoted 2 lines · [email protected]> wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

ISTR I tried this and ran into a different set of different

problems.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in

yocto is still

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

evolving towards maturity. If you can see better ways to

do things,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

patches would be most welcome - but 'no network access

except in

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

bitbake fetchers' is a core rule.

Alex

Quoted 2 lines · Frédéric Martinsons

I understand there should not be a fetch during build

step , I'll try

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to find why .
In the mean time, if this is a strong rule, shouldn't we

invoke "cargo

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

build" with --offline option ?

On Mon, 13 Mar 2023 at 09:38, Alexander Kanavin <

Quoted 4 lines · [email protected]> wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line · Frédéric Martinsons

But the compile step tried to git clone anyway and

so it failed like I

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

showed in my first email.

You need to figure out why this happens, because it

should not. We

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

have plenty of recipes that list crates in SRC_URI in

oe-core, so you

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

could start by studying how they work.

Accessing network anywhere except do_fetch is not

allowed for

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

reproducibility and license compliance reasons (both

break if that

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

happens).

'cargo bitbake' is a 3rd party project with

questionable maintenance

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

status that we do not support. The official way is to

use

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

cargo-update-recipe-crates class.

Alex

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

--
Alex Kiernan

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#178480):
https://lists.openembedded.org/g/openembedded-core/message/178480
Mute This Topic: https://lists.openembedded.org/mt/97555635/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#21 lore raw
Quoted 5 lines · Alexander Kanavin

The other option is to add checksumming support to crate fetcher in bitbake. If all items in src_uri are verified by fetchers directly for integrity from checksums in the recipe, the cargo can generate cargo.lock anytime it wants to.

Alex

Ok, so If I understand correctly, that would mean a modification inside bitbake/lib/bb/fetch2/crate.py to add such support ?
And if we have that, where the Cargo.lock generation should be made ? it can only be during do_fetch step since it will
require network if a "git" dependencies is to be found in the project manifest.

#22 lore raw
Quoted 5 lines · Frédéric Martinsons
Quoted 3 lines · Alexander Kanavin

The other option is to add checksumming support to crate fetcher in bitbake. If all items in src_uri are verified by fetchers directly for integrity from checksums in the recipe, the cargo can generate cargo.lock anytime it wants to.

Quoted 8 lines
Quoted 2 lines

Alex

Ok, so If I understand correctly, that would mean a modification inside bitbake/lib/bb/fetch2/crate.py to add such support ?
And if we have that, where the Cargo.lock generation should be made ? it can only be during do_fetch step since it will
require network if a "git" dependencies is to be found in the project manifest.

You'd have to list all the crate checksums in your recipe, so the crate fetcher could check them.

--
Alex Kiernan

#23 lore raw
In reply to Frédéric Martinsons
Quoted 24 lines · Frédéric Martinsons

Hello, I finally found why my setup didn't work. That was not related to https versus ssh.
For the patch provided in
https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]/
to work, the repository should have a Cargo.lock file. WIthout the Cargo.lock file, the cargo build will try to generate it (and so needs network access).

Note that I made a little change to the patch to support user in url (the patch process in cargo only work if url is exact match):

(quoted diff) +4 -1
@@ -135,7 +135,10 @@ python cargo_common_do_patch_paths() {
             name = ud.parm.get('name')
             destsuffix = ud.parm.get('destsuffix')
             if name is not None and destsuffix is not None:
-                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                if ud.user:
+                    repo = '%s://%s@%s%s' % (ud.proto, ud.user,
 ud.host, ud.path)
+                else:
+                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
                 path = '%s = { path = "%s" }' % (name,
 os.path.join(workdir, destsuffix))
Quoted 2 lines
                 patches.setdefault(repo, []).append(path)

Cool, will pick that up... the code needs some tests which is where it stalled last time.

Quoted 8 lines

Anyway, I would like to know how to you feel with this need for Cargo.lock problem ?
Because for it to be generated, the project needs to be built once and if this is expected for binary type to have Cargo.lock under version control:
https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
It is not for "library" project.

So I'm clear, you're building a library project here?

Could you consume your library into a wrapper project which brings Cargo.lock along with it? Other than running tests on it, I'm not sure what the value in building a library project is.

Quoted 6 lines

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

Let me chuck that across our code base, I was fairly sure I'd seen different problems with it, but it may have been whilst I was debugging this first time around and am mis-remembering.

Quoted 175 lines

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/
| NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --offline --target x86_64-poky-linux-gnu
--release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml | error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  can't checkout from

'ssh://[email protected]/fmartinsons/zbus-git-dep-test.git': you are in the offline mode (--offline)
| WARNING: exit code 101 from a shell command.

Instead of

| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/rust-targets/ | NOTE: cargo = /var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/recipe-sysroot-native/usr/bin/cargo | NOTE: cargo build -v --target x86_64-poky-linux-gnu --release --manifest-path=/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git//Cargo.toml
| Updating git repository
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git
| warning: spurious network error (2 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| warning: spurious network error (1 tries remaining): failed to
resolve address for gitlab.com: Temporary failure in name resolution;
class=Net (12)
| error: failed to get zbus-git-dep-test as a dependency of package
zbus-test v0.0.1 (/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/git)

|
| Caused by:
|  failed to load source for dependency `zbus-git-dep-test`
|
| Caused by:
|  Unable to update
ssh://[email protected]/fmartinsons/zbus-git-dep-test.git?branch=main
|
| Caused by:
|  failed to clone into:
/var/lib/jenkins/YOCTO_POKY_MASTER_TEST/poky/build/tmp/work/core2-64-poky-linux/zbus-test/0.1.0.AUTOINC+507d3327e2-r0/cargo_home/git/db/zbus-git-dep-test-7f2b6322e3bb0cd5
|
| Caused by:
|  network failure seems to have happened
|  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
|  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
|
| Caused by:
|  failed to resolve address for gitlab.com: Temporary failure in name

resolution; class=Net (12)|
| WARNING: exit code 101 from a shell command.

In any case, I'll be glad to work on this patch if you tell me what strategy to adopt.

Quoted 103 lines · Frédéric Martinsons

I understand, I'll pursue all my tests with the current poky master. Thanks

Quoted 99 lines · Alexander Kanavin

You need to use kirkstone or newer to see the bitbake-driven network disabling in action. Can you try on current poky master? You don't need to port the whole build to that, just one sample recipe.

It is also expected that if you submit patches for master, they were tested on master.

Alex

Quoted 87 lines · Frédéric Martinsons

hmm, how is the network disabled ?

Because with the patch I provided , the git clone in do_compile is successful.

One major thing I forget to mention is that I use dunfell branch where I backport all the cargo/rust relatives from master.
The backport was functional and straightforward, I just have to keep the custom crate fetcher in openembedded-core (meta/lib/crate.py) because I'm stuck with bitbake 1.46.

   -
Quoted 73 lines · Alex Kiernan

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 61 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 54 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 37 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178420): https://lists.openembedded.org/g/openembedded-core/message/178420
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan

--
Alex Kiernan

Signature · 1 line
Alex Kiernan
#24 lore raw
Quoted 2 lines

Cool, will pick that up... the code needs some tests which is where it stalled last time.

Any help I can provide to you for having this patch merge ? Are you talking about unit testing ?

Quoted 10 lines
Quoted 3 lines

https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
It is not for "library" project.

So I'm clear, you're building a library project here?

Could you consume your library into a wrapper project which brings Cargo.lock along with it? Other than running tests on it, I'm not sure what the value in building a library project is.

No, I build a binary but was not aware of the recommendation to add Cargo.lock in version control (and was not aware that was the missing of this file which make
cargo execute network operation during build),
I just started using rust several weeks ago so every day I learn new things ;)

Quoted 11 lines
Quoted 6 lines

Maybe we can add a "cargo generate-lockfile"
(https://doc.rust-lang.org/cargo/commands/cargo-generate-lockfile.html)
at the end of do_fetch ?
Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

Let me chuck that across our code base, I was fairly sure I'd seen different problems with it, but it may have been whilst I was debugging this first time around and am mis-remembering.

Alright.

#25 lore raw
In reply to Alex Kiernan
Quoted 4 lines · Alex Kiernan
Quoted 1 line · Frédéric Martinsons
Quoted 8 lines
Quoted 3 lines

Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

Let me chuck that across our code base, I was fairly sure I'd seen different problems with it, but it may have been whilst I was debugging this first time around and am mis-remembering.

I'm wrong, seems to be fine.

Signature · 1 line
Alex Kiernan
#26 lore raw

I think I'll dig into the checksum capability on crate fetcher in the coming days.

For the offline option and the modication of the patch mentioned above, how do you want me to proceed?

Submit a dedicated patch for each of these (one for offline option and one for checksum if I can come up with a suitable patch)?

For the modification of the patch for using a local directory and not relying on git clone during build time, do you want me to provide something in particular?

Note that I saw the downloaded crates not being cleaned up from DL_DIR when I issue a bitbake -c cleanall but I need to confirm this on master branch.

Quoted 20 lines · Alex Kiernan
Quoted 4 lines · Alex Kiernan
Quoted 1 line · Frédéric Martinsons
Quoted 8 lines
Quoted 3 lines

Moreover, I think we should add the --offline option to cargo build because the error generated will be more clear:

Let me chuck that across our code base, I was fairly sure I'd seen different problems with it, but it may have been whilst I was debugging this first time around and am mis-remembering.

I'm wrong, seems to be fine.

--
Alex Kiernan

A message that is not in the archive (sent off-list or not yet received): [email protected]

#28 lore raw

For reference this was my test case:

https://github.com/akiernan/uuid-test

I was wondering if this was an ssh vs. https problem, but just checking we've a mix of ssh and https access to local git repos.

Quoted 76 lines · Alex Kiernan via lists.openembedded.org

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 55 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 48 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 6 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Quoted 33 lines

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

--
Alex Kiernan

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178424): https://lists.openembedded.org/g/openembedded-core/message/178424
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 1 line
Alex Kiernan
#29 lore raw

Thank you very much, I tested your project and it worked.

So I'll try https for my git instead of ssh.

Quoted 91 lines · Alex Kiernan

For reference this was my test case:

https://github.com/akiernan/uuid-test

I was wondering if this was an ssh vs. https problem, but just checking we've a mix of ssh and https access to local git repos.

Quoted 68 lines · Alex Kiernan via lists.openembedded.org

Are you listing your ssh repos in your SRC_URI? We run with network disabled everywhere other than fetch and with that patch we don't see fetches from local ssh repos in compile.

Quoted 55 lines · Frédéric Martinsons

Thanks for the patch alex, I did try it and the same happened (git clone failure during do_compile), I'll
dig more on that and possibly submit patches if I found a way of supporting git dependencies in Cargo.toml

Quoted 48 lines · Alex Kiernan

ISTR I tried this and ran into a different set of different problems.

Quoted 6 lines · Alexander Kanavin

Note that we're all rust neophytes, and rust support in yocto is still evolving towards maturity. If you can see better ways to do things, patches would be most welcome - but 'no network access except in bitbake fetchers' is a core rule.

Quoted 33 lines

Alex

Quoted 26 lines · Frédéric Martinsons

I understand there should not be a fetch during build step , I'll try to find why .
In the mean time, if this is a strong rule, shouldn't we invoke "cargo build" with --offline option ?

Quoted 19 lines · Alexander Kanavin
Quoted 2 lines · Frédéric Martinsons

But the compile step tried to git clone anyway and so it failed like I showed in my first email.

You need to figure out why this happens, because it should not. We have plenty of recipes that list crates in SRC_URI in oe-core, so you could start by studying how they work.

Accessing network anywhere except do_fetch is not allowed for reproducibility and license compliance reasons (both break if that happens).

'cargo bitbake' is a 3rd party project with questionable maintenance status that we do not support. The official way is to use cargo-update-recipe-crates class.

Alex

--
Alex Kiernan

--
Alex Kiernan

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#178424): https://lists.openembedded.org/g/openembedded-core/message/178424
Mute This Topic: https://lists.openembedded.org/mt/97555635/3618097
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alex Kiernan