devtools: Initial recipe for repo 2.17.3
PATCH 1/2 devtools: Initial recipe for repo 2.17.3
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
--- .../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 +++++++++++++++++++ meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++ 4 files changed, 86 insertions(+) create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +21 -0
@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch new file +33 -0
@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Python modules +using GPG-signed git tags, and stores those files as part of the project +that it is working with. + +So in order to have a reproducible repo installation within the project +folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1
meta/recipes-devtools/repo/repo.inc new file +25 -0
@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow." +HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main" +MIRRORS = "git://[email protected]/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n" + +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"
meta/recipes-devtools/repo/repo_2.17.3.bb new file +7 -0
@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
Signature · 1 line
2.33.1
PATCH 2/2 base.bbclass: Add sysroot deps for repo fetcher
From: Jasper Orschulko <[email protected]>
Add git-native and repo-native as prerequisite for the repo fetcher.
--- meta/classes/base.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes/base.bbclass +5 -0
@@ -665,6 +665,11 @@ python () { elif uri.scheme == "npm": d.appendVarFlag('do_fetch', 'depends', ' nodejs-native:do_populate_sysroot') + elif uri.scheme == "repo": + needsrcrev = True + d.appendVarFlag('do_fetch', 'depends', ' repo-native:do_populate_sysroot') + d.appendVarFlag('do_fetch', 'depends', ' git-native:do_populate_sysroot') + # *.lz4 should DEPEND on lz4-native for unpacking if path.endswith('.lz4'): d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
Signature · 1 line
2.33.1
Is this implementation of repo being patched against phoning home to google everytime and trying to do that pesky auto-update of the tool?
Otherwise this will become an issue when at one point in time an (then) outdated version of the tool is used in a BB_NO_NETWORK build.
Quoted 137 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>--- .../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 +++++++++++++++++++ meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++ 4 files changed, 86 insertions(+)create mode 100644
meta/recipes-devtools/repo/files/0001-python3-shebang.patchcreate mode 100644meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchcreate mode 100644 meta/recipes-devtools/repo/repo.inc
create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bbmeta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +21
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch new file +33
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Python modules +using GPG-signed git tags, and stores those files as part of the project +that it is working with. + +So in order to have a reproducible repo installation within the project +folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes-devtools/repo/repo.inc new file +25
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow." +HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main" +MIRRORS = "git://[email protected]/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n" + +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-1@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57" -=-=-=-=-=-=-=-=-=-=-=-Links: You receive all messages sent to this group.
View/Reply Online (#12893): https://lists.openembedded.org/g/bitbake-devel/message/12893
Mute This Topic: https://lists.openembedded.org/mt/86840271/3647476
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Konrad,
that might be a good point. I patched the REPO_REV to a fixed refspec within the wrapper tool, as to stop repo from using the latest stable release. But you are right, and this might not be enough. Will do some more digging on this.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 158 lines · Konrad Weihmann
Is this implementation of repo being patched against phoning home to google everytime and trying to do that pesky auto-update of the tool?
Otherwise this will become an issue when at one point in time an (then)
outdated version of the tool is used in aBB_NO_NETWORKbuild.Quoted 150 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>---
.../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
+++++++++++++++++++
meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++4 files changed, 86 insertions(+)create mode 100644 meta/recipes-devtools/repo/files/0001-python3- shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001- Set-REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb(quoted diff) +0
-0shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- shebang.patch
new file mode 100644
index 0000000000..09ccf58264meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +1
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +20
-0+From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0(quoted diff) +0
-0REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 0000000000..294a3af53ameta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +0
-0v2.17.3.patch
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +1
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +5
-0+From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Pythonmodules
+using GPG-signed git tags, and stores those files as part of the projectmeta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +3
-0+that it is working with. + +So in order to have a reproducible repo installation within theproject
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +4
-0+folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko+--- + repo | 2 +-+ 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +16
-0+ +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes- renamed from meta/recipes-devtools/repo/repo.inc +0
-0devtools/repo/repo.inc
new file mode 100644
index 0000000000..60b32e4d74meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helpsmanage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main"
+MIRRORS = "git://[email protected]/git-repo.gitgit://github.com/GerritCodeReview/git-repo.git\n"meta/recipes-devtools/repo/repo.inc new file +13
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM ="file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
Quoted 11 lines
Quoted 11 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#12893):
https://lists.openembedded.org/g/bitbake-devel/message/12893
Mute This Topic: https://lists.openembedded.org/mt/86840271/3647476
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]
]
-=-=-=-=-=-=-=-=-=-=-=-
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFNegACgkQYgqew07V MNVw6Af/WN3h3P19RWdnE7gncdbOgMbS/FBX2viw8HucAfjXovlTQyUzCyN5bROG CyeSwOp5Fvblyg52CzPqkVuXvAoj01mqtHWhLdkzqqB+MUDR1p3Y0qPFJo7BBspP vN8pY4S5dEW3np6zg+9nSkH5nrrZ6AjU+wZipMEGeaGLuEOGvCh5pwDK8d9VzAAu mXq5i8f+ZTr75LNnOpmWz0LgkN32PUCD0VS42/aQFloxvTkssdVKupyCwjUHttxN Fn2J+BNIXbMkJLFiMY6dYaqJKai25l8YH6l8bMVWQLPpJg1d2DpPf1FuGGG7aidh /KNnb8B/tYMF9MNrKXduBayNIj01Vg==
=THWb
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Actually, I don't believe this to be an issue for the following reasons:
1. The recipe does nothing more than taking the wrapper script as is and copying it to the sysroot. As the repo binary is never called when building, it does not have the "chance" to look for an update during the yocto offline build process. The outcome of an offline build is therefore predictable.
2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the do_fetch step, which obviously has no issues with any network access.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 8 lines · Jasper Orschulko
Hi Konrad,
that might be a good point. I patched the
REPO_REVto a fixed refspec within the wrapper tool, as to stop repo from using the latest stable release. But you are right, and this might not be enough. Will do some
more digging on this.
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFOvsACgkQYgqew07V MNVfxQf/VZLnjKv7To1VaqM6dMIX/R3P9cWQtuTxOg6G05Ziooizao5VgQDUXEIy Zs3h5lCFMuXdJlEZc9q2AJQC55BIGqtW8Njo6lNQDPI0Iz51HpXhNgsWdoC3E7qB tphfw+H0WNnIhtDMkK/+hUAZWAsHHUb7I3rAyruE34GKqvEmbBwHo45B0NpgeCqF rNPH7FeI9VvehloZB/5+jnJbrnmyWWogXCNx3PXPlgyrnxyRDuibOUivW6R/MRGd RkSdYvllMgS9ajBLyK/PEyeA6rupEVQquKv0n9ghxZDOzVm6mboCvBMcjxsaSCpR UlFP+h+4sYw0DTfAN9J4UIGWq4VmJA==
=P/xG
-----END PGP SIGNATURE-----
Quoted 9 lines · Jasper Orschulko
Actually, I don't believe this to be an issue for the following reasons:
1. The recipe does nothing more than taking the wrapper script as is and copying it to the sysroot. As the repo binary is never called when building, it does not have the "chance" to look for an update during the yocto offline build process. The outcome of an offline build is therefore predictable.
Quoted 5 lines
2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the
do_fetchstep, which obviously has no issues with any network access.
Okay fine that makes sense, even if it is still phoning home.
Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a SHA revision, as I wouldn't bet my life on google not deleting/removing tags from their repos - and unpleasantly we wouldn't even notice that easily
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Quoted 1 line
Okay fine that makes sense, even if it is still phoning home.
True. Bad software design at best. But luckily not an issue from a build system perspective. :)
Quoted 6 lines
Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a
SHA revision, as I wouldn't bet my life on google not deleting/removing
tags from their repos - and unpleasantly we wouldn't even notice that easily
Fair point, will do!
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 18 lines · Konrad Weihmann
Quoted 10 lines · Jasper Orschulko
Actually, I don't believe this to be an issue for the following reasons:
1. The recipe does nothing more than taking the wrapper script as is and copying it to the sysroot. As the repo binary is never called when
building, it does not have the "chance" to look for an update during the yocto offline build process. The outcome of an offline build is therefore predictable.Quoted 5 lines
2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the
do_fetchstep,
which obviously has no issues with any network access.
Quoted 7 lines
>
Okay fine that makes sense, even if it is still phoning home. Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a SHA revision, as I wouldn't bet my life on google not deleting/removing tags from their repos - and unpleasantly we wouldn't even notice that easily
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFRU4ACgkQYgqew07V MNX3ywf+MQAwmH9TVF0izfVVIYhwiTASgZLlG0pwMBXfQw6iIeWAXmO8LccJNrwb fceVMFFXPi1/2SjAdjhOzzh5X7uSGbcVT0G13RpeuYfsnu7HyJ3n6TAdzoXiQtNa PqJqPVsZBEIpHilTYEZqPu/ENHNq74Sj/sQXqM0XjNMTaJKKW7Syc9jbGdEdn+MV gU9iNIQmYWEPSUS2Tp+lgFh9xC3qQOQJZPpcJ5V3hWfAIibaRMed4kGdd3mlJsYj /Aoq+7gXe1iZOXB7OxXX+bk7ueoHCnp7QUPyWFVxslKx/r/faH+f+PwHmJHZp0VD gOrzbjaIK2E2CVnRobaOkSknIffJzQ==
=Tfz+
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
SHA rev set in v2 of the patch series :)
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 22 lines · Konrad Weihmann
Quoted 12 lines · Jasper Orschulko
Actually, I don't believe this to be an issue for the following reasons:
1. The recipe does nothing more than taking the wrapper script as is
and copying it to the sysroot. As the repo binary is never called when
building, it does not have the "chance" to look for an update during
the yocto offline build process. The outcome of an offline build is therefore predictable.Quoted 5 lines
2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the
do_fetchstep,
which obviously has no issues with any network access.>
Quoted 7 lines
Okay fine that makes sense, even if it is still phoning home. Another thing, could you please exchange "+REPO_REV = 'v2.17.3'" for a
SHA revision, as I wouldn't bet my life on google not deleting/removing
tags from their repos - and unpleasantly we wouldn't even notice that easily
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFTxsACgkQYgqew07V MNW2ugf/RS4zCitkPe9ZEDJvApdZGie1dJmc3gGvcweNv6d6kd1bFd2bS4BYdbiK BOhHF3kMtdznaJNyQlAHBSmysfvTUWymqUahRT3f7+kEuHqrwQcYJFoDqM3tXR4s 4VKYpbRo8klw62XjuBHdwJ8LxJiz5EWQdQF8gdQTl3Tiojp6y2KGDrjo3j/MYjAk 6KfwP0bAca6o/vMF7Jjhbs9sIlCqOkUXrNRbktEonO7hPZxdpgiEC6QPHCq9ONiN 3hHPhV13gd+BTa/7k1GNPSLW4XV6ye7qc26CpqMDing8dxz4RJnzpk2IxTNFHQ0m 95mLdHAygo7o9RJYdz8NIO4hXVqb7Q==
=nQVK
-----END PGP SIGNATURE-----
Quoted 8 lines · Jasper Orschulko
Actually, I don't believe this to be an issue for the following reasons:
[...]
2. When using repo within the yocto build itself (with the repo fetcher), the repo binary is only ever called during the
do_fetchstep, which obviously has no issues with any network access.
This is not true. The system may be configured for no network access and is meant just to be building from MIRRORS/PREMIRRORS (see also BB_NO_NETWORK).
One of the reasons for the fetcher abstraction is to allow us to support features like that. do_fetch is where network access should happen, if the system is configured for it.
I guess the question becomes whether the usages of repo when using DL_DIR as a cache will still touch the network?
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard,
I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the write-up). Martin, please corrent me if I'm missing something:
Quoted 9 lines
a) network access for sources is only expected to happen in the
do_fetchstep.
This is not enforced or tested but is required so that we can:i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways
v) allow rebuilding of the software in X years time
check
Quoted 1 line
b) network access is not expected in
do_unpack
check
Quoted 1 line
c) you can take
DL_DIRand use it as a mirror for offline builds
check
Quoted 3 lines
d) access to the network is only made when explicitly configured in recipes
(e.g. use of AUTOREV, or use of git tags which change revision)
check
Quoted 4 lines
e) fetcher output is deterministic
(i.e. if you fetch configuration XXX now it will match in future exactly in
a clean build with a newDL_DIR)
check. When a fixed refspec is set within the recipe, the fetcher will check, if all repositories in the repo manifest are set to a fixed refspec as well and otherwise throw an error.
Quoted 3 lines
f) network access is expected to work with the standard linux proxy variables
environment but only in thedo_fetchtasks)
this should work, as we keep the GIT_PROXY_COMMAND environment and run repo via runfetchcmd(). Not further tested though.
Quoted 3 lines
g) access during parsing has to be minimal, a "git ls-remote" for an AUTOREV
git recipe might be ok but you can't expect to checkout a git tree
unfortunately, do to the nature of repo, we need to clone the manifest repo, so that we can run "git ls-remote" on the referenced git repos and therefore ensure that
Quoted 3 lines
h) we need to provide revision information during parsing such that a version
for the recipe can be constructed.
check. We create a hash value from git ls-remote of the recipe file, as well as the git ls-remote of each repo referenced in the manifest.
Quoted 4 lines
i) versions are expected to be able to increase in a way which sorts allowing
package feeds to operate (see PR server required for git revisions to sort)
check. We use the PR server and didn't notice any issues.
Quoted 5 lines
j) API to query for possible version upgrades of a url is highly desireable to
allow out automated upgrage code to function (it is implied this does always
have network access)
check (if I understand you correctly). We support AUTOINC.
Quoted 9 lines
k) Where fixes or changes to behaviour in the fetcher are made, we ask that
test cases are added (run with "bitbake-selftest bb.tests.fetch"). We do
have fairly extensive test coverage of the fetcher as it is the only way
to track all of it's corner cases, it still doesn't give entire coverage
though sadly.
We are currently still missing any tests for the new fetcher. We will add them in the course of the next days. Meanwhile, I have prepared a demo environment so if anymeone is interested in doing some further testing, feel free: https://github.com/Jasper-Ben/demo-kas :)
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 1 line
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGKWtQACgkQYgqew07V MNU3Lgf+PlDNyHxoCive3HL8V00KlfrsjTcuQVYiODav39n/h/cRvq8F8Wh7NiyM Hjrr9O7MCOZLNGQiW9vBN/R6FbWESZ+sxl2nLdGH98eCYPKBurcHzvIXlJgincc+ dL1HtxDqjdk9TOHfaKAD2zc8U0tY6xu6Hgj7QIAqELt8HAwOQeIherAuIQw7g+KH eWY5xhn8+KrY//RlnnRmDMO20LBY68bdkCOCH3kAIBfcSIzj8fzf1A4PZhGFi8AM dydVCBMrKvPg3bfJnenZrVPijWnb82pOwjWzoZltzYxtCjqAhwevoCr8twGNDbx4 IVPcEqtMWRrf8rpQiHsZFTheap8BdA==
=hdir
-----END PGP SIGNATURE-----
Quoted 10 lines · Jasper Orschulko
Quoted 6 lines
e) fetcher output is deterministic
(i.e. if you fetch configuration XXX now it will match in future exactly in
a clean build with a newDL_DIR)check. When a fixed refspec is set within the recipe, the fetcher will check, if all repositories in the repo manifest are set to a fixed refspec as well and otherwise throw an error.
When you say "fixed refspec", will that be a definitive sha revision or a tag? We always force resolution of tags as they tend to cause problems and can change even if it is bad form.
Quoted 6 lines
Quoted 3 lines
f) network access is expected to work with the standard linux proxy variables
environment but only in thedo_fetchtasks)this should work, as we keep the
GIT_PROXY_COMMANDenvironment and run repo viarunfetchcmd(). Not further tested though.
If you're using runfetchcmd that should work.
Quoted 8 lines
Quoted 4 lines
g) access during parsing has to be minimal, a "git ls-remote" for an AUTOREV
git recipe might be ok but you can't expect to checkout a git treeunfortunately, do to the nature of repo, we need to clone the manifest repo, so that we can run "git ls-remote" on the referenced git repos and therefore ensure that
This is potentially a big issue. Cloning operations during parsing is pretty horrible. We'd not expect any thing being written out like that during a parse. It would probably work "ok" for one recipe but if you start getting the hundreds of git recipes we have in some layers, it wouldn't scale if we allowed that :(.
Not sure what to recommend here but it is definitely problematic.
Quoted 1 line
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard,
Quoted 5 lines
When you say "fixed refspec", will that be a definitive sha revision or a tag?
We always force resolution of tags as they tend to cause problems and can change
even if it is bad form.
that's a good point. Actually, Martin and I have just been discussing this, as we noticed that this point actually got "lost" during our implementation. We are currently working on fixing this. Good to know how you handle this. I will keep you posted.
Quoted 10 lines
This is potentially a big issue. Cloning operations during parsing is pretty
horrible. We'd not expect any thing being written out like that during a parse.
It would probably work "ok" for one recipe but if you start getting the hundreds
of git recipes we have in some layers, it wouldn't scale if we allowed that :(.Not sure what to recommend here but it is definitely problematic.
Just to make sure that we are on the same page: This ONLY affects recipes which use the repo fetcher. And it ONLY clones the repository containing the repo manifest (which tend to be small in size). So unless developers start using hundreds of repo-based recipes, which I find a very unlikely scenario, this should not be an issue.
Unfortunately, I don't see any other way to access the repo manifest file, as we need to calculate the commit hashes of the git repos referenced in the repo manifest file. Otherwise, it is impossible for us to determinate the necessity of an update when SRCREV = "${AUTOREV}".
However, I see one potential improvement here. Currently the cloning of the manifest repo is done on a per-recipe basis. E.g. this means if we have 10 recipes inheriting a bbclass containing a repo fetcher, we will clone 10 identical manifest repos. We'll work on improving this.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 52 lines · Richard Purdie
Quoted 11 lines · Jasper Orschulko
Quoted 6 lines
e) fetcher output is deterministic
(i.e. if you fetch configuration XXX now it will match in future exactly in
a clean build with a newDL_DIR)check. When a fixed refspec is set within the recipe, the fetcher will
check, if all repositories in the repo manifest are set to a fixed refspec as well and otherwise throw an error.When you say "fixed refspec", will that be a definitive sha revision or a tag?
We always force resolution of tags as they tend to cause problems and can change
even if it is bad form.Quoted 8 lines
Quoted 4 lines
f) network access is expected to work with the standard linux proxy
variables
environment but only in thedo_fetchtasks)this should work, as we keep the
GIT_PROXY_COMMANDenvironment and run
repo viarunfetchcmd(). Not further tested though.If you're using runfetchcmd that should work.
Quoted 7 lines
Quoted 6 lines
g) access during parsing has to be minimal, a "git ls-remote" for anin
AUTOREV
git recipe might be ok but you can't expect to checkout a git treeQuoted 5 lines
unfortunately, do to the nature of repo, we need to clone the manifest
repo, so that we can run "git ls-remote" on the referenced git repos
and therefore ensure thatThis is potentially a big issue. Cloning operations during parsing is pretty
horrible. We'd not expect any thing being written out like that during a parse.
It would probably work "ok" for one recipe but if you start getting the hundreds
of git recipes we have in some layers, it wouldn't scale if we allowed that :(.
Quoted 8 lines
Not sure what to recommend here but it is definitely problematic.
Quoted 1 line
Cheers,
Richard
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGLzpwACgkQYgqew07V MNV3XQf9FDzIq3yihsO5FCEn/QOm7v48VuuOZF/85K6dxRgexCfdHHxkn7zJ0luE ZxgPDmcXM83HcFh6B5XKis88/vnkU2R+YITgWe9+81l1foQryKTP9u7E2giIHW/F JpGzxTtTb5F3N0+xjmqnyR7OYEB3TqJ1VFsaLlYdYs/sWaDYbt/9AEtcD39ynCr5 dEEqEgiIk05X03kiNnyUd2jDpy0bAbihqJu7OPzU4zSvNn/+zXRM0CMKDemyONb1 u2lINROQpk98qaVzBTX+uOskQTkMrkRRuuncUY0ggq6pHGz3TRubv15mePYIeLlJ y2lBBB6O8f/iPesjbvKFa85EeNhdAg==
=4Ewh
-----END PGP SIGNATURE-----
Quoted 11 lines · Jasper Orschulko
Hi Richard,
Quoted 4 lines
When you say "fixed refspec", will that be a definitive sha revision or a tag?
We always force resolution of tags as they tend to cause problems and can change even if it is bad form.that's a good point. Actually, Martin and I have just been discussing this, as we noticed that this point actually got "lost" during our implementation. We are currently working on fixing this. Good to know how you handle this. I will keep you posted.
Ok, it is good to be clear on that one. I know the fact we hit the network for tags does concern some but it really is the only way to handle them.
Quoted 9 lines
Quoted 9 lines
This is potentially a big issue. Cloning operations during parsing is pretty
horrible. We'd not expect any thing being written out like that during a parse.
It would probably work "ok" for one recipe but if you start getting the hundreds
of git recipes we have in some layers, it wouldn't scale if we allowed that :(.
Quoted 5 lines
Quoted 1 line
Not sure what to recommend here but it is definitely problematic.
Just to make sure that we are on the same page: This ONLY affects recipes which use the repo fetcher. And it ONLY clones the repository containing the repo manifest (which tend to be small in size).
Correct, we are on the same page. This is still quite problematic as the recipes are meant to parse quickly and a repository clone is definitely not expected.
Quoted 2 lines
So unless developers start using hundreds of repo-based recipes, which I find a very unlikely scenario, this should not be an issue.
Even ten recipes using this will show a degradation in parsing speed and I do get a lot of complaints when parsing slows down for any reason. The user doesn't expect this and it won't be visible what bitbake is doing (sitting at 99% parsed for a period).
Also, the "tend to be small" implies someone will create a huge one at some point even if that is a bad idea for whatever reasons, I just know how these things end up going :(.
Quoted 5 lines
Unfortunately, I don't see any other way to access the repo manifest file, as we need to calculate the commit hashes of the git repos referenced in the repo manifest file. Otherwise, it is impossible for us to determinate the necessity of an update when SRCREV = "${AUTOREV}".
Some further questions:
* Does it only clone a repo in the AUTOREV case?
* Could it only obtain the manifest file somehow without a clone of the repo?
Quoted 4 lines
However, I see one potential improvement here. Currently the cloning of the manifest repo is done on a per-recipe basis. E.g. this means if we have 10 recipes inheriting a bbclass containing a repo fetcher, we will clone 10 identical manifest repos. We'll work on improving this.
At least for wget or git, it is assumed that for a given url, there would be one tarball/clone and that there is locking in place to share it between them. This means you'll see do_fetch tasks for binutils, binutils-cross-XXX, nativesdk- binutils and binutils-native and one will block the others but the fetch will happen once and be shared between them. I guess with repo it may not be as simple as that but we should try and share what we can if possible.
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard,
Quoted 8 lines
Even ten recipes using this will show a degradation in parsing speed and I do
get a lot of complaints when parsing slows down for any reason. The user doesn't
expect this and it won't be visible what bitbake is doing (sitting at 99% parsed
for a period).
Quoted 5 lines
Also, the "tend to be small" implies someone will create a huge one at some
point even if that is a bad idea for whatever reasons, I just know how these
things end up going :(.
I believe this is something that could be addressed in the documentation of the repo fetcher? Something along the lines of "Using the repo fetcher can increase the time spent in the parsing phase, as we need to clone and inspect the manifest repository. To avoid this, keep the size of your manifest repository at a minimum."
When this is done, the dalay is barely noticable. We have a testsetup with 24 recipes and reasonable parsing speeds.
Quoted 1 line
* Does it only clone a repo in the AUTOREV case?
No, it also clones with fixed refspecs, as we want to ensure that each of the git repos referenced in the manifest also uses fixed refspecs. Otherwise, the sources in the background might change, while the recipe still (correctly) references the same revision of the manifest repo.
Quoted 2 lines
* Could it only obtain the manifest file somehow without a clone of the repo?
Unfortunately not really. git does not offer a way to list the content of a file without cloning it beforehand.
The only other way I could imagine this to work would require us to keep the manifest file within the meta layer and do a somewhat "hacky" workaround:
It is possible to run repo init on a local git folder. As such, something like this could work:
1) create temp folder (e.g. /tmp/repo-XXXXXXXXX)
2) run git init in temp folder
3) install manifest file into temp folder
4) add & commit
5) run repo init -u file://$temp-folder
The repo manifest could even still be used outside of yocto, by using the metalayer repo as basis for the repo init: repo init -u $metalayer-repo -m $path-to-manifest-file. Not pretty, but it works.
So if cloning within the parser is a dealbreaker for you, this option might be worth looking into.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 96 lines · Richard Purdie
Quoted 9 lines · Jasper Orschulko
Hi Richard,
Quoted 6 lines
When you say "fixed refspec", will that be a definitive sha revision
or a tag?
We always force resolution of tags as they tend to cause problems and
can change even if it is bad form.Quoted 6 lines
that's a good point. Actually, Martin and I have just been discussing
this, as we noticed that this point actually got "lost" during our implementation. We are currently working on fixing this. Good to know
how you handle this. I will keep you posted.Ok, it is good to be clear on that one. I know the fact we hit the network for
tags does concern some but it really is the only way to handle them.Quoted 11 lines
Quoted 11 lines
This is potentially a big issue. Cloning operations during parsing is
pretty
horrible. We'd not expect any thing being written out like that during a parse.
It would probably work "ok" for one recipe but if you start getting
the hundreds
of git recipes we have in some layers, it wouldn't scale if we allowed that :(.Quoted 6 lines
Quoted 1 line
Not sure what to recommend here but it is definitely problematic.
Just to make sure that we are on the same page: This ONLY affects recipes which use the repo fetcher. And it ONLY clones the repository
containing the repo manifest (which tend to be small in size).Correct, we are on the same page. This is still quite problematic as the recipes
are meant to parse quickly and a repository clone is definitely not expected.Quoted 3 lines
So unless developers start using hundreds of repo-based recipes, which I
find a very unlikely scenario, this should not be an issue.Even ten recipes using this will show a degradation in parsing speed and I do
get a lot of complaints when parsing slows down for any reason. The user doesn't
expect this and it won't be visible what bitbake is doing (sitting at 99% parsed
for a period).Also, the "tend to be small" implies someone will create a huge one at some
point even if that is a bad idea for whatever reasons, I just know how these
things end up going :(.Quoted 7 lines
Unfortunately, I don't see any other way to access the repo manifest
file, as we need to calculate the commit hashes of the git repos referenced in the repo manifest file. Otherwise, it is impossible for
us to determinate the necessity of an update when SRCREV = "${AUTOREV}".Some further questions:
* Does it only clone a repo in the AUTOREV case?
* Could it only obtain the manifest file somehow without a clone of the repo?Quoted 7 lines
However, I see one potential improvement here. Currently the cloning of
the manifest repo is done on a per-recipe basis. E.g. this means if we
have 10 recipes inheriting a bbclass containing a repo fetcher, we will
clone 10 identical manifest repos. We'll work on improving this.At least for wget or git, it is assumed that for a given url, there would be one
tarball/clone and that there is locking in place to share it between them. This
means you'll seedo_fetchtasks for binutils, binutils-cross-XXX, nativesdk-
binutils and binutils-native and one will block the others but the fetch will
happen once and be shared between them. I guess with repo it may not be as
simple as that but we should try and share what we can if possible.
Quoted 5 lines
Cheers,
Richard
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNAZwACgkQYgqew07V MNUYpAf/foJRPcTJZTmDixuz5ZmI/4x9RB47XWupJrwy//1hHNJDPyc3iHqj79te 6T8EuqIYUtYGIE2FHzkbVwAiyNQYCODhh3Uc/LmNoo5/59XypLNVgf+UH7KowytX 1YptbB477Dl/hR39ul7oDxdC9D+j0gHjClzs8KmHt0iMzzc1nLvih9G8yXkYNYVH +0ovvlnpjHsC+WFYackEJe8CphUTFEn/EgffYtr/vl1SefRCzqs8aFB/gnhXKMjU 3zXH3DoeUZBdNahyEP9lAlbceM1AVoh9kNCXcVp/lUu1m7M3MVyxREMoeVfo22up J1kZetA1LqX02PwDacU4Fftoc502Dw==
=BQ5W
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard and folks,
Quoted 5 lines
The only other way I could imagine this to work would require us to keep the manifest file within the meta layer and do a somewhat "hacky"
workaround:
Quoted 8 lines
It is possible to run
repo initon a local git folder. As such, something like this could work:1) create temp folder (e.g.
/tmp/repo-XXXXXXXXX)
2) run git init in temp folder
3) install manifest file into temp folder
4) add & commit
5) run repo init -u file://$temp-folder
today I discovered that repo also supports the --standalone-manifest flag, which makes the workaround described above obsolete. Thus, I believe supporting repo manifests from the layer repo only is the cleanest and easiest way to implement this and we wouldn't have to rely on cloning manifest repos during recipe parsing.
However, this would be a breaking change to the current repo fetcher implementation, as it supports remote manifest repos.
I am pretty confident however that no one uses the arguably already broken and unusable current repo fetcher implementation in a production workload so this shouldn't be a major issue.
What are your thoughts on this?
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 58 lines · Jasper Orschulko
Hi Richard,
Quoted 10 lines
Even ten recipes using this will show a degradation in parsing speed
and I do
get a lot of complaints when parsing slows down for any reason. The user doesn't
expect this and it won't be visible what bitbake is doing (sitting at
99% parsed
for a period).Quoted 5 lines
Also, the "tend to be small" implies someone will create a huge one at some
point even if that is a bad idea for whatever reasons, I just know how these
things end up going :(.I believe this is something that could be addressed in the documentation of the repo fetcher? Something along the lines of "Using
the repo fetcher can increase the time spent in the parsing phase, as we need to clone and inspect the manifest repository. To avoid this, keep the size of your manifest repository at a minimum."When this is done, the dalay is barely noticable. We have a testsetup with 24 recipes and reasonable parsing speeds.
Quoted 1 line
* Does it only clone a repo in the AUTOREV case?
No, it also clones with fixed refspecs, as we want to ensure that each
of the git repos referenced in the manifest also uses fixed refspecs. Otherwise, the sources in the background might change, while the recipe
still (correctly) references the same revision of the manifest repo.Quoted 2 lines
* Could it only obtain the manifest file somehow without a clone of the repo?
Unfortunately not really. git does not offer a way to list the content
of a file without cloning it beforehand.The only other way I could imagine this to work would require us to keep the manifest file within the meta layer and do a somewhat "hacky"
workaround:It is possible to run
repo initon a local git folder. As such, something like this could work:1) create temp folder (e.g.
/tmp/repo-XXXXXXXXX)
2) run git init in temp folder
3) install manifest file into temp folder
4) add & commit
5) run repo init -u file://$temp-folder
Quoted 7 lines
The repo manifest could even still be used outside of yocto, by using the metalayer repo as basis for the repo init:
repo init -u $metalayer-repo -m $path-to-manifest-file. Not pretty, but it works.So if cloning within the parser is a dealbreaker for you, this option might be worth looking into.
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGeYnwACgkQYgqew07V MNWXFAgAkZ7+CRZadUItM+Ay8uePffysuQhlwxHaAgzx1H+4Pk/HKK9QpEy3jFWz kODr3DPOWlf6eS+AcqpJXVYdMsTucvE34eLIWb8reFkrPQKMtxp0N6U5gqz7e+Bm Wf4lS04tqGZzC/Ld2DOs/zEnGKEwXBRHMX/9j6YvVMYxwKjxghW2LRloWpzghduH j0FTAp1aGKLhCQjqeEuNZRMT+rn0i8k6u8Lxel1wRHi2LNhyT3VD3NThjVL64acH FF83jhaxQ0JxpsSn9ouvJD3wMJHi72rdfGzznuQHmFT0lOxO5AjlcIOtK3KisiAI KQ/jEV/44WN0JO7OpmHrVyOSfPeVBA==
=H0FZ
-----END PGP SIGNATURE-----
Quoted 30 lines
-----Original Message-----
From: [email protected] <bitbake- [email protected]> On Behalf Of Jasper Orschulko
Sent: den 9 november 2021 12:26
To: [email protected]; openembedded- [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; bitbake- [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Richard,
I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the write-up). Martin, please corrent me if I'm missing something:
Quoted 9 lines
a) network access for sources is only expected to happen in the
do_fetchstep.
This is not enforced or tested but is required so that we can:i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways
v) allow rebuilding of the software in X years time
Quoted 5 lines
check
Quoted 1 line
b) network access is not expected in
do_unpackcheck
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
//Peter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Peter,
Quoted 3 lines
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
we don't. This recipe only installs the repo wrapper. The source code is downloaded and installed when running repo init.
However, in our opinion this is not an issue. When installing repo as a package to a target, this is the expected behaviour. The target would only have the repo wrapper installed, just like on any other linux distrobution. The actual usage of repo on the target is decoupled from the bitbake build process.
When using the repo fetcher, the repo source is fetched during the do_fetch stage by running repo init (when SRCREV = AUTOREV, changes to the recipe or no previous sources available in DL_DIR). By executing this within the "runfetchcmd" function, this also works with the usual network features bitbake provides, e.g. proxy.
If the recipe has not changed and sources are already available from a previous run, repo will not be rerun. As such, reproducing a build offline is also possible.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 36 lines · Peter Kjellerstedt
Quoted 36 lines
-----Original Message-----
From: [email protected] <bitbake- [email protected]> On Behalf Of Jasper Orschulko
Sent: den 9 november 2021 12:26
To: [email protected]; openembedded- [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; bitbake- [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Richard,
I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the write-up). Martin, please corrent me if I'm missing something:
Quoted 9 lines
a) network access for sources is only expected to happen in the
do_fetchstep.
This is not enforced or tested but is required so that we can:i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways
v) allow rebuilding of the software in X years timecheck
Quoted 1 line
b) network access is not expected in
do_unpack
Quoted 8 lines
Quoted 1 line
check
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
//Peter
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5 fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
=AOyp
-----END PGP SIGNATURE-----
Quoted 20 lines
-----Original Message-----
From: Jasper Orschulko <[email protected]>
Sent: den 11 november 2021 11:05
To: [email protected]; openembedded- [email protected]; [email protected]; Peter Kjellerstedt <[email protected]>; [email protected]
Cc: [email protected]; Daniel Baumgart <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Peter,
Quoted 3 lines
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
Quoted 8 lines
we don't. This recipe only installs the repo wrapper. The source code is downloaded and installed when running
repo init.However, in our opinion this is not an issue. When installing repo as a package to a target, this is the expected behaviour. The target would only have the repo wrapper installed, just like on any other linux distrobution. The actual usage of repo on the target is decoupled from the bitbake build process.
It might be how repo is designed, but it will still break the bitbake expectations. I.e., if I have an environment with the layers available, a populated DL_DIR and BB_NO_NETWORK = "1", and then disconnect the build host from the Internet, I should still be able to source oe-init-build-env for a new machine and build it. However, this means that only the source for the repo wrapper is available in DL_DIR. So when the repo wrapper executes it will try to go on the Internet to fetch the rest of the repo source, which will fail.
Quoted 87 lines
When using the repo fetcher, the repo source is fetched during the
do_fetchstage by runningrepo init(when SRCREV = AUTOREV, changes to the recipe or no previous sources available inDL_DIR). By executing this within the "runfetchcmd" function, this also works with the usual network features bitbake provides, e.g. proxy.
If the recipe has not changed and sources are already available from a previous run, repo will not be rerun. As such, reproducing a build offline is also possible.- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 BerlinQuoted 36 lines · Peter Kjellerstedt
Quoted 36 lines
-----Original Message-----
From: [email protected] <bitbake- [email protected]> On Behalf Of Jasper Orschulko
Sent: den 9 november 2021 12:26
To: [email protected]; openembedded- [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; bitbake- [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Richard,
I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the write-up). Martin, please corrent me if I'm missing something:
Quoted 9 lines
a) network access for sources is only expected to happen in the
do_fetchstep.
This is not enforced or tested but is required so that we can:i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways
v) allow rebuilding of the software in X years timecheck
Quoted 1 line
b) network access is not expected in
do_unpackQuoted 8 lines
Quoted 1 line
check
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
//Peter
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5 fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
=AOyp
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Peter,
Quoted 4 lines
However, this means
that only the source for the repo wrapper is available inDL_DIR. So when the repo wrapper executes it will try to go on the Internet to fetch the rest of the repo source, which will fail.
just to clearify: Are we talking about installing repo on a target, or are we talking about the repo fetcher?
When installing repo on the target, we do not call any "repo" command within the bitbake build process. We just clone the git repo containing the wrapper script and install the wrapper script into the sysroot. At that point the bitbake build process is finished and the user ends up with just the wrapper script on his device. Just as he would when installing repo from the system repositories on debian, arch, etc.
When using the fetcher, the repo sources are fetched during the initial do_fetch (this also works with bitbake --runonly=do_fetch) and placed into the DL_DIR (in the .repo folder, next to the sources as described in the repo manifest). So unless the user changes something fetcher related in the recipe or deletes his DL_DIR, he can use this build environment for an offline build.
You can try this for yourself, if you like:
1) clone https://github.com/Jasper-Ben/demo-kas
2) (with docker and docker-compose installed) run make offline-build
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 34 lines · Peter Kjellerstedt
Quoted 26 lines
-----Original Message-----
From: Jasper Orschulko <[email protected]>
Sent: den 11 november 2021 11:05
To: [email protected]; openembedded- [email protected]; [email protected]; Peter Kjellerstedt
<[email protected]>; [email protected]
Cc: [email protected]; Daniel Baumgart <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Peter,
Quoted 3 lines
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
we don't. This recipe only installs the repo wrapper. The source code is downloaded and installed when running
repo init.Quoted 7 lines
However, in our opinion this is not an issue. When installing repo as a
package to a target, this is the expected behaviour. The target would only have the repo wrapper installed, just like on any other linux distrobution. The actual usage of repo on the target is decoupled from
the bitbake build process.
Quoted 101 lines
It might be how repo is designed, but it will still break the bitbake expectations. I.e., if I have an environment with the layers available, a populated
DL_DIRandBB_NO_NETWORK= "1", and then disconnect the build host from the Internet, I should still be able to source oe-init-build-env for a new machine and build it. However, this means that only the source for the repo wrapper is available inDL_DIR. So when the repo wrapper executes it will try to go on the Internet to fetch the rest of the repo source, which will fail.Quoted 92 lines
When using the repo fetcher, the repo source is fetched during the
do_fetchstage by runningrepo init(when SRCREV = AUTOREV, changes to the recipe or no previous sources available inDL_DIR). By executing
this within the "runfetchcmd" function, this also works with the usual
network features bitbake provides, e.g. proxy.
If the recipe has not changed and sources are already available from a
previous run, repo will not be rerun. As such, reproducing a build offline is also possible.- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 BerlinQuoted 38 lines · Peter Kjellerstedt
Quoted 38 lines
-----Original Message-----
From: [email protected] <bitbake- [email protected]> On Behalf Of Jasper Orschulko
Sent: den 9 november 2021 12:26
To: [email protected]; openembedded- [email protected]; [email protected]; [email protected]
Cc: [email protected]; [email protected]; bitbake- [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools:
Initial
recipe
for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Richard,
I think our implementation of the repo fetcher checks the (what I believe to be) most relevant parts of your checklist (thanks for the
write-up). Martin, please corrent me if I'm missing something:Quoted 9 lines
a) network access for sources is only expected to happen in the
do_fetchstep.
This is not enforced or tested but is required so that we can:i) audit the sources used (i.e. for license/manifest reasons) ii) support offline builds with a suitable cache iii) allow work to continue even with downtime upstream iv) allow for changes upstream in incompatible ways
v) allow rebuilding of the software in X years timecheck
Quoted 1 line
b) network access is not expected in
do_unpackQuoted 8 lines
Quoted 1 line
check
How do you avoid the repo wrapper fetching the repo source itself and instead fetch it using the bitbake fetcher? I have seen nothing to that extent in the patches so far.
//Peter
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGM6rUACgkQYgqew07V MNUCIAf+MXGp9Hnfxmu+8w3BasVsP2N7ttW2FN3Zroiyr/hKrJzeq/qDQwO+/K3U zWZJ85H6L2eTjOP8fPaHbVMRD1jUoYVpYUAE/fN64FbhCBmurVuWFrLo/u6Gy2cU DCd3SIejXidB25EQRoS4Bfl25il1wG4iMw1pCFA6ku5rGhb8q5jvfZECf0Wuhh7X FnMQlI3VZsSk4CakF3g88AFTLFrsQYcN2vDUskdhMfTZpuRA8duIvW4rVgOvHJQT v07rASNR/9yzPRVkzpgPUI9Vl2LA3vEZ3Rccw3jscYHFwkzj0096DO4+jeDwyza5 fFm0dDT9HjGuzTz66c5JL8sdZZi9pw==
=AOyp
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNCDIACgkQYgqew07V MNXG5ggAlwuv5I3Pi3aG9typiSePABvw0cQT+GbLysr74UTU7UJNDd+ByeM5U7Vc lmFuVO6SKRct0brPy5W7weZxpS7YxgwrFc6QliUpsd4yIZJkPTZVBzdqBxskMU/f z0vHb9lipyXPkg4OU49TGv2k3rBBcrYGMSwG8WiqJL+f2gXHDaCbwb/1oJuI1Ie5 LXW1nkIPViaRuoPgKdlvzptFNI+vBuIRg8KvwSf4W5pfJYC03jZLdRthWj1TNQvz 653Krzyvu3hhxIKgn9YYPb/mI3ZI1NgYm/9hpEYwjtZSLq303uLMB70qYWj6fG/V uHaBFN7b2sF4hbMffJ3jKOEIj/mgSQ==
=PARI
-----END PGP SIGNATURE-----
Quoted 16 lines
-----Original Message-----
From: Jasper Orschulko <[email protected]>
Sent: den 11 november 2021 13:10
To: [email protected]; openembedded- [email protected]; [email protected]; Peter Kjellerstedt <[email protected]>; [email protected]
Cc: [email protected]; Daniel Baumgart <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Peter,
Quoted 7 lines
Quoted 4 lines
However, this means
that only the source for the repo wrapper is available inDL_DIR. So when the repo wrapper executes it will try to go on the Internet to fetch the rest of the repo source, which will fail.just to clearify: Are we talking about installing repo on a target, or are we talking about the repo fetcher?
The repo fetcher, i.e., when repo is used by bitbake.
Quoted 7 lines
When installing repo on the target, we do not call any "repo" command within the bitbake build process. We just clone the git repo containing the wrapper script and install the wrapper script into the sysroot. At that point the bitbake build process is finished and the user ends up with just the wrapper script on his device. Just as he would when installing repo from the system repositories on debian, arch, etc.
Quoted 6 lines
When using the fetcher, the repo sources are fetched during the initial
do_fetch(this also works withbitbake --runonly=do_fetch) and placed into theDL_DIR(in the .repo folder, next to the sources as described in the repo manifest). So unless the user changes something fetcher related in the recipe or deletes hisDL_DIR, he can use this build environment for an offline build.
How can it work? The repo source, which is fetched by the repo wrapper, will not be in DL_DIR or handled by bitbake at all. This means the repo wrapper will try to contact googlesource.com directly and fetch it when it is run by bitbake, which cannot work since the host is disconnected from the network. How or from where would it be able to get the sources it needs?
Quoted 15 lines
You can try this for yourself, if you like:
1) clone https://github.com/Jasper-Ben/demo-kas
2) (with docker and docker-compose installed) runmake offline-build- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
Quoted 7 lines
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
//Peter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Peter,
during this initial fetch it is true, that the repo fetcher has to download the repo sources from googlesource.com. But during this first fetch the fetcher also needs to download any sources listed in the repo manifest anyway, so internet access is a given.
Consequential fetches will only re-run repo if:
a) the SRC_URI or SRCREV has changed within the recipe
b) the SRCREV is set to AUTOREV
Both cases do not apply when reproducing a build offline.
So if you have done this initial fetch of your sources and stash your working dir away, you can do an offline build.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 80 lines · Peter Kjellerstedt
Quoted 26 lines
-----Original Message-----
From: Jasper Orschulko <[email protected]>
Sent: den 11 november 2021 13:10
To: [email protected]; openembedded- [email protected]; [email protected]; Peter Kjellerstedt
<[email protected]>; [email protected]
Cc: [email protected]; Daniel Baumgart <[email protected]>; [email protected]
Subject: Re: [bitbake-devel] [oe-core][PATCH 1/2] devtools: Initial recipe
for repo 2.17.3-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Peter,
Quoted 6 lines
However, this means
that only the source for the repo wrapper is available inDL_DIR. So
when the repo wrapper executes it will try to go on the Internet to
fetch the rest of the repo source, which will fail.Quoted 3 lines
just to clearify: Are we talking about installing repo on a target, or
are we talking about the repo fetcher?The repo fetcher, i.e., when repo is used by bitbake.
Quoted 11 lines
When installing repo on the target, we do not call any "repo" command
within the bitbake build process. We just clone the git repo containing
the wrapper script and install the wrapper script into the sysroot. At
that point the bitbake build process is finished and the user ends up
with just the wrapper script on his device. Just as he would when installing repo from the system repositories on debian, arch, etc.Quoted 9 lines
When using the fetcher, the repo sources are fetched during the initial
do_fetch(this also works withbitbake --runonly=do_fetch) and placed
into theDL_DIR(in the .repo folder, next to the sources as described
in the repo manifest). So unless the user changes something fetcher related in the recipe or deletes hisDL_DIR, he can use this build environment for an offline build.How can it work? The repo source, which is fetched by the repo wrapper,
will not be inDL_DIRor handled by bitbake at all. This means the repo wrapper will try to contact googlesource.com directly and fetch it
when it is run by bitbake, which cannot work since the host is disconnected from the network. How or from where would it be able to get the sources it needs?Quoted 18 lines
You can try this for yourself, if you like:
1) clone https://github.com/Jasper-Ben/demo-kas
2) (with docker and docker-compose installed) runmake offline- build- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]• • • • • • • • • • • • • • • • • • • • • • • • • •
Quoted 8 lines
Quoted 5 lines
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin//Peter
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGNMesACgkQYgqew07V MNU//Af+IFFqmbtV/k3SscMUWYRNDpnSeK6elwfpwnDi84tQHjRo66Xtrzi12D+H zzya3H87unVY3BWqWnQIE3K4aShquz+p9tQA7zuzlARImCyqAsDQzKddD7hhK7AI 4toNobOqNoJZ2D1kJQyu7o7VsnvbiyvYz8Z8l+t4FF35+bwU26aE0iCFF63toZSi GLvmTzLigYNAu9UUW+UgSUAhHVsaCyOolnGwBdHEJW3NliCZEOLDf8omDygG3xDp UOKcX2Q238sGN2fCwAeS2ABC4ffMNDfrpzAS0Q1qy2WVmheWmmtGoaksW2z3sDZk FKxLlRWNQvPWOy/tY2JLyB1QNyedyg==
=c20l
-----END PGP SIGNATURE-----
Quoted 3 lines · Jasper Orschulko
So if you have done this initial fetch of your sources and stash your working dir away, you can do an offline build.
But can you do an offline build without a prepopulated working dir? That's the crucial point: offline build should work in a freshly set up build directory, without sstate cache, and with prepopulated DL_DIR only.
Alex
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Alex,
thanks for your input. You are absolutely correct, this currently does not work. We'll look into this.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 6 lines · Alexander Kanavin
Quoted 3 lines · Jasper Orschulko
So if you have done this initial fetch of your sources and stash your
working dir away, you can do an offline build.
Quoted 6 lines
But can you do an offline build without a prepopulated working dir? That's the crucial point: offline build should work in a freshly set up build directory, without sstate cache, and with prepopulated
DL_DIRonly.Alex
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGOXJwACgkQYgqew07V MNXaxwf+NDOx5TYR4dY4iCsb0IfdfoB01+H7UDZR2s5GZKKA9TyMKbEqBdZrHBdj PTqZJ339IOXBlks1KPAUci0VOXBdwHoybR2XFMGENTnrj6XA+9Yr96ofSPHqQTWr amD/low0cFvfqZClb+md/FcxhnVK45rFhfLJckKElPiYmKjcghs/PXAWbh+Tw6S7 m5KCq8agPqXnaU/bWDmj2eC9t6t+VT/0OnC0tQtohMbYVd8wul4cf5Ic3JilecjI qCH4rK7EBFfzWhs1WTYDGeXX5iWNmZOsdufpXF0URJa5cpjGA5bhjfyXjoUZPzCR WxErD1cnQSNQqysqWlnB5EgM3y+cBg==
=q0zZ
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Alexander,
"[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains a fix for this issue. Reproducing a build offline with only the DL_DIR should work now.
We are currently still lacking the appropriate tests for the fetcher. These will follow soon.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 6 lines · Alexander Kanavin
Quoted 3 lines · Jasper Orschulko
So if you have done this initial fetch of your sources and stash your
working dir away, you can do an offline build.
Quoted 6 lines
But can you do an offline build without a prepopulated working dir? That's the crucial point: offline build should work in a freshly set up build directory, without sstate cache, and with prepopulated
DL_DIRonly.Alex
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS sY39fiky57aEgMe7g1OZX8oRODowPA==
=hpAK
-----END PGP SIGNATURE-----
Thanks, maybe you should write a test for offline builds as well then :)
Alex
Quoted 59 lines · Jasper Orschulko
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Alexander,
"[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains a fix for this issue. Reproducing a build offline with only the
DL_DIRshould work now.We are currently still lacking the appropriate tests for the fetcher. These will follow soon.
- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 BerlinQuoted 6 lines · Alexander Kanavin
Quoted 3 lines · Jasper Orschulko
So if you have done this initial fetch of your sources and stash your
working dir away, you can do an offline build.Quoted 6 lines
But can you do an offline build without a prepopulated working dir? That's the crucial point: offline build should work in a freshly set up build directory, without sstate cache, and with prepopulated
DL_DIRonly.Alex
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS sY39fiky57aEgMe7g1OZX8oRODowPA==
=hpAK
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Will do! :)
This only affects the other patch series though, the recipe itself should be fine now.
It seems the only question left open is this:
Quoted 8 lines
Quoted 8 lines
I think we can replace the patch 0001-Set-REPO_REV-to-v2.17.3.patch with a post function
and with it we can reuse the SRCREV of the recipe. Something like:do_fix_rev(){ sed -i "s/REPO_REV = 'stable'/REPO_REV = '${SRCREV}'" ${S}/repo }
Quoted 5 lines
Quoted 1 line
do_patch[postfuncs] += "
do_fix_rev"Yeah... was thinking about that as well. Wasn't sure though if this would be considered bad practise. What do the others think about this?
Some feedback on this would be appreciated. Thanks!
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 72 lines · Alexander Kanavin
Thanks, maybe you should write a test for offline builds as well then :)
Alex
Quoted 65 lines · Jasper Orschulko
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256Hi Alexander,
"[PATCH v3] fetch2/repo: Implement AUTOREV for repo fetcher" contains
a
fix for this issue. Reproducing a build offline with only theDL_DIR
should work now.We are currently still lacking the appropriate tests for the fetcher.
These will follow soon.- --
With best regardsJasper Orschulko
DevOps EngineerTel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 BerlinQuoted 10 lines · Alexander Kanavin
Quoted 4 lines · Jasper Orschulko
So if you have done this initial fetch of your sources and stash
your
working dir away, you can do an offline build.But can you do an offline build without a prepopulated working dir?
That's the crucial point: offline build should work in a freshlyset
Quoted 4 lines
up build directory, without sstate cache, and with prepopulated
DL_DIRonly.Alex
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSWb4ACgkQYgqew07V MNVUYQgAgA6+8XI+JxtA532FNNIMAZG5ZGZ12GKvVkjoJErnzRf0RZxUG3UyNYmm pSDcfbuodBmBbHxDBpN8VspToKZy8dKcm/jfe9UY7f9U39bZSNXSS3hNPqkqKP5J jCg1zQfxjbW1aVSHJkQB5MoY/EDeoQVv7RI6RmcVsLJZxozPeYaOQWSdTNG/7czg oFjKr6jtSOYO45teMGt3+2+AtAXhlzTgKZo33rX/tknS7f1+kkyNGLv0ssSP5Jcf Bd8YbcdG9lqXKUo0A7rn9Q83YBdpIyA6+By159zzPy7bP8j9aqMH8nNWauahhcFS sY39fiky57aEgMe7g1OZX8oRODowPA==
=hpAK
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGSXNkACgkQYgqew07V MNUKmQf/UYTURPKZ0iY2MKPFB60WdnmsTKJMjotQlcVBUpiwQ6LhSpkfWRqOFLpN na9cNZbYjGFeDNdL3d0ILOEQ+oVhLY7bQpj5Xb2nDsIWspjlHg1IkkvCVGa9zFb3 XALaDrGBrlad9KNhI4C2CmXjdJZgx3yBuuvK6mSiga5fs44QB/lBb9JOQGYucRt+ QeLkIatvFo1Jxzmc6tNkRr3osgacQroagHj4CISh5W+ezWqtdWHA5ndf4HvCNwJV hkpyXtxeQDINGVGBsMnJkOIWRSStOviL66Q1h5/dgbJ23FXQuEkM9CyD19kXWSIW ZSI6as4bzjDqDVPzCPWHDmQdSnJyXQ==
=aNK6
-----END PGP SIGNATURE-----
Wait, what is the use case for the repo fetcher?
Alex
Quoted 153 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>--- .../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 +++++++++++++++++++ meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++ 4 files changed, 86 insertions(+) create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bbmeta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +21
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchb/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchnew file mode 100644
index 0000000000..294a3af53a
---/dev/null
+++b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchmeta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +33
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Python modules +using GPG-signed git tags, and stores those files as part of the project +that it is working with. + +So in order to have a reproducible repo installation within the project +folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage manyGit repositories, does the uploads to revision control systems, and automates parts of the development workflow."
meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main" +MIRRORS = "
git://[email protected]/git-repo.gitgit://
github.com/GerritCodeReview/git-repo.git \n"meta/recipes-devtools/repo/repo.inc new file +13
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-1@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57" --2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#12893):
https://lists.openembedded.org/g/bitbake-devel/message/12893
Mute This Topic: https://lists.openembedded.org/mt/86840271/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Ah I didn't notice it's something that already exists. Still, I'm curious for which items is it used, and why repo over other options?
Alex
Quoted 161 lines · Alexander Kanavin
Wait, what is the use case for the repo fetcher?
Alex
Quoted 154 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>--- .../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 +++++++++++++++++++ meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++ 4 files changed, 86 insertions(+) create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bbmeta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +21
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0diff --git
a/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchb/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchnew file mode 100644
index 0000000000..294a3af53a
---/dev/null
+++b/meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patchmeta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +6
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Pythonmodules
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +27
-0+using GPG-signed git tags, and stores those files as part of the project +that it is working with. + +So in order to have a reproducible repo installation within the project +folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps managemany Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main" +MIRRORS = "
git://[email protected]/git-repo.gitgit://
github.com/GerritCodeReview/git-repo.git \n"meta/recipes-devtools/repo/repo.inc new file +13
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-1@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57" --2.33.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#12893):
https://lists.openembedded.org/g/bitbake-devel/message/12893
Mute This Topic: https://lists.openembedded.org/mt/86840271/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Yeah. Unfortunately the current repo fetcher is pretty much broken, this patch is more or less a requirement for changes to the repo fetcher we proposed in the bitbake mailinglist.
In our case some binary which is made up from some 50ish separately versioned sub-components. We used to have separate recipes for each of this components and static link them using DEPENDS. However, this does not scale well. E.g. if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes. You could move all the components into a single recipe and do 50x git fetcher, but keeping this versioned is still painful. Especially since the developers mainly don't use yocto but use the SDK for cross-compiling. As such this used to mean, that versioning in the development workflow and the release workflow worked differently -> more maintenance work.
With repo fetcher we can streamline the versioning within the yocto build process and the daily developer workflow, as both can fetch the repo manifest from a central stored git repo. The idea is, that the repo manifest defaults to develop branch for all component repos and if we want to create a release we can use repos own tooling to automatically create fixed refspecs from the development manifest. something like repo manifest -r -o release-1.0.0.xml, push that as a release to the manifest repo, set this release as refspec in the recipe within the meta layer and proceed with the release process. So no more manually maintaining the component versioning within the yocto recipes.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 151 lines · Alexander Kanavin
Wait, what is the use case for the repo fetcher?
Alex
Quoted 144 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>---
.../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
+++++++++++++++++++
meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++4 files changed, 86 insertions(+)create mode 100644 meta/recipes-devtools/repo/files/0001-python3- shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001- Set-
REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb(quoted diff) +0
-0shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- shebang.patch
new file mode 100644
index 0000000000..09ccf58264meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +1
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +20
-0+From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0(quoted diff) +0
-0REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 0000000000..294a3af53ameta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +0
-0v2.17.3.patch
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +1
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +5
-0+From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Pythonmodules
+using GPG-signed git tags, and stores those files as part of the projectmeta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +3
-0+that it is working with. + +So in order to have a reproducible repo installation within theproject
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +4
-0+folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko+--- + repo | 2 +-+ 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +16
-0+ +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes- renamed from meta/recipes-devtools/repo/repo.inc +0
-0devtools/repo/repo.inc
new file mode 100644
index 0000000000..60b32e4d74meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helpsmanage
many Git repositories, does the uploads to revision control systems,
and automates parts of the development workflow."meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI ="
git://[email protected]/git-repo.git;protocol=https;branch=main
"
+MIRRORS = "git://[email protected]/git-repo.gitgit://github.com/GerritCodeReview/git-repo.git\n"meta/recipes-devtools/repo/repo.inc new file +13
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3" + +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +0
-0@@ -0,0 +1,7 @@
Quoted 8 lines
Quoted 8 lines
meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-0+# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM ="file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFTK4ACgkQYgqew07V MNXGKwf8CbYnBqQhNvouhixH8rqaEjhFBAcbEvK1Pgke4B0zMuSJVz2YmTKPiHgK dFznVtBxIJS7zwhuj/DyfhrG33Ws3Rw1m8IppZ5FP2XivNbmnimH13a7Pu5hw3FH o1ka6VEqyuR/L4pyVZ+ocSgdEIyReSOJ6E1zyIomz13j5hScQIKFdAu4tinLNLck XnC/SCDe3lZbijo41GK1v9GhKFjwY3Hpko6ezG/CEFPHKZomjHNL9Mi8gbv4Y/50 8+pYnvlYU7agsW4aN/koeRWOm78/Qi6rlOX5pVxAaVVVArrIKnU2aG6UBwk61BtC UIqLBYOMyZ4IXlNp5y2tGbr6QS0YrQ==
=M87Y
-----END PGP SIGNATURE-----
Quoted 13 lines · Jasper Orschulko
In our case some binary which is made up from some 50ish separately versioned sub-components. We used to have separate recipes for each of this components and static link them using DEPENDS. However, this does not scale well. E.g. if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes. You could move all the components into a single recipe and do 50x git fetcher, but keeping this versioned is still painful. Especially since the developers mainly don't use yocto but use the SDK for cross-compiling. As such this used to mean, that versioning in the development workflow and the release workflow worked differently -> more maintenance work.
Quoted 11 lines
With repo fetcher we can streamline the versioning within the yocto build process and the daily developer workflow, as both can fetch the repo manifest from a central stored git repo. The idea is, that the repo manifest defaults to develop branch for all component repos and if we want to create a release we can use repos own tooling to automatically create fixed refspecs from the development manifest. something like
repo manifest -r -o release-1.0.0.xml, push that as a release to the manifest repo, set this release as refspec in the recipe within the meta layer and proceed with the release process. So no more manually maintaining the component versioning within the yocto recipes.
I see. I don't particularly endorse that you invented a custom, proprietary workflow that you have to support entirely by yourselves - instead of finding ways to do what you need with standard yocto tooling, improving the tooling where/if necessary. But if that works out ok for you, then fine :)
Alex
Hi Alex,
Quoted 2 lines
that you invented a custom, proprietary
workflow that you have to support entirely by > yourselves
We are not though. We are integrating an established tool for multi-repository management into yocto. Google repo is not proprietary by the way, it is permissive licensed. It is just a little "special" in it's modus operandi.
Quoted 3 lines
instead of finding ways to do what you need
with standard yocto tooling, improving the
tooling where/if necessary.
But that is exactly what we are doing, by integrating the repo fetcher into the yocto workflow, thus improving the yocto tooling :) Why reinvent the wheel, when you can reuse whats already there? You wouldn't reinvent git just for yocto, would you?
Best regards,
Jasper
Quoted 34 lines · Alexander Kanavin
Quoted 13 lines · Jasper Orschulko
In our case some binary which is made up from some 50ish separately versioned sub-components. We used to have separate recipes for each of this components and static link them using DEPENDS. However, this does not scale well. E.g. if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes. You could move all the components into a single recipe and do 50x git fetcher, but keeping this versioned is still painful. Especially since the developers mainly don't use yocto but use the SDK for cross-compiling. As such this used to mean, that versioning in the development workflow and the release workflow worked differently -> more maintenance work.
Quoted 11 lines
With repo fetcher we can streamline the versioning within the yocto build process and the daily developer workflow, as both can fetch the repo manifest from a central stored git repo. The idea is, that the repo manifest defaults to develop branch for all component repos and if we want to create a release we can use repos own tooling to automatically create fixed refspecs from the development manifest. something like
repo manifest -r -o release-1.0.0.xml, push that as a release to the manifest repo, set this release as refspec in the recipe within the meta layer and proceed with the release process. So no more manually maintaining the component versioning within the yocto recipes.I see. I don't particularly endorse that you invented a custom, proprietary workflow that you have to support entirely by yourselves - instead of finding ways to do what you need with standard yocto tooling, improving the tooling where/if necessary. But if that works out ok for you, then fine :)
Alex
Quoted 6 lines · Jasper Orschulko
But that is exactly what we are doing, by integrating the repo fetcher into the yocto workflow, thus improving the yocto tooling :) Why reinvent the wheel, when you can reuse whats already there? You wouldn't reinvent git just for yocto, would you?
What I mean is that I would try harder to find a workable setup while keeping the recipes for individual items that need to be built. For instance, devtool is capable of updating source revisions in recipes automatically, it may not work exactly as you want, but that can be fixed. Yes, repo itself is not proprietary, but your organizational workflow is.
Particularly this:
"if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes"
looks odd. Why would you make copies? Just set the needed flags with a variable that is set from a global config,
perhaps the distro or local.conf.
Alex
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Alex,
we are getting a bit off topic here, but whatever... ;)
Quoted 2 lines
What I mean is that I would try harder to find a workable setup while keeping the recipes for individual items that need to be built.
the individual recipes don't really add that much benefit for us, mostly just maintenance overhead and complexity, as we need to constantly ensure that the various components depend on each other in the right order. The only beneficial aspect to the multitude of recipes is the sstate cache. However, as the packages heavily depend on one another, the benefit is marginal, as we have to rebuild most of the application anyways.
Additionally, this is not how the developers build the application, so basically we currently have to maintain two separate build workflows. Changing to repo also improves the developer workflow outside of yocto, e.g. thanks to the notion of "topics". The "devtool" option, whilst duable I assume, would again limit the usability to within the Yocto environment. We are currently trying to get to a point, where we manage most of our (automated) workloads (e.g. testing) within Yocto but historically this was not the case and for daily development this is just not really practical.
Quoted 6 lines
Particularly this:
"if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes" looks odd. Why would you make copies? Just set the needed flags with a variable that is set from a global config,
perhaps the distro or local.conf.
As far as I understand, this is not reasonably possible (though I might be wrong. TBH, understanding variable scopes in Yocto is a nightmare).
To give this some context, our current base setup looks as follows:
- - We have one distro.conf
- - We have three image recipes with slightly different config + featureset
- - AFAIK an image recipe cannot set variables within another recipe, it just "consumes" existing recipes with their configuration.
- -> Thus we have three respective recipes with slightly different configuration. If we where to use individual recipes in this setup, we'd have 50ish x 3 recipes.
The approach you are describing might circumvent this particular situation but basically just moves the issue around, as you're now unabe to build all three configurations in one go, but need to start juggling your local.confs. Alternatively you might think about using multi-confs. However, each multiconf doubles the parse times for your recipes. As we already use a couple of multiconfs for different machine configs, we would basically increase the time it takes to parse the recipes by ~fifteen-fold (5 pre-existing multi-confs x 3 (or more) disto confs). Additionally, you now need to start worrying about artifact names, as by default the distro name is not included.
So yeah... as far as I can tell, there are multiple ways to approach this issue, but none of them seem straight forward nor pretty. Bitbake as it is is just fundamentally not good at handling highly dynamic configurations. The combination with KAS somewhat defuses the situation, but there are still some situations where there is no easy answer.
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 19 lines · Alexander Kanavin
Quoted 7 lines · Jasper Orschulko
But that is exactly what we are doing, by integrating the repo fetcher into the yocto workflow, thus improving the yocto tooling :)
Why reinvent the wheel, when you can reuse whats already there? You wouldn't reinvent git just for yocto, would you?What I mean is that I would try harder to find a workable setup while keeping the recipes for individual items that need to be built. For instance, devtool is capable of updating source revisions in recipes automatically, it may not work exactly as you want, but that can be fixed.
Yes, repo itself is not proprietary, but your organizational workflow is.
Quoted 8 lines
Particularly this:
"if you want to create another version with different cmake flags, you would have to create copies of all 50ish recipes" looks odd. Why would you make copies? Just set the needed flags with a variable that is set from a global config,
perhaps the distro or local.conf.Alex
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGFlPMACgkQYgqew07V MNXoCAf/fN4gi1dfx7r4acfbu7kaw9+3rjdMu34J6SX/ahGjCfbTZAg1Twf8N6RZ LBuOpDkXPU77iJuJixZySve5EZc4A2/NS0hqXEpD0aWf8AXaZzxq0UbEBM78oTgH Hk1zVEFxwLwEHpVIQyvwx5rFz/JvZ2EN4aH576ciPaw5n+bHIZBevUfhssPrjQWX yUTuWyeUoMJlXz41E+JfSd1VIQOC3hdWKF26er152zSyhYvhCagBMsHZMTKdrahn 2jmInGMSMizXR6vXBUTfwaB48Ba0iqFoi/tr7+jNIHV5WZNx4AvU61UOu8Lca6he bGvM5nM7xRpeHuNX2xa9xWs3SkUNEg==
=UJaE
-----END PGP SIGNATURE-----
Quoted 7 lines · Jasper Orschulko
So yeah... as far as I can tell, there are multiple ways to approach this issue, but none of them seem straight forward nor pretty. Bitbake as it is is just fundamentally not good at handling highly dynamic configurations. The combination with KAS somewhat defuses the situation, but there are still some situations where there is no easy answer.
Yes, yocto does not support 'variations' of the build that is already set up. Once you set the distro, and the machine, there's only one way to build a recipe; if you want another way, you need to indeed make another recipe, or another distro. I would still say making a set of distros is still the proper 'yocto way' to solve your problem. There are examples in meta-poky:
poky-altcfg and poky-tiny, and there has been poky-bleeding-revisions or similar in the past.
This is a reoccurring topic; there have been proposal in the past to introduce 'variants' as yet another abstraction, but they add so much complexity on top of what is already very complex, that the tradeoff is not acceptable to the project.
Alex
Quoted 5 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>
Quoted 10 lines
--- .../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33 +++++++++++++++++++ meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++ 4 files changed, 86 insertions(+) create mode 100644 meta/recipes-devtools/repo/files/0001-python3-shebang.patch create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bb
This basically looks ok to me, I've some minor comments below.
The patch is missing adding an entry to:
meta/conf/distro/include/maintainers.inc
which is required for OE-Core recipes and will trip up automated testing if we don't.
meta/recipes-devtools/repo/files/0001-python3-shebang.patch
Quoted 6 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +0
-0
I'd put this in a folder called "repo" rather than files.
meta/recipes-devtools/repo/files/0001-python3-shebang.patch line 8
Quoted 10 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +8
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-)
Missing Upstream-Status: (you got the second one ok though! :)
Quoted 59 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +12
-0+diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch new file +33
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:00 2001 +From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Python modules +using GPG-signed git tags, and stores those files as part of the project +that it is working with. + +So in order to have a reproducible repo installation within the project +folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes-devtools/repo/repo.inc new file +2
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors
meta/recipes-devtools/repo/repo.inc line 12
Quoted 10 lines
meta/recipes-devtools/repo/repo.inc new file +10
-0+ +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow." +HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-repo.git;protocol=https;branch=main" +MIRRORS = "git://[email protected]/git-repo.git git://github.com/GerritCodeReview/git-repo.git \n"
You shouldn't be clearing MIRRORS with "=" as the user may have something set in there the system needs to work, you probably want += here.
meta/recipes-devtools/repo/repo.inc line 18
Quoted 6 lines
meta/recipes-devtools/repo/repo.inc new file +6
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3"
This is old overrides syntax, it is RDEPENDS:${PN} now as others have mentioned.
Quoted 20 lines
meta/recipes-devtools/repo/repo.inc new file +7
-0+ +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-0@@ -0,0 +1,7 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard and Peter,
I've addressed your comments and will post the v3 patches momentarily.
Quick comment regarding the new overrides syntax, or rather the documentation. I just noticed that my go-to page for the mega-manual is outdated
(https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html)
and that the new location now seems to be
https://docs.yoctoproject.org/3.4/singleindex.html.
There is a redirect in place, e.g. when trying to call a 3.4 version of the mega manual like this
https://www.yoctoproject.org/docs/3.4/mega-manual/mega-manual.html.
However, this does not extend to the "latest" page, which is just a source for confusion. Is there any chance we could change that?
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 168 lines · Richard Purdie
Quoted 11 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>---
.../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
+++++++++++++++++++
meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++Quoted 7 lines
4 files changed, 86 insertions(+)create mode 100644 meta/recipes-devtools/repo/files/0001-python3- shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set- REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bbThis basically looks ok to me, I've some minor comments below.
The patch is missing adding an entry to:
meta/conf/distro/include/maintainers.inc
which is required for OE-Core recipes and will trip up automated testing if we
don't.meta/recipes-devtools/repo/files/0001-python3-shebang.patch
Quoted 8 lines
(quoted diff) +0
-0shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- shebang.patch
new file mode 100644
index 0000000000..09ccf58264meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +0
-0I'd put this in a folder called "repo" rather than files.
meta/recipes-devtools/repo/files/0001-python3-shebang.patch line 8
Quoted 11 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +1
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +7
-0+From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-)Missing Upstream-Status: (you got the second one ok though! :)
Quoted 78 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +12
-0+diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0(quoted diff) +0
-0REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 0000000000..294a3af53ameta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +0
-0v2.17.3.patch
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +1
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +5
-0+From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Pythonmodules
+using GPG-signed git tags, and stores those files as part of the projectmeta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +3
-0+that it is working with. + +So in order to have a reproducible repo installation within theproject
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +23
-0+folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes- renamed from meta/recipes-devtools/repo/repo.inc +0
-0devtools/repo/repo.inc
new file mode 100644
index 0000000000..60b32e4d74meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps managemany Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-Quoted 3 lines
repo.git;protocol=https;branch=main"
+MIRRORS = "git://[email protected]/git-repo.gitgit://github.com/GerritCodeReview/git-repo.git\n"You shouldn't be clearing MIRRORS with "=" as the user may have something set in
there the system needs to work, you probably want += here.meta/recipes-devtools/repo/repo.inc line 18
Quoted 6 lines
meta/recipes-devtools/repo/repo.inc new file +6
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3"This is old overrides syntax, it is RDEPENDS:${PN} now as others have mentioned.
Quoted 14 lines
meta/recipes-devtools/repo/repo.inc new file +7
-0+ +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +0
-0@@ -0,0 +1,7 @@
Quoted 10 lines
Quoted 8 lines
meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-0+# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM ="file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGJEEsACgkQYgqew07V MNUO0Qf/Rxt9hZHWTmDX6eGjEEbMxxSGyJ2You7H421dZZH9WMQxnXVHsl2n2C6B 2gzMqyRZJdnR/60tfPr97wuQbeyxlHjsxaYa5F8p23bYAyGVYaSm/uDXmLnqDmXQ qDmliY1XYkh+UhP/FkzmjCyCrEMPul1CLEZemY++RQZhBj5ge8W0G2JMsm7GAoa1 1dRghiclvvi5aHB2caSPsz9RlDHsSAdMZ78LheULVxAXBy4tE4h7p6JFMrK/pUy5 1wCLLQGSeqqlFbmRs3W3FKavlrPIvw80lq5DaxNJauMhhKGTPMKp6yu6V+Lr0BmX LkduXEmbmJRQ5oKxA5FFFzwXCiUe3g==
=48qv
-----END PGP SIGNATURE-----
Hi Michael and Michael,
This sounds like something we should prioritise fixing?
Cheers,
Richard
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hi Richard and Peter,
I've addressed your comments and will post the v3 patches momentarily.
Quick comment regarding the new overrides syntax, or rather the documentation. I just noticed that my go-to page for the mega-manual is outdated
(https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html)
and that the new location now seems to be
https://docs.yoctoproject.org/3.4/singleindex.html.
There is a redirect in place, e.g. when trying to call a 3.4 version of the mega manual like this
https://www.yoctoproject.org/docs/3.4/mega-manual/mega-manual.html.
However, this does not extend to the "latest" page, which is just a source for confusion. Is there any chance we could change that?
- --
With best regards
Jasper Orschulko
DevOps Engineer
Tel. +49 30 58 58 14 265
Fax +49 30 58 58 14 999
[email protected]
• • • • • • • • • • • • • • • • • • • • • • • • • •
iris-GmbH
infrared & intelligent sensors
Schnellerstraße 1-5 | 12439 Berlin
Quoted 168 lines · Richard Purdie
Quoted 11 lines · Jasper Orschulko via lists.openembedded.org
From: Jasper Orschulko <[email protected]>
Add a recipe for repo, prerequisite for the repo fetcher.
Signed-off-by Jasper Orschulko <[email protected]>---
.../repo/files/0001-python3-shebang.patch | 21 ++++++++++++ .../0001-Set-REPO_REV-to-v2.17.3.patch | 33
+++++++++++++++++++
meta/recipes-devtools/repo/repo.inc | 25 ++++++++++++++ meta/recipes-devtools/repo/repo_2.17.3.bb | 7 ++++Quoted 7 lines
4 files changed, 86 insertions(+)create mode 100644 meta/recipes-devtools/repo/files/0001-python3- shebang.patch
create mode 100644 meta/recipes-devtools/repo/repo-2.17.3/0001-Set- REPO_REV-to-v2.17.3.patch
create mode 100644 meta/recipes-devtools/repo/repo.inc create mode 100644 meta/recipes-devtools/repo/repo_2.17.3.bbThis basically looks ok to me, I've some minor comments below.
The patch is missing adding an entry to:
meta/conf/distro/include/maintainers.inc
which is required for OE-Core recipes and will trip up automated testing if we
don't.meta/recipes-devtools/repo/files/0001-python3-shebang.patch
Quoted 8 lines
(quoted diff) +0
-0shebang.patch b/meta/recipes-devtools/repo/files/0001-python3- shebang.patch
new file mode 100644
index 0000000000..09ccf58264meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +0
-0I'd put this in a folder called "repo" rather than files.
meta/recipes-devtools/repo/files/0001-python3-shebang.patch line 8
Quoted 11 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +1
-0@@ -0,0 +1,21 @@ +From b8e84b202cd302a7c99288d3835dc9c63071f8f2 Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +7
-0+From: Jasper Orschulko <[email protected]> +Date: Tue, 14 Sep 2021 16:46:51 +0200 +Subject: [PATCH] python3 shebang + +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-)Missing Upstream-Status: (you got the second one ok though! :)
Quoted 78 lines
meta/recipes-devtools/repo/files/0001-python3-shebang.patch new file +12
-0+diff --git a/repo b/repo +index b13e34c..205e0e5 100755 +--- a/repo ++++ b/repo +@@ -1,4 +1,4 @@ +-#!/usr/bin/env python ++#!/usr/bin/env python3 + # -*- coding:utf-8 -*- + # + # Copyright (C) 2008 The Android Open Source Project +-- +2.33.0(quoted diff) +0
-0REPO_REV-to-v2.17.3.patch b/meta/recipes-devtools/repo/repo-
2.17.3/0001-Set-REPO_REV-to-v2.17.3.patch
new file mode 100644
index 0000000000..294a3af53ameta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +0
-0v2.17.3.patch
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +1
-0@@ -0,0 +1,33 @@ +From bdd2a528da59c28db8ae2986834926de7cebf3ab Mon Sep 17 00:00:002001
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +5
-0+From: Jasper Orschulko <[email protected]> +Date: Thu, 4 Nov 2021 16:55:12 +0100 +Subject: [PATCH] Set REPO_REV to v2.17.3 + +repo is an unusual tool because it downloads all of its own Pythonmodules
+using GPG-signed git tags, and stores those files as part of the projectmeta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +3
-0+that it is working with. + +So in order to have a reproducible repo installation within theproject
meta/recipes-devtools/repo/repo-2.17.3/0001-Set-REPO_REV-to- new file +23
-0+folders, we hardcode the REPO_REV variable to this recipes PV. + +Upstream-Status: Inappropriate [configuration] +Signed-off-by: Jasper Orschulko <[email protected]> +--- + repo | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/repo b/repo +index 4cddbf1..cf5f6b1 100755 +--- a/repo ++++ b/repo +@@ -142,7 +142,7 @@ if __name__ == '__main__': + REPO_URL = os.environ.get('REPO_URL', None) + if not REPO_URL: + REPO_URL = 'https://gerrit.googlesource.com/git-repo' +-REPO_REV = os.environ.get('REPO_REV') ++REPO_REV = 'v2.17.3' + if not REPO_REV: + REPO_REV = 'stable' + # URL to file bug reports for repo tool issues. +-- +2.33.1meta/recipes- renamed from meta/recipes-devtools/repo/repo.inc +0
-0devtools/repo/repo.inc
new file mode 100644
index 0000000000..60b32e4d74meta/recipes-devtools/repo/repo.inc new file +5
-0@@ -0,0 +1,25 @@ +# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +SUMMARY = "Tool for managing many Git repositories" +DESCRIPTION = "Repo is a tool built on top of Git. Repo helps managemany Git repositories, does the uploads to revision control systems, and automates parts of the development workflow."
meta/recipes-devtools/repo/repo.inc new file +6
-0+HOMEPAGE = "https://android.googlesource.com/tools/repo" +SECTION = "console/utils" + +LICENSE = "Apache-2.0" + +SRC_URI = "git://[email protected]/git-Quoted 3 lines
repo.git;protocol=https;branch=main"
+MIRRORS = "git://[email protected]/git-repo.gitgit://github.com/GerritCodeReview/git-repo.git\n"You shouldn't be clearing MIRRORS with "=" as the user may have something set in
there the system needs to work, you probably want += here.meta/recipes-devtools/repo/repo.inc line 18
Quoted 6 lines
meta/recipes-devtools/repo/repo.inc new file +6
-0+ +SRC_URI += "file://0001-python3-shebang.patch" + +S = "${WORKDIR}/git" + +RDEPENDS_${PN} = "python3"This is old overrides syntax, it is RDEPENDS:${PN} now as others have mentioned.
Quoted 14 lines
meta/recipes-devtools/repo/repo.inc new file +7
-0+ +do_install() { + install -d ${D}${bindir} + install -m 755 ${WORKDIR}/git/repo ${D}${bindir} +} + +BBCLASSEXTEND = "native nativesdk"meta/recipes-devtools/repo/repo_2.17.3.bb new file +0
-0@@ -0,0 +1,7 @@
Quoted 10 lines
Quoted 8 lines
meta/recipes-devtools/repo/repo_2.17.3.bb new file +7
-0+# SPDX-License-Identifier: MIT +# Copyright (C) 2021 iris-GmbH infrared & intelligent sensors + +require recipes-devtools/repo/repo.inc + +SRCREV = "11b30b91df1f0e03b53da970ec2588e85817bacc" +LIC_FILES_CHKSUM ="file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
-----BEGIN PGP SIGNATURE-----
iQEzBAEBCAAdFiEE4WyPMIC5Ap4+Ooo1Ygqew07VMNUFAmGJEEsACgkQYgqew07V MNUO0Qf/Rxt9hZHWTmDX6eGjEEbMxxSGyJ2You7H421dZZH9WMQxnXVHsl2n2C6B 2gzMqyRZJdnR/60tfPr97wuQbeyxlHjsxaYa5F8p23bYAyGVYaSm/uDXmLnqDmXQ qDmliY1XYkh+UhP/FkzmjCyCrEMPul1CLEZemY++RQZhBj5ge8W0G2JMsm7GAoa1 1dRghiclvvi5aHB2caSPsz9RlDHsSAdMZ78LheULVxAXBy4tE4h7p6JFMrK/pUy5 1wCLLQGSeqqlFbmRs3W3FKavlrPIvw80lq5DaxNJauMhhKGTPMKp6yu6V+Lr0BmX LkduXEmbmJRQ5oKxA5FFFzwXCiUe3g==
=48qv
-----END PGP SIGNATURE-----
Quoted 4 lines · Richard Purdie
Hi Michael and Michael,
This sounds like something we should prioritise fixing?
I agree.
Hijacking for a different (though related) topic:
I think we also need to add a banner stating that this doc is out-of-date. But I just don't have the skills to do it quickly and still have something nice enough.
Considering how long the pages in the documentation are, I was trying to implement a banner that would stick to the top of the view (stays visible even when scrolling or linking to somewhere in the middle of the page).
And the same to do for the current docs.yoctoproject.org (though there's already a header, it;s not at the top of the view).
If anyone has the skills to do it quickly, please do so :) otherwise, I guess I'll come to it the day I can't stand it anymore :)
Cheers,
Quentin
Richard, Jasper,
Quoted 3 lines · Richard Purdie
Hi Michael and Michael,
This sounds like something we should prioritise fixing?
Good catch, thanks!
I agree,
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
should also redirect to https://docs.yoctoproject.org/singleindex.html
I'll ask Michael H. to change this.
Cheers
Michael
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 5 lines · Michael Opdenacker
Richard, Jasper,
Quoted 2 lines · Richard Purdie
Hi Michael and Michael,
Quoted 8 lines
Quoted 1 line
This sounds like something we should prioritise fixing?
Good catch, thanks!
I agree,
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
should also redirect to https://docs.yoctoproject.org/singleindex.htmlI'll ask Michael H. to change this.
Michael H. took care of this.
Thanks again
Michael.
Signature · 3 lines
Michael Opdenacker, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 9 lines · Michael Opdenacker
Quoted 8 lines · Michael Opdenacker
Richard, Jasper,
Quoted 3 lines · Richard Purdie
Hi Michael and Michael,
This sounds like something we should prioritise fixing?
Good catch, thanks!
Quoted 7 lines
Quoted 5 lines
I agree,
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html
should also redirect to https://docs.yoctoproject.org/singleindex.htmlI'll ask Michael H. to change this.
Michael H. took care of this.
Thanks all, that should be a useful fix.
Cheers,
Richard