Skip to contentSkip to conversation

insane.bbclass: in file-rdeps do not look into RDEPENDS recursively

PATCH In review Alexander Kanavin 3 participants 18 messages Aug 7, 2019 – Aug 13, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/7 insane.bbclass: in file-rdeps do not look into RDEPENDS recursively

Recursive RDEPENDS resolution requires that all of the dependent recipes' packaging has completed. There is no mechanism to ensure that and therefore races were observed.

This change effectively requires recipes to list their runtime file dependencies explicitly rather than have them pulled indirectly. This may require a bit of fixing in layers, but should result in a better definition of runtime file dependencies.

---
 meta/classes/insane.bbclass | 18 ------------------
 1 file changed, 18 deletions(-)
meta/classes/insane.bbclass +0 -18
index 35c4fdb4913..9b886d13805 100644
@@ -722,25 +722,7 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d):
                             filerdepends[subkey] = key[13:]
 
             if filerdepends:
-                next = rdepends
                 done = rdepends[:]
-                # Find all the rdepends on the dependency chain
-                while next:
-                    new = []
-                    for rdep in next:
-                        rdep_data = oe.packagedata.read_subpkgdata(rdep, d)
-                        sub_rdeps = rdep_data.get("RDEPENDS_" + rdep)
-                        if not sub_rdeps:
-                            continue
-                        for sub_rdep in bb.utils.explode_deps(sub_rdeps):
-                            if sub_rdep in done:
-                                continue
-                            if oe.packagedata.has_subpkgdata(sub_rdep, d):
-                                # It's a new rdep
-                                done.append(sub_rdep)
-                                new.append(sub_rdep)
-                    next = new
-
                 # Add the rprovides of itself
                 if pkg not in done:
                     done.insert(0, pkg)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 2/7 stress-ng: add a patch to remove unneeded bash dependency

---
 ...tion-remove-the-shebang-at-the-start.patch | 23 +++++++++++++++++++
 .../stress-ng/stress-ng_0.10.00.bb            |  1 +
 2 files changed, 24 insertions(+)
 create mode 100644 meta/recipes-extended/stress-ng/stress-ng/0001-bash-completion-remove-the-shebang-at-the-start.patch
meta/recipes-extended/stress-ng/stress-ng/0001-bash-completion-remove-the-shebang-at-the-start.patch new file +23 -0
new file mode 100644
index 00000000000..66d99dd885f
@@ -0,0 +1,23 @@
+From 042147675c7c2ea7dd65b2597f2e350376a710aa Mon Sep 17 00:00:00 2001
+From: Alexander Kanavin <[email protected]>
+Date: Tue, 6 Aug 2019 17:28:56 +0200
+Subject: [PATCH] bash-completion: remove the shebang at the start
+
+bash completion files do not need to specify that.
+
+Upstream-Status: Pending
+Signed-off-by: Alexander Kanavin <[email protected]>
+---
+ bash-completion/stress-ng | 2 --
+ 1 file changed, 2 deletions(-)
+
+diff --git a/bash-completion/stress-ng b/bash-completion/stress-ng
+index 8b1421c..7f195be 100755
+--- a/bash-completion/stress-ng
++++ b/bash-completion/stress-ng
+@@ -1,5 +1,3 @@
+-#!/bin/bash
+-#
+ # stress-ng tab completion for bash.
+ #
+ # Copyright (C) 2019 Canonical
meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb +1 -0
index f7faf97e681..e800040c371 100644
@@ -7,6 +7,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
 
 SRC_URI = "https://kernel.ubuntu.com/~cking/tarballs/${BPN}/${BP}.tar.xz \
            file://0001-Do-not-preserve-ownership-when-installing-example-jo.patch \
+           file://0001-bash-completion-remove-the-shebang-at-the-start.patch \
            "
 SRC_URI[md5sum] = "46aa41d37690324ceab4febfcc549018"
 SRC_URI[sha256sum] = "d09dd2a1aea549e478995bf9be90b38906a4cdf33ea7b245ef9d46aa5213c074"
Signature · 1 line
2.17.1
#2 lore raw

PATCH 3/7 elfutils: use PRIVATE_LIBS for the ptest package

In reply to Alexander Kanavin

EXCLUDE_PACKAGES_FROM_SHLIBS is too broad: it suppresses both generation of required and provided shlibs. We need to suppress only the provided shlibs (to avoid clashes with the main package providing the same shlibs), and run the required shlib dependencies generator as usual.

---
 meta/recipes-devtools/elfutils/elfutils_0.176.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/elfutils/elfutils_0.176.bb +1 -1
index 75acf59d26e..d98457e3f9f 100644
@@ -142,4 +142,4 @@ INHIBIT_PACKAGE_STRIP_FILES = "\
     ${PKGD}${PTEST_PATH}/backends/libebl_x86_64.so \
 "
 
-EXCLUDE_PACKAGES_FROM_SHLIBS = "${PN}-ptest"
+PRIVATE_LIBS_${PN}-ptest = "libdw.so.1 libelf.so.1"
Signature · 1 line
2.17.1
#3 lore raw

PATCH 4/7 apt: add a missing perl runtime dependency

In reply to Alexander Kanavin

/usr/lib/dpkg/methods/apt/setup is a perl script.

---
 meta/recipes-devtools/apt/apt-package.inc | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-devtools/apt/apt-package.inc +1 -0
index da91451d7f6..922f739d0d0 100644
@@ -35,6 +35,7 @@ FILES_${PN} = "${bindir}/apt-cdrom ${bindir}/apt-get \
 	       ${libdir}/dpkg \
 	       ${systemd_unitdir}/system \
            "
+RDEPENDS_${PN} += "perl"
 FILES_${PN}-utils = "${bindir}/apt-sortpkgs ${bindir}/apt-extracttemplates"
 FILES_${PN}-doc = "${@get_files_apt_doc(d, bb, d.getVar('apt-manpages'))} \
 		   ${docdir}/apt"
Signature · 1 line
2.17.1
#4 lore raw

PATCH 5/7 attr: add a missing perl runtime dependency

In reply to Alexander Kanavin

/usr/lib/attr/ptest/test/sort-getfattr-output is a perl script.

---
 meta/recipes-support/attr/attr.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/attr/attr.inc +1 -1
index 1a29eb339e7..3503d66b9e1 100644
@@ -41,6 +41,6 @@ do_install_ptest() {
 	sed -e 's|; @echo|; echo|' -i ${D}${PTEST_PATH}/test/Makefile
 }
 
-RDEPENDS_${PN}-ptest = "attr coreutils perl-module-filehandle perl-module-getopt-std perl-module-posix make"
+RDEPENDS_${PN}-ptest = "attr coreutils perl-module-filehandle perl-module-getopt-std perl-module-posix make perl"
 
 BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.17.1
#5 lore raw

PATCH 6/7 ofono: correct the python3 runtime dependency

In reply to Alexander Kanavin
---
 meta/recipes-connectivity/ofono/ofono.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-connectivity/ofono/ofono.inc +1 -1
index 03d0fd53a61..bdbb0b5bc61 100644
@@ -35,5 +35,5 @@ RRECOMMENDS_${PN} += "kernel-module-tun mobile-broadband-provider-info"
 
 FILES_${PN} += "${systemd_unitdir}"
 FILES_${PN}-tests = "${libdir}/${BPN}/test"
-RDEPENDS_${PN}-tests = "python3 python3-dbus"
+RDEPENDS_${PN}-tests = "python3-core python3-dbus"
 RDEPENDS_${PN}-tests += "${@bb.utils.contains('GI_DATA_ENABLED', 'True', 'python3-pygobject', '', d)}"
Signature · 1 line
2.17.1
#6 lore raw

PATCH 7/7 bluez5: correct the python3 runtime dependency

In reply to Alexander Kanavin
---
 meta/recipes-connectivity/bluez5/bluez5.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-connectivity/bluez5/bluez5.inc +1 -1
index 28aaeea418f..bffd174c0a0 100644
@@ -139,7 +139,7 @@ def get_noinst_tools_paths (d, bb, tools):
 
 FILES_${PN}-noinst-tools = "${@get_noinst_tools_paths(d, bb, d.getVar('NOINST_TOOLS'))}"
 
-RDEPENDS_${PN}-testtools += "python3 python3-dbus"
+RDEPENDS_${PN}-testtools += "python3-core python3-dbus"
 RDEPENDS_${PN}-testtools += "${@bb.utils.contains('GI_DATA_ENABLED', 'True', 'python3-pygobject', '', d)}"
 
 SYSTEMD_SERVICE_${PN} = "${@bb.utils.contains('PACKAGECONFIG', 'systemd', 'bluetooth.service', '', d)}"
Signature · 1 line
2.17.1
#7 lore raw
In reply to Alexander Kanavin
Quoted 10 lines · Alexander Kanavin

Recursive RDEPENDS resolution requires that all of the dependent recipes' packaging has completed. There is no mechanism to ensure that
and therefore races were observed.

This change effectively requires recipes to list their runtime file dependencies explicitly rather than have them pulled indirectly. This may require a bit of fixing in layers, but should result in a better definition of runtime file dependencies.

Quoted 4 lines
---
 meta/classes/insane.bbclass | 18 ------------------
 1 file changed, 18 deletions(-)

You might want to have a look at
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/621
and maybe other pieces of that build yet to finish, not sure...

Cheers,

Richard

#9 lore raw
Quoted 11 lines · Alexander Kanavin
Quoted 4 lines · Richard Purdie

You might want to have a look at
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/621
and maybe other pieces of that build yet to finish, not sure...

I ran bitbake world to check for newly exposed missing dependencies, but gcc-cross-canadian wasn't included in that. Will fix that too now.

There are several packages in meta-openembedded which are showing these errors one that I see is net kit-rusers seems to be failing for wrong reason

I will send full list once I am close to computer

Quoted 8 lines

Alex
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#10 lore raw
Quoted 12 lines · Khem Raj
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines · Richard Purdie

You might want to have a look at
https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/621
and maybe other pieces of that build yet to finish, not sure...

Quoted 6 lines
Quoted 1 line

I ran bitbake world to check for newly exposed missing dependencies, but gcc-cross-canadian wasn't included in that. Will fix that too now.

There are several packages in meta-openembedded which are showing these errors one that I see is net kit-rusers seems to be failing for wrong reason

I will send full list once I am close to computer

here is full list

https://errors.yoctoproject.org/Errors/Build/86513/
all do_package_qa are due to this change.

Quoted 7 lines
Quoted 7 lines

Alex
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#11 lore raw
Quoted 10 lines · Khem Raj
Quoted 1 line

There are several packages in meta-openembedded which are showing these

errors one that I see is net kit-rusers seems to be failing for wrong reason

Quoted 2 lines

I will send full list once I am close to computer

here is full list

https://errors.yoctoproject.org/Errors/Build/86513/
all do_package_qa are due to this change.

There are plenty of recipes in that list that come from oe-core (not meta-oe), and do not show errors on the AB, e.g. glib-2.0:

ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1(ZLIB_1.2.2)(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libglib-2.0.so.0.6000.6 contained in package glib-2.0 requires libpcre.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgobject-2.0.so.0.6000.6 contained in package glib-2.0 requires libffi.so.6()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]

Can I see the configuration where this happened? Might be some other setting that breaks shlib resolution (that was previously unnoticed due to recursive qa RDEPENDS).

Alex

#12 lore raw
Quoted 13 lines · Alexander Kanavin
Quoted 9 lines · Khem Raj
Quoted 4 lines

There are several packages in meta-openembedded which are showing these errors one that I see is net kit-rusers seems to be failing for wrong reason

I will send full list once I am close to computer

here is full list

https://errors.yoctoproject.org/Errors/Build/86513/
all do_package_qa are due to this change.

Quoted 9 lines

There are plenty of recipes in that list that come from oe-core (not meta-oe), and do not show errors on the AB, e.g. glib-2.0:

ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1(ZLIB_1.2.2)(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libglib-2.0.so.0.6000.6 contained in package glib-2.0 requires libpcre.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgobject-2.0.so.0.6000.6 contained in package glib-2.0 requires libffi.so.6()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]

Can I see the configuration where this happened? Might be some other setting that breaks shlib resolution (that was previously unnoticed due to recursive qa RDEPENDS).

a running musl world build did not show this issue but that was rebased tree without this fix. I will have to do a new build for glibc and see if it was my build system injected behavior. For this patch, I would suggest it's tested across multiple layers before it's installed to oe-core.

Quoted 2 lines

Alex

#13 lore raw
Quoted 6 lines · Khem Raj

a running musl world build did not show this issue but that was rebased tree without this fix. I will have to do a new build for glibc and see if it was my build system injected behavior. For this patch, I would suggest it's tested across multiple layers before it's installed to oe-core.

We should certainly at least fix the meta-oe issues, if you can produce a list of affected recipes I would appreciate :)
(don't really want to start a build of entire meta-oe set here)

Alex

#14 lore raw
In reply to Alexander Kanavin
Quoted 13 lines · Alexander Kanavin
Quoted 9 lines · Khem Raj
Quoted 4 lines

There are several packages in meta-openembedded which are showing these errors one that I see is net kit-rusers seems to be failing for wrong reason

I will send full list once I am close to computer

here is full list

https://errors.yoctoproject.org/Errors/Build/86513/
all do_package_qa are due to this change.

Quoted 8 lines

There are plenty of recipes in that list that come from oe-core (not meta-oe), and do not show errors on the AB, e.g. glib-2.0:

ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1(ZLIB_1.2.2)(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgio-2.0.so.0.6000.6 contained in package glib-2.0 requires libz.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libglib-2.0.so.0.6000.6 contained in package glib-2.0 requires libpcre.so.1()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]
ERROR: QA Issue: /usr/lib/libgobject-2.0.so.0.6000.6 contained in package glib-2.0 requires libffi.so.6()(64bit), but no providers found in RDEPENDS_glib-2.0? [file-rdeps]

Can I see the configuration where this happened? Might be some other setting that breaks shlib resolution (that was previously unnoticed due to recursive qa RDEPENDS).

well, this seems to go away once I disable hash equivalency

#INHERIT += "reproducible_build"
#BB_HASHSERVE = "localhost:0"
#BB_SIGNATURE_HANDLER = "OEEquivHash"

Richard, another issue I saw was slowness in world builds when I had this enabled
I was on master-next for both core and bitbake. It was spending a lot of time in ensuring
the build is not needed so much that the build took almost same time than the build with
out it where the build without it will build everything.

#15 lore raw
Quoted 9 lines · Khem Raj

Richard, another issue I saw was slowness in world builds when I had this enabled
I was on master-next for both core and bitbake. It was spending a lot of time in ensuring
the build is not needed so much that the build took almost same time than the build with
out it where the build without it will build everything.

world builds seem to have regressed (preparation time wise) even without having hash equivalency. Just oe-core is bearable, but adding meta-oe layers makes it spent many minutes after initializing tasks but before any tasks actually begin.

Alex

#16 lore raw
Quoted 6 lines · Alexander Kanavin
Quoted 5 lines · Khem Raj

Richard, another issue I saw was slowness in world builds when I had this enabled I was on master-next for both core and bitbake. It was spending a lot of time in ensuring the build is not needed so much that the build took almost same time than the build with out it where the build without it will build everything.

Were you using the patch in meta-yocto master-next or your own?

There is definitely a server connection problem with the hashserver right now but if you used the setting in meta-yocto master-next you'd probably only ever see timeouts which would be unbearably slow.

Quoted 5 lines

world builds seem to have regressed (preparation time wise) even without having hash equivalency. Just oe-core is bearable, but adding meta-oe layers makes it spent many minutes after initializing tasks but before any tasks actually begin.

Is that with master or current master-next?

I'm curious if master-next is better or worse...

To be honest I'm worrying more about correctness right now.

Cheers,

Richard

#17 lore raw
Quoted 6 lines · On Mon, 12 Aug 2019 at 22:50, <[email protected]> wrote:
Quoted 5 lines

world builds seem to have regressed (preparation time wise) even without having hash equivalency. Just oe-core is bearable, but adding meta-oe layers makes it spent many minutes after initializing tasks but before any tasks actually begin.

Quoted 6 lines

Is that with master or current master-next?

I'm curious if master-next is better or worse...

To be honest I'm worrying more about correctness right now.

Certainly. I didn't want to raise this now as you have plenty on your plate, but as Khem did, I had to chime in.

This was with master@9983b07fffd19082abded7c3f15cc77d306dd69c.

Alex