Skip to contentSkip to conversation

python: Run pybench once for optimisation task in pgo

PATCH In review Richard Purdie 6 participants 20 messages Sep 5, 2018 – Sep 7, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/8 python: Run pybench once for optimisation task in pgo

From: Ross Burton <[email protected]>

There is no good reason to loop 10 times on the test since the profile optimisation won't change between runs, we don't need/want a statistical average. This is just burning cpu cycles, just make 1 run.

---
 meta/recipes-devtools/python/python3_3.5.5.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3_3.5.5.bb +1 -1
index bfe91fb1c71..8afe15eee6a 100644
@@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline"
 SDK_CC_ARCH += "-DNDEBUG -fno-inline"
 EXTRA_OEMAKE += "CROSS_COMPILE=yes"
 EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations"
-PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10"
+PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1"
 
 export CROSS_COMPILE = "${TARGET_PREFIX}"
 export _PYTHON_PROJECT_BASE = "${B}"
Signature · 1 line
2.17.1
#1 lore raw

PATCH 2/8 os-release: fix to install in the expected location

From: Joshua Lock <[email protected]>

os-release (5) recommends that the os-release file be installed in /usr/lib/os-release and that /etc/os-release be a relative symlink to it.

---
 meta/recipes-core/os-release/os-release.bb | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
meta/recipes-core/os-release/os-release.bb +6 -3
index f9887047561..dae5748ba7a 100644
@@ -1,7 +1,7 @@
 inherit allarch
 
 SUMMARY = "Operating system identification"
-DESCRIPTION = "The /etc/os-release file contains operating system identification data."
+DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data."
 LICENSE = "MIT"
 INHIBIT_DEFAULT_DEPS = "1"
 
@@ -42,6 +42,9 @@ python do_compile () {
 do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
 
 do_install () {
-    install -d ${D}${sysconfdir}
-    install -m 0644 os-release ${D}${sysconfdir}/
+    install -d ${D}${libdir} ${D}${sysconfdir}
+    install -m 0644 os-release ${D}${libdir}/
+    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea
 }
+
+FILES_${PN} += "${libdir}/os-release"
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH 2/8] os-release: fix to install in the expected location

Quoted 26 lines · Richard Purdie

From: Joshua Lock <[email protected]>

os-release (5) recommends that the os-release file be installed in /usr/lib/os-release and that /etc/os-release be a relative symlink to it.

---
 meta/recipes-core/os-release/os-release.bb | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
meta/recipes-core/os-release/os-release.bb +1 -1
index f9887047561..dae5748ba7a 100644
@@ -1,7 +1,7 @@
 inherit allarch
 
 SUMMARY = "Operating system identification"
-DESCRIPTION = "The /etc/os-release file contains operating system identification data."
+DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data."
 LICENSE = "MIT"
 INHIBIT_DEFAULT_DEPS = "1"
 

meta/recipes-core/os-release/os-release.bb python do_compile () { · line 47

Quoted 8 lines
meta/recipes-core/os-release/os-release.bb +3 -2
@@ -42,6 +42,9 @@ python do_compile () {
 do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
 
 do_install () {
-    install -d ${D}${sysconfdir}
-    install -m 0644 os-release ${D}${sysconfdir}/
+    install -d ${D}${libdir} ${D}${sysconfdir}
+    install -m 0644 os-release ${D}${libdir}/
+    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea

Typo.

Please add the rm too when you fix that.

Quoted 11 lines
meta/recipes-core/os-release/os-release.bb +2 -1
 }
+
+FILES_${PN} += "${libdir}/os-release"
--

2.17.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#3 lore raw

Re: [PATCH 2/8] os-release: fix to install in the expected location

Quoted 43 lines · Andre McCurdy
Quoted 30 lines · Richard Purdie

From: Joshua Lock <[email protected]>

os-release (5) recommends that the os-release file be installed in /usr/lib/os-release and that /etc/os-release be a relative symlink to it.

---
 meta/recipes-core/os-release/os-release.bb | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
meta/recipes-core/os-release/os-release.bb +0 -1
index f9887047561..dae5748ba7a 100644
@@ -1,7 +1,7 @@
 inherit allarch
 
 SUMMARY = "Operating system identification"
-DESCRIPTION = "The /etc/os-release file contains operating system

identification data."
+DESCRIPTION = "The /usr/lib/os-release file contains operating system identification data."
LICENSE = "MIT"
INHIBIT_DEFAULT_DEPS = "1"

meta/recipes-core/os-release/os-release.bb python do_compile () { · line 47

Quoted 8 lines
meta/recipes-core/os-release/os-release.bb +3 -2
@@ -42,6 +42,9 @@ python do_compile () {
 do_compile[vardeps] += "${OS_RELEASE_FIELDS}"
 
 do_install () {
-    install -d ${D}${sysconfdir}
-    install -m 0644 os-release ${D}${sysconfdir}/
+    install -d ${D}${libdir} ${D}${sysconfdir}
+    install -m 0644 os-release ${D}${libdir}/
+    lnr ${D}${libdir}/os-release ${D}${sysconfdir}/os-relea

Typo.

Quoted 1 line

Please add the rm too when you fix that.

Too late, was already spotted and fixed. I'm not adding the rm since we're not going through all the do_installs adding rm statements in for things, I am totally against doing that.

Cheers,

Richard

#4 lore raw

PATCH 3/8 gcc: Merge two related patches together

In reply to Richard Purdie

These patches may as well be merged together as they affect the same code and things are clearer this way.

---
 meta/recipes-devtools/gcc/gcc-8.2.inc         |  1 -
 .../0003-gcc-poison-system-directories.patch  |  6 ++-
 .../gcc-8.2/0004-gcc-poison-dir-extend.patch  | 39 -------------------
 3 files changed, 4 insertions(+), 42 deletions(-)
 delete mode 100644 meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch
meta/recipes-devtools/gcc/gcc-8.2.inc +0 -1
index 8503a66b074..889187e9c42 100644
@@ -33,7 +33,6 @@ SRC_URI = "\
            ${BASEURI} \
            file://0001-gcc-4.3.1-ARCH_FLAGS_FOR_TARGET.patch \
            file://0003-gcc-poison-system-directories.patch \
-           file://0004-gcc-poison-dir-extend.patch \
            file://0005-gcc-4.3.3-SYSROOT_CFLAGS_FOR_TARGET.patch \
            file://0006-64-bit-multilib-hack.patch \
            file://0007-optional-libstdc.patch \
meta/recipes-devtools/gcc/gcc-8.2/0003-gcc-poison-system-directories.patch +4 -2
index 1839d4c7b8e..05138976c3b 100644
@@ -162,7 +162,7 @@ index b11c6a57939..e3b7a21966f 100644
  
  /* Microsoft Windows does not natively support inodes.
     VMS has non-numeric inodes.  */
-@@ -393,6 +394,24 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
+@@ -393,6 +394,26 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
  	}
        fprintf (stderr, _("End of search list.\n"));
      }
@@ -176,7 +176,9 @@ index b11c6a57939..e3b7a21966f 100644
 +         {
 +          if ((!strncmp (p->name, "/usr/include", 12))
 +              || (!strncmp (p->name, "/usr/local/include", 18))
-+              || (!strncmp (p->name, "/usr/X11R6/include", 18)))
++              || (!strncmp (p->name, "/usr/X11R6/include", 18))
++              || (!strncmp (p->name, "/sw/include", 11))
++              || (!strncmp (p->name, "/opt/include", 12)))
 +            warning (OPT_Wpoison_system_directories,
 +                     "include location \"%s\" is unsafe for "
 +                     "cross-compilation",
meta/recipes-devtools/gcc/gcc-8.2/0004-gcc-poison-dir-extend.patch deleted +0 -39
deleted file mode 100644
index 1072d5b18fb..00000000000
@@ -1,39 +0,0 @@
-From 97370eadcd10da5da529dfb734df56993ce24644 Mon Sep 17 00:00:00 2001
-From: Khem Raj <[email protected]>
-Date: Fri, 29 Mar 2013 09:00:34 +0400
-Subject: [PATCH 04/39] gcc-poison-dir-extend
-
-Add /sw/include and /opt/include based on the original
-zecke-no-host-includes.patch patch.  The original patch checked for
-/usr/include, /sw/include and /opt/include and then triggered a failure and
-aborted.
-
-Instead, we add the two missing items to the current scan.  If the user
-wants this to be a failure, they can add "-Werror=poison-system-directories".
-
-Signed-off-by: Mark Hatle <[email protected]>
-Signed-off-by: Khem Raj <[email protected]>
-
-Upstream-Status: Pending
----
- gcc/incpath.c | 4 +++-
- 1 file changed, 3 insertions(+), 1 deletion(-)
-
-diff --git a/gcc/incpath.c b/gcc/incpath.c
-index e3b7a21966f..9a457e02dd3 100644
---- a/gcc/incpath.c
-+++ b/gcc/incpath.c
-@@ -404,7 +404,9 @@ merge_include_chains (const char *sysroot, cpp_reader *pfile, int verbose)
-          {
-           if ((!strncmp (p->name, "/usr/include", 12))
-               || (!strncmp (p->name, "/usr/local/include", 18))
--              || (!strncmp (p->name, "/usr/X11R6/include", 18)))
-+              || (!strncmp (p->name, "/usr/X11R6/include", 18))
-+              || (!strncmp (p->name, "/sw/include", 11))
-+              || (!strncmp (p->name, "/opt/include", 12)))
-             warning (OPT_Wpoison_system_directories,
-                      "include location \"%s\" is unsafe for "
-                      "cross-compilation",
--- 
-2.17.0
-
Signature · 1 line
2.17.1
#5 lore raw

PATCH 4/8 binutils: Fix variable conflict

In reply to Richard Purdie

A recent binutils patch added the LDGOLD variable but its already used for controlling EXTRA_OECONF options for gold. Separate the two variables to different names to avoid build warnings and confusion.

---
 meta/recipes-devtools/binutils/binutils.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-devtools/binutils/binutils.inc +3 -3
index b56fcd4cd01..81ecbb50cc1 100644
@@ -38,8 +38,8 @@ FILES_${PN}-dev = " \
 # Rather than duplicating multiple entries for these, make one
 # list and reuse it.
 
-LDGOLD ?= "ld.gold dwp"
-LDGOLD_riscv64 = ""
+LDGOLD_ALTS ?= "ld.gold dwp"
+LDGOLD_ALTS_riscv64 = ""
 
 USE_ALTERNATIVES_FOR = " \
 	addr2line \
@@ -50,7 +50,7 @@ USE_ALTERNATIVES_FOR = " \
 	gprof \
 	ld \
 	ld.bfd \
-	${LDGOLD} \
+	${LDGOLD_ALTS} \
 	nm \
 	objcopy \
 	objdump \
Signature · 1 line
2.17.1
#6 lore raw

Re: [PATCH 4/8] binutils: Fix variable conflict

Quoted 25 lines · Richard Purdie

A recent binutils patch added the LDGOLD variable but its already used for controlling EXTRA_OECONF options for gold. Separate the two variables to different names to avoid build warnings and confusion.

---
 meta/recipes-devtools/binutils/binutils.inc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/recipes-devtools/binutils/binutils.inc +2 -2
index b56fcd4cd01..81ecbb50cc1 100644
@@ -38,8 +38,8 @@ FILES_${PN}-dev = " \
 # Rather than duplicating multiple entries for these, make one
 # list and reuse it.
 
-LDGOLD ?= "ld.gold dwp"
-LDGOLD_riscv64 = ""
+LDGOLD_ALTS ?= "ld.gold dwp"
+LDGOLD_ALTS_riscv64 = ""
 
 USE_ALTERNATIVES_FOR = " \
 	addr2line \

meta/recipes-devtools/binutils/binutils.inc USE_ALTERNATIVES_FOR = " \ · line 53

Quoted 5 lines
meta/recipes-devtools/binutils/binutils.inc +1 -1
@@ -50,7 +50,7 @@ USE_ALTERNATIVES_FOR = " \
 	gprof \
 	ld \
 	ld.bfd \
-	${LDGOLD} \
+	${LDGOLD_ALTS} \

aw my bad. Thanks for taking care of it.

Quoted 4 lines
meta/recipes-devtools/binutils/binutils.inc +0 -0
 	nm \
 	objcopy \
 	objdump \
#7 lore raw

PATCH 5/8 populate_base_sdk: Stop running gcc --version all the time

In reply to Richard Purdie

Running 'gcc --version' for every image recipe is slow and increases parsing time/resource usage for no good reason. Only compute the value in when we're really running the task/function.

---
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/lib/oe/utils.py                   | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)
meta/classes/populate_sdk_base.bbclass +1 -1
index c456c52866a..bbf1ff1cd29 100644
@@ -259,7 +259,7 @@ EOF
 		-e 's#@SDK_VERSION@#${SDK_VERSION}#g' \
 		-e '/@SDK_PRE_INSTALL_COMMAND@/d' \
 		-e '/@SDK_POST_INSTALL_COMMAND@/d' \
-		-e 's#@SDK_GCC_VER@#${@oe.utils.host_gcc_version(d)}#g' \
+		-e 's#@SDK_GCC_VER@#${@oe.utils.host_gcc_version(d, taskcontextonly=True)}#g' \
 		${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.sh
 
 	# add execution permission
meta/lib/oe/utils.py +4 -1
index f22a6ab596d..fbda7035f1e 100644
@@ -342,9 +342,12 @@ def format_pkg_list(pkg_dict, ret_format=None):
 
     return '\n'.join(output)
 
-def host_gcc_version(d):
+def host_gcc_version(d, taskcontextonly=False):
     import re, subprocess
 
+    if taskcontextonly and d.getVar('BB_WORKERCONTEXT') != '1':
+        return
+
     compiler = d.getVar("BUILD_CC")
     try:
         env = os.environ.copy()
Signature · 1 line
2.17.1
#8 lore raw

PATCH 6/8 maintainers: Add entry for fribidi

In reply to Richard Purdie
---
 meta/conf/distro/include/maintainers.inc | 1 +
 1 file changed, 1 insertion(+)
meta/conf/distro/include/maintainers.inc +1 -0
index de4f9af47f6..4ec3c20b3bc 100644
@@ -164,6 +164,7 @@ RECIPE_MAINTAINER_pn-fontconfig = "Maxin B. John <[email protected]>"
 RECIPE_MAINTAINER_pn-foomatic-filters = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-formfactor = "Maxin B. John <[email protected]>"
 RECIPE_MAINTAINER_pn-freetype = "Maxin B. John <[email protected]>"
+RECIPE_MAINTAINER_pn-fribidi = "Ross Burton <[email protected]>"
 RECIPE_MAINTAINER_pn-fts = "Khem Raj <[email protected]>"
 RECIPE_MAINTAINER_pn-gawk = "Chen Qi <[email protected]>"
 RECIPE_MAINTAINER_pn-gcc = "Khem Raj <[email protected]>"
Signature · 1 line
2.17.1
#9 lore raw

PATCH 7/8 busybox: Put klogd/syslogd alternative links in syslog package

In reply to Richard Purdie

Currently these are in ${PN} and ${PN}-syslog may get replaced by other packages but update-alternatives would error in the postinst if other files were installed first. Avoid the problems by putting the links in the correct package.

---
 meta/recipes-core/busybox/busybox.inc | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-core/busybox/busybox.inc +4 -1
index 586d5342e60..c26ef56bf78 100644
@@ -367,7 +367,10 @@ python do_package_prepend () {
             # Match coreutils
             if alt_name == '[':
                 alt_name = 'lbracket'
-            d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
+            if alt_name == 'klogd' or alt_name == 'syslogd':
+                d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name)
+            else:
+                d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
             d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name)
             if os.path.exists('%s%s' % (dvar, target)):
                 d.setVarFlag('ALTERNATIVE_TARGET', alt_name, target)
Signature · 1 line
2.17.1
#10 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Hi Richard,

I think my previous patch has covered this case.

git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busybox-alternatives

Chen Qi (1):
busybox: fix the alternatives logic

Is there some problem with the patch?

Best Regards,
Chen Qi

Quoted 26 lines · Richard Purdie

Currently these are in ${PN} and ${PN}-syslog may get replaced by other packages but update-alternatives would error in the postinst if other files were installed first. Avoid the problems by putting the links in the correct package.

---
  meta/recipes-core/busybox/busybox.inc | 5 ++++-
  1 file changed, 4 insertions(+), 1 deletion(-)
meta/recipes-core/busybox/busybox.inc +4 -1
index 586d5342e60..c26ef56bf78 100644
@@ -367,7 +367,10 @@ python do_package_prepend () {
              # Match coreutils
              if alt_name == '[':
                  alt_name = 'lbracket'
-            d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
+            if alt_name == 'klogd' or alt_name == 'syslogd':
+                d.appendVar('ALTERNATIVE_%s-syslog' % (pn), ' ' + alt_name)
+            else:
+                d.appendVar('ALTERNATIVE_%s' % (pn), ' ' + alt_name)
              d.setVarFlag('ALTERNATIVE_LINK_NAME', alt_name, alt_link_name)
              if os.path.exists('%s%s' % (dvar, target)):
                  d.setVarFlag('ALTERNATIVE_TARGET', alt_name, target)
#11 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Quoted 8 lines · ChenQi

Hi Richard,

I think my previous patch has covered this case.

git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
ox-alternatives

Quoted 5 lines

Chen Qi (1):
busybox: fix the alternatives logic

Is there some problem with the patch?

Sorry, I thought I'd replied about this. There were build failures from this patch and I had to drop it from the test branch. I'm afraid with all the build failures we're had recently I'm struggling to find which failures those were...

I merged my simpler but less complete fix simply to get patches moving as I'd assumed this one was being debugged. I guess we need the failures for that though...

Cheers,

Richard

#12 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Quoted 19 lines · On 09/06/2018 04:19 PM, [email protected] wrote:
Quoted 8 lines · ChenQi

Hi Richard,

I think my previous patch has covered this case.

git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
ox-alternatives

Quoted 5 lines

Chen Qi (1):
busybox: fix the alternatives logic

Is there some problem with the patch?

Sorry, I thought I'd replied about this. There were build failures from this patch and I had to drop it from the test branch. I'm afraid with all the build failures we're had recently I'm struggling to find which failures those were...

Quoted 8 lines

I merged my simpler but less complete fix simply to get patches moving as I'd assumed this one was being debugged. I guess we need the failures for that though...

Cheers,

Richard

Got it.
I just looked at some build failures, the ones that might be related are those of testimage failures complaining 'No syslog daemon'. I'll check things locally first.

Best Regards,
Chen Qi

#13 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Quoted 27 lines · ChenQi
Quoted 26 lines · On 09/06/2018 04:19 PM, [email protected] wrote:
Quoted 8 lines · ChenQi

Hi Richard,

I think my previous patch has covered this case.

git://git.pokylinux.org/poky-contrib ChenQi/busybox-alternatives
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/busyb
ox-alternatives

Quoted 5 lines

Chen Qi (1):
busybox: fix the alternatives logic

Is there some problem with the patch?

Sorry, I thought I'd replied about this. There were build failures from this patch and I had to drop it from the test branch. I'm afraid with all the build failures we're had recently I'm struggling to find which failures those were...

I merged my simpler but less complete fix simply to get patches moving as I'd assumed this one was being debugged. I guess we need the failures for that though...

Cheers,

Quoted 7 lines
Quoted 2 lines

Richard

Got it.
I just looked at some build failures, the ones that might be related are those of testimage failures complaining 'No syslog daemon'. I'll check things locally first.

It seems that we have problems in master even with my version of the patch. Its not a deterministic problem, it depends upon rpm/dnf package installation order.

The dnf.DnfRepoTest.test_dnf_installroot image test does:

dnf --repofrompath=oe-testimage-repo-i586,http://192.168.7.7:36895/i586 --repofrompath=oe-testimage-repo-qemux86,http://192.168.7.7:36895/qemux86 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:36895/noarch --nogpgcheck install --installroot=/home/root/chroot/test -v -y --rpmverbosity=debug busybox run-postinst

which figures out:

================================================================================

|  Package                   Arch    Version      Repository                 Size
| ================================================================================
| Installing:
|  busybox                   i586    1.29.2-r0.0  oe-testimage-repo-i586    352 k
|  run-postinsts             noarch  1.0-r10.0    oe-testimage-repo-noarch  8.7 k
| Installing dependencies:
|  libc6                     i586    2.28-r0.0    oe-testimage-repo-i586    1.3 M
|  update-alternatives-opkg  i586    0.3.6-r0.0   oe-testimage-repo-i586    8.5 k
| Installing weak dependencies:
|  busybox-syslog            i586    1.29.2-r0.0  oe-testimage-repo-i586     10 k
|  busybox-udhcpc            i586    1.29.2-r0.0  oe-testimage-repo-i586    8.1 k
| 

so far so good. It then tries to install busybox-syslog *before* busybox. During the postinstall of busybox-syslog we see:

update-alternatives --install /sbin/klogd klogd /bin/busybox.nosuid 50 | /usr/bin/update-alternatives: line 145: sed: command not found | /usr/bin/update-alternatives: line 145: echo: write error: Broken pipe

which makes sense as sed comes from busybox which hasn't installed yet.

I added:

RDEPENDS_${PN}-syslog = "busybox"

to the busybox recipe but this doesn't seem to help. I see in the busybox.spec file for the -syslog package:

Requires: /bin/sh
Requires: busybox
Requires: update-alternatives-opkg
Requires(post): /bin/sh
Requires(post): busybox
Requires(post): update-alternatives-opkg

but it still tries to install busybox-syslog *before* busybox. (busybox only RRECOMMENDS busybox-syslog).

The autobuilder with the above RDEPENDS added also shows this.

Interestingly this test doesn't fail, its the parselogs test later which fails due to the postinstall errors being detected in the dnf log files.

This would seem to be an rpm/dnf bug :(.

Sometimes we don't see the error as if it chooses to install busybox first, things work ok. THis is currently breaking builds quite badly on master :(.

Cheers,

Richard

#14 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Just to follow up, I'm trying separating out busybox-syslog's recommends from busybox into the higher level packagegroups. The patch is in -next and will run tests overnight.

Doing this is probably good anyway but it does feel like we're working around an rpm/dnf issue. I'm open to better ideas...

Cheers,

Richard

#15 lore raw

Re: [PATCH 7/8] busybox: Put klogd/syslogd alternative links in syslog package

Quoted 6 lines · Richard Purdie

Just to follow up, I'm trying separating out busybox-syslog's recommends from busybox into the higher level packagegroups. The patch is in -next and will run tests overnight.

Doing this is probably good anyway but it does feel like we're working around an rpm/dnf issue. I'm open to better ideas...

Circular dependencies can result in non-deterministic behavior. RPM does not have any difference in dependency resolution (ordering) for depends vs recommends. (The only difference is a recommend is not an error.) Due to this, a circular dependency will result in behavior that may change the order of circular dependent items based on external factors such as file list order, date codes or ???

RPM does (or at least used to) permit you to specify a way to break circular dependencies. But this was a static list of known circular dependencies which you would simply specify in the macros file which had to be installed first.

The 'correct' way to break this is only specify ACTUAL dependencies, and realistically there should never be a circular dependency based on a pre-post install action. Otherwise, expect non-deterministic breakage.

There is also a way to explicitly set dependencies for the install scriptlets themselves, but I don't know how this influences the ordering of the
installation. But this would likely require some special syntax for busybox
(or other early load environments) which likely would be worse then a package group.

(I don't know how or if DNF handles this differently BTW.)

--Mark

#16 lore raw

PATCH 8/8 busybox: Sync arch and link locations with coreutils

In reply to Richard Purdie

coreutils installs these into bindir, folllow its lead to avoid postinst failures when installing coreutils and busybox together.

---
 meta/recipes-core/busybox/busybox.inc | 4 ++++
 1 file changed, 4 insertions(+)
meta/recipes-core/busybox/busybox.inc +4 -0
index c26ef56bf78..c02e0623e37 100644
@@ -208,6 +208,10 @@ do_install () {
 	sed -i "s:^/usr/bin/:BINDIR/:" busybox.links*
 	sed -i "s:^/usr/sbin/:SBINDIR/:" busybox.links*
 
+	# Move arch/link to BINDIR to match coreutils
+	sed -i "s:^BASE_BINDIR/arch:BINDIR/arch:" busybox.links*
+	sed -i "s:^BASE_BINDIR/link:BINDIR/link:" busybox.links*
+
 	sed -i "s:^BASE_BINDIR/:${base_bindir}/:" busybox.links*
 	sed -i "s:^BASE_SBINDIR/:${base_sbindir}/:" busybox.links*
 	sed -i "s:^BINDIR/:${bindir}/:" busybox.links*
Signature · 1 line
2.17.1
#17 lore raw
In reply to Richard Purdie

Hi RP,

Quoted 16 lines · Richard Purdie

From: Ross Burton <[email protected]>

There is no good reason to loop 10 times on the test since the profile optimisation won't change between runs, we don't need/want a statistical average. This is just burning cpu cycles, just make 1 run.

---
  meta/recipes-devtools/python/python3_3.5.5.bb | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-devtools/python/python3_3.5.5.bb +0 -0
index bfe91fb1c71..8afe15eee6a 100644

meta/recipes-devtools/python/python3_3.5.5.bb TARGET_CC_ARCH += "-DNDEBUG -fno-inline" · line 77

Quoted 5 lines
meta/recipes-devtools/python/python3_3.5.5.bb +1 -1
@@ -74,7 +74,7 @@ TARGET_CC_ARCH += "-DNDEBUG -fno-inline"
  SDK_CC_ARCH += "-DNDEBUG -fno-inline"
  EXTRA_OEMAKE += "CROSS_COMPILE=yes"
  EXTRA_OECONF += "CROSSPYTHONPATH=${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dynload/ --without-ensurepip --enable-optimizations"
-PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 10"
+PYTHON3_PROFILE_TASK ?= "${S}/Tools/pybench/pybench.py -n 1"

What's the fault if I don't run it, please ? This is very slow, here are some rough thoughts to speed it up:

1) Make it optional to run, and default to no if there isn't any faults without running it.

2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried it, it can reduce form 12m -> 4m, saved 8 mins:
* Without kvm

     $ bitbake python3 -ccleansstate && time bitbake python3
     real    12m28.070s
     user    0m1.316s
     sys     0m0.184s

* With kvm:

   real    3m53.781s
   user    0m1.080s
   sys     0m0.220s

We may need a oe-core/scripts/qemu-kvm-check to check strictly whether kvm can be used or not.

Please feel free to give your comments.

// Robert

Quoted 4 lines
meta/recipes-devtools/python/python3_3.5.5.bb +0 -0
  export CROSS_COMPILE = "${TARGET_PREFIX}"
  export _PYTHON_PROJECT_BASE = "${B}"
 
#18 lore raw
Quoted 5 lines · Robert Yang

What's the fault if I don't run it, please ? This is very slow, here are some rough thoughts to speed it up:

1) Make it optional to run, and default to no if there isn't any faults without running it.

Without running it, python is not profile optimised and runs slower.

Quoted 14 lines

2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried it, it can

    reduce form 12m -> 4m, saved 8 mins:
    * Without kvm
      $ bitbake python3 -ccleansstate && time bitbake python3
      real    12m28.070s
      user    0m1.316s
      sys     0m0.184s

    * With kvm:
    real    3m53.781s
    user    0m1.080s
    sys     0m0.220s
Quoted 5 lines

We may need a oe-core/scripts/qemu-kvm-check to check strictly
whether kvm
can be used or not.

Please feel free to give your comments.

Ross has some patches coming in this area which should improve things a lot so I'd suggest we wait for those. I merged the n10 -> n1 change simply because it was an easy fix to same some time whilst the other patches a debugged.

Cheers,

Richard

#19 lore raw
Quoted 27 lines · On 09/06/2018 04:17 PM, [email protected] wrote:
Quoted 5 lines · Robert Yang

What's the fault if I don't run it, please ? This is very slow, here are some rough thoughts to speed it up:

1) Make it optional to run, and default to no if there isn't any faults without running it.

Without running it, python is not profile optimised and runs slower.

Quoted 18 lines

2) Use qemu-kvm to run the tests if qemu kvm has been setup (I tried it, it can

     reduce form 12m -> 4m, saved 8 mins:
     * Without kvm
       $ bitbake python3 -ccleansstate && time bitbake python3
       real    12m28.070s
       user    0m1.316s
       sys     0m0.184s

     * With kvm:
     real    3m53.781s
     user    0m1.080s
     sys     0m0.220s

    We may need a oe-core/scripts/qemu-kvm-check to check strictly

whether kvm
can be used or not.

Quoted 6 lines
Quoted 1 line

Please feel free to give your comments.

Ross has some patches coming in this area which should improve things a lot so I'd suggest we wait for those. I merged the n10 -> n1 change simply because it was an easy fix to same some time whilst the other patches a debugged.

Sounds great, thanks.

// Robert

Quoted 5 lines

Cheers,

Richard