Skip to content

Packaging performance improvements, round 2

Richard Purdie openembedded-core 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest22 patches 4 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/22package.bbclass: Multithread per file dependency generation codemeta/classes/base.bbclass meta/classes/package.bbclass meta/classes/update-alternatives.bbclass +1 more+85 -620 replies
2/22package: Don't export PATHmeta/classes/package.bbclass+5 -90 replies
3/22package: Process package stripping in parallelmeta/classes/package.bbclass meta/lib/oe/package.py+58 -451 reply
4/22insane.bbclass: Add pkgvarcheck to check for suboptimal usages of variablesmeta/classes/insane.bbclass+9 -10 replies
5/22insane.bbclass: Add documentation headers for logical code blocksmeta/classes/insane.bbclass+17 -00 replies
6/22staging/insane.bbclass: Move legacy do_stage check iinto insane.bbclassmeta/classes/insane.bbclass meta/classes/staging.bbclass+3 -50 replies
7/22staging.bbclass: Drop unused/legacy functionmeta/classes/staging.bbclass+0 -40 replies
8/22update-rc.d: Drop OVERRIDES codemeta/classes/update-rc.d_real.bbclass+6 -110 replies
9/22qemu: Set RDEPENDS on the specific package that needs itmeta/recipes-devtools/qemu/qemu.inc+1 -10 replies
10/22gdb-cross-canadian: Set RDEPENDS on the specific package that needs itmeta/recipes-devtools/gdb/gdb-cross-canadian.inc+2 -20 replies
11/22initramfs-live-boot: Set RDEPENDS on the specific package that needs itmeta/recipes-core/initrdscripts/initramfs-live-boot_1.0.bb+1 -10 replies
12/22package.bbclass: Fix up bb.mkdirhier/bb.copyfile usagemeta/classes/package.bbclass+7 -70 replies
13/22package.bbclass: Rewrite split_and_strip_filesmeta/classes/package.bbclass+97 -1082 replies
14/22kernel.bbclass: Improve populate_packages_prependmeta/classes/kernel.bbclass+21 -270 replies
15/22package.bbclass: Make use of cleandirs and dirs function flagsmeta/classes/package.bbclass+3 -50 replies
16/22package.bbclass: Various minor performance tweaksmeta/classes/package.bbclass+6 -80 replies
17/22package.bbclass: Simplify empty directory removalmeta/classes/package.bbclass+1 -50 replies
18/22package.bbclass: Add PACKAGESPLITFUNCS variablemeta/classes/kernel.bbclass meta/classes/package.bbclass meta/classes/update-alternatives.bbclass +1 more+15 -30 replies
19/22sstate/path.py: Add copyhardlinktree() function and use for performance optimisationmeta/classes/sstate.bbclass meta/lib/oe/path.py+6 -20 replies
20/22package.bbclass: Better document the different phases of operationmeta/classes/package.bbclass+16 -00 replies
21/22package.bbclass: Pre-expand some variables to save timemeta/classes/package.bbclass+14 -00 replies
22/22classes: Drop none package specific packaging variable accessesmeta/classes/gconf.bbclass meta/classes/gtk-icon-cache.bbclass meta/classes/gtk-immodules-cache.bbclass +4 more+23 -310 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 00/22] Packaging performance improvements, round 2

The following patch series contains various performance improvements for do_package.

It also includes changes to the sanity testing to start giving warnings about packaging variables used without packages being specified, eg:

RDEPENDS = "x"

instead of

RDEPENDS_${PN} = "x"

The final patch in the series stops the packaging code using the non-package versions in many cases. I'm not planning to merge that one right away, its just included for illustration and benchmarking purposes now.

There were three cases I found using bad syntax in OE-Core and the series includes patches to fix them.

There is also a useful performance improvement to sstate manipulations to use hardlinks instead of file copies which should be universally helpful to build performance of tasks.

The following changes are available in the git repository at:

git://git.openembedded.org/openembedded-core master-next
http://cgit.openembedded.org/cgit.cgi/openembedded-core/log/?h=master-next

Richard Purdie (22):
package.bbclass: Multithread per file dependency generation code
package: Don't export PATH
package: Process package stripping in parallel insane.bbclass: Add pkgvarcheck to check for suboptimal usages of
variables
insane.bbclass: Add documentation headers for logical code blocks staging/insane.bbclass: Move legacy do_stage check iinto
insane.bbclass
staging.bbclass: Drop unused/legacy function
update-rc.d: Drop OVERRIDES code
qemu: Set RDEPENDS on the specific package that needs it
gdb-cross-canadian: Set RDEPENDS on the specific package that needs
it
initramfs-live-boot: Set RDEPENDS on the specific package that needs
it
package.bbclass: Fix up bb.mkdirhier/bb.copyfile usage package.bbclass: Rewrite split_and_strip_files kernel.bbclass: Improve populate_packages_prepend package.bbclass: Make use of cleandirs and dirs function flags package.bbclass: Various minor performance tweaks package.bbclass: Simplify empty directory removal package.bbclass: Add PACKAGESPLITFUNCS variable sstate/path.py: Add copyhardlinktree() function and use for
performance optimisation
package.bbclass: Better document the different phases of operation package.bbclass: Pre-expand some variables to save time
classes: Drop none package specific packaging variable accesses

 meta/classes/base.bbclass                          |    2 +-
 meta/classes/gconf.bbclass                         |    9 +-
 meta/classes/gtk-icon-cache.bbclass                |    4 +-
 meta/classes/gtk-immodules-cache.bbclass           |    4 +-
 meta/classes/insane.bbclass                        |   30 +-
 meta/classes/kernel.bbclass                        |   52 +--
 meta/classes/mime.bbclass                          |    8 +-
 meta/classes/package.bbclass                       |  460 +++++++++-----------
 meta/classes/sstate.bbclass                        |    4 +-
 meta/classes/staging.bbclass                       |    9 -
 meta/classes/update-alternatives.bbclass           |   10 +-
 meta/classes/update-rc.d_real.bbclass              |   21 +-
 meta/lib/oe/package.py                             |   96 ++++
 meta/lib/oe/path.py                                |    4 +
 .../initrdscripts/initramfs-live-boot_1.0.bb       |    2 +-
 meta/recipes-devtools/gdb/gdb-cross-canadian.inc   |    4 +-
 meta/recipes-devtools/qemu/qemu.inc                |    2 +-
 17 files changed, 390 insertions(+), 331 deletions(-)
 create mode 100644 meta/lib/oe/package.py
Signature · 1 line
1.7.10.4