Skip to content

rootfs generation improvements v2

Paul Eggleton openembedded-core 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest6 patches 1 replyIn review

Changes since v1 of this series from the cover letter

* The way packages were mapped to their dev equivalents was extremely naive. We now use pkgdata to map back to the actual dev package, which handles debian renaming properly (e.g. the zlib recipe produces a main package called libz1, whose corresponding dev package is libz-dev). This relies upon Andrei Gerzhan's pending fix to add symlinks to do reverse pkgdata lookups, which has been updated (to use a separate directory, always create the symlink even if the name is the same, and force the symlink so that it can be re-run without errors) and has been included in this series.
* Rework the rpm backend to use a separate C program to do the resolution of package name to package file, and consolidate the package list processing code.
* Improve performance of buildhistory data collection, using python helper script (ipk/deb backends) and the aforementioned C program (rpm backend) to avoid lots of slow sed/grep operations. As an example, a build of core-image-minimal-dev with the rpm backend went from about 13 mins to about 5 mins, so less than half the time.

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6package.bbclass: Create symlinks for packages with different packaged namemeta/classes/license.bbclass meta/classes/package.bbclass+6 -20 replies
2/6classes/rootfs_rpm: improve speed of RPM rootfs constructionmeta/classes/package_rpm.bbclass meta/classes/rootfs_rpm.bbclass meta/recipes-devtools/rpm/rpmresolve/rpmresolve.c +1 more+341 -640 replies
3/6classes/package_deb: add PackageArch fieldmeta/classes/package_deb.bbclass+1 -00 replies
4/6Rework installation of dev, dbg, doc, and locale packagesmeta/classes/image.bbclass meta/classes/package_rpm.bbclass meta/classes/populate_sdk_deb.bbclass +7 more+386 -1100 replies
5/6classes/image: add staticdev-pkgs IMAGE_FEATURES featuremeta/classes/image.bbclass+3 -10 replies
6/6buildhistory: improve performance of image info collectionmeta/classes/buildhistory.bbclass meta/classes/rootfs_deb.bbclass meta/classes/rootfs_ipk.bbclass +2 more+162 -1010 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH 0/6] rootfs generation improvements v2

Fix up handling of "complementary" package installation (dev, dbg, doc, etc.) and introduce a staticdev-pkgs IMAGE_FEATURES feature.

I've tested this with all backends using buildhistory to examine the changes to the image contents, and compared the lists of files in the SDK-from-image output with the rpm backend.

Changes since v1 of this series:

* The way packages were mapped to their dev equivalents was extremely naive. We now use pkgdata to map back to the actual dev package, which handles debian renaming properly (e.g. the zlib recipe produces a main package called libz1, whose corresponding dev package is libz-dev). This relies upon Andrei Gerzhan's pending fix to add symlinks to do reverse pkgdata lookups, which has been updated (to use a separate directory, always create the symlink even if the name is the same, and force the symlink so that it can be re-run without errors) and has been included in this series.
* Rework the rpm backend to use a separate C program to do the resolution of package name to package file, and consolidate the package list processing code.
* Improve performance of buildhistory data collection, using python helper script (ipk/deb backends) and the aforementioned C program (rpm backend) to avoid lots of slow sed/grep operations. As an example, a build of core-image-minimal-dev with the rpm backend went from about 13 mins to about 5 mins, so less than half the time.

The latter two changes seem unrelated, but due to us triggering some kind of environment size limit with the rpm backend, a rethink of how that code works was required, and this is the result. Overall, if buildhistory is not enabled, with the rpm backend do_rootfs actually takes a little bit longer than it did previously, however without the work on performance I believe the impact would have been much greater.

Note: building core-image-lsb-sdk with these changes uncovered an issue with the dhcp recipe packaging (patch already submitted to fix this), and another issue with locale packages (bug 2815 filed). Additionally it emphasises how the greedy dev/dbg package dependencies drag in a lot of other seemingly extraneous packages; this will be worked on separately (see bug 2599).

The following changes since commit da1de01aa74da0f38f07af523a9861824f7b0c95:

Pyphon-native: Fix typo (2012-07-22 14:20:26 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/rootfs2
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/rootfs2

Andrei Gherzan (1):
package.bbclass: Create symlinks for packages with different packaged
name

Paul Eggleton (5):
classes/rootfs_rpm: improve speed of RPM rootfs construction classes/package_deb: add PackageArch field
Rework installation of dev, dbg, doc, and locale packages classes/image: add staticdev-pkgs IMAGE_FEATURES feature
buildhistory: improve performance of image info collection

 meta/classes/buildhistory.bbclass                 |   51 ++-
 meta/classes/image.bbclass                        |   88 ++---
 meta/classes/license.bbclass                      |    2 +-
 meta/classes/package.bbclass                      |    6 +-
 meta/classes/package_deb.bbclass                  |    1 +
 meta/classes/package_rpm.bbclass                  |  183 +++++-----
 meta/classes/populate_sdk_deb.bbclass             |    2 +
 meta/classes/populate_sdk_ipk.bbclass             |    2 +
 meta/classes/populate_sdk_rpm.bbclass             |    3 +
 meta/classes/rootfs_deb.bbclass                   |   51 ++-
 meta/classes/rootfs_ipk.bbclass                   |   50 ++-
 meta/classes/rootfs_rpm.bbclass                   |   69 ++--
 meta/recipes-devtools/rpm/rpmresolve/rpmresolve.c |  376 +++++++++++++++++++++
 meta/recipes-devtools/rpm/rpmresolve_1.0.bb       |   22 ++
 scripts/oe-pkgdata-util                           |  167 +++++++++
 scripts/opkg-query-helper.py                      |   76 +++++
 16 files changed, 885 insertions(+), 264 deletions(-)
 create mode 100644 meta/recipes-devtools/rpm/rpmresolve/rpmresolve.c
 create mode 100644 meta/recipes-devtools/rpm/rpmresolve_1.0.bb
 create mode 100755 scripts/oe-pkgdata-util
 create mode 100755 scripts/opkg-query-helper.py
Signature · 1 line
1.7.9.5