Skip to contentSkip to conversation

Add package managers as a plugin

PATCH In review Fredrik Gustafsson 6 participants 13 messages Jun 23, 2020 – Jun 30, 2020

lore
Share & tools
lore
#0 lore raw

Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

#1 lore raw

PATCH 1/2 nopackages.bbclass: Move to nopackages_base.bbclass

When for example adding an other package mananger a two rows deltask do_package_write_PKGMGR
deltask do_package_write_PKGMGR_setscene
is needed to be added to nopackages.bbclass. When adding the new package manager in a new layer in order to extend poky (rather than forking it) the whole nopackages.bbclass needs to be copied which is a maintenance problem. Moving nopackages.bbclass to nopackages.bbclass would make the new layer just adding an image.bbclass containing:
inherit nopackages_base.bbclass
deltask do_package_write_PKGMGR
deltask do_package_write_PKGMGR_setscene

Which is much easier to maintain.

---
 meta/classes/nopackages.bbclass      | 15 +++------------
 meta/classes/nopackages_base.bbclass | 12 ++++++++++++
 2 files changed, 15 insertions(+), 12 deletions(-)
 create mode 100644 meta/classes/nopackages_base.bbclass
meta/classes/nopackages.bbclass +3 -12
index 559f5078bd..78bbd8c85c 100644
@@ -1,12 +1,3 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+# This file should only inherit nopackages_base. This is to make
+# this file easy to extend from an other layer.
+inherit nopackages_base
meta/classes/nopackages_base.bbclass new file +12 -0
new file mode 100644
index 0000000000..e898cef00f
@@ -0,0 +1,12 @@
+deltask do_package
+deltask do_package_qa
+deltask do_packagedata
+deltask do_package_setscene
+deltask do_package_qa_setscene
+deltask do_packagedata_setscene
+deltask do_package_write_rpm
+deltask do_package_write_deb
+deltask do_package_write_ipk
+deltask do_package_write_rpm_setscene
+deltask do_package_write_deb_setscene
+deltask do_package_write_ipk_setscene
Signature · 1 line
2.20.1
#2 lore raw

Re: [OE-core] [PATCH 1/2] nopackages.bbclass: Move to nopackages_base.bbclass

Quoted 16 lines · Fredrik Gustafsson

When for example adding an other package mananger a two rows deltask do_package_write_PKGMGR
deltask do_package_write_PKGMGR_setscene
is needed to be added to nopackages.bbclass. When adding the new package
manager in a new layer in order to extend poky (rather than forking it)
the whole nopackages.bbclass needs to be copied which is a maintenance
problem. Moving nopackages.bbclass to nopackages.bbclass would make the
new layer just adding an image.bbclass containing:
inherit nopackages_base.bbclass
deltask do_package_write_PKGMGR
deltask do_package_write_PKGMGR_setscene

Quoted 8 lines

Which is much easier to maintain.

---
 meta/classes/nopackages.bbclass      | 15 +++------------
 meta/classes/nopackages_base.bbclass | 12 ++++++++++++
 2 files changed, 15 insertions(+), 12 deletions(-)
 create mode 100644 meta/classes/nopackages_base.bbclass

I personally think "_base" bbclasses are not helpful to making our code easier to understand and I'd prefer to see fewer of them, not more.

I can't remember if deltask support multiple parameters but if not, it can be made to. That would then allow this to be configured with a variable. I'd suggest we explore other options along those lines for this.

Cheers,

Richard

#3 lore raw

PATCH 2/2 lib/oe: Split package manager code to multiple files

In reply to Fredrik Gustafsson

Today OE-Core has support for three package managers, even if there are many more package managers that could be interesting to use. Adding a new package manager to OE-Core would increase the test matrix significantly. In order to let users use their favorite package manager without need for it to have support from upstream, this patch refactors the package manager code so that it's easy to add a new package manager in another layer.

This patch is mostly moving code and duplicating some code (because of the hard coupling between deb and ipk).

---
 meta/classes/populate_sdk_base.bbclass        |   11 +-
 meta/lib/oe/manifest.py                       |  149 +-
 meta/lib/oe/package_manager.py                | 1402 +----------------
 meta/lib/oe/package_managers/deb/manifest.py  |   31 +
 .../package_managers/deb/package_manager.py   |  719 +++++++++
 meta/lib/oe/package_managers/deb/rootfs.py    |  213 +++
 meta/lib/oe/package_managers/deb/sdk.py       |   97 ++
 .../ipk/.package_manager.py.swo               |  Bin 0 -> 45056 bytes
 meta/lib/oe/package_managers/ipk/manifest.py  |   79 +
 .../package_managers/ipk/package_manager.py   |  588 +++++++
 meta/lib/oe/package_managers/ipk/rootfs.py    |  395 +++++
 meta/lib/oe/package_managers/ipk/sdk.py       |  104 ++
 meta/lib/oe/package_managers/rpm/manifest.py  |   62 +
 .../package_managers/rpm/package_manager.py   |  423 +++++
 meta/lib/oe/package_managers/rpm/rootfs.py    |  154 ++
 meta/lib/oe/package_managers/rpm/sdk.py       |  104 ++
 meta/lib/oe/rootfs.py                         |  631 +-------
 meta/lib/oe/sdk.py                            |  311 +---
 meta/lib/oeqa/utils/package_manager.py        |   29 +-
 meta/recipes-core/meta/package-index.bb       |    2 +-
 20 files changed, 3014 insertions(+), 2490 deletions(-)
 create mode 100644 meta/lib/oe/package_managers/deb/manifest.py
 create mode 100644 meta/lib/oe/package_managers/deb/package_manager.py
 create mode 100644 meta/lib/oe/package_managers/deb/rootfs.py
 create mode 100644 meta/lib/oe/package_managers/deb/sdk.py
 create mode 100644 meta/lib/oe/package_managers/ipk/.package_manager.py.swo
 create mode 100644 meta/lib/oe/package_managers/ipk/manifest.py
 create mode 100644 meta/lib/oe/package_managers/ipk/package_manager.py
 create mode 100644 meta/lib/oe/package_managers/ipk/rootfs.py
 create mode 100644 meta/lib/oe/package_managers/ipk/sdk.py
 create mode 100644 meta/lib/oe/package_managers/rpm/manifest.py
 create mode 100644 meta/lib/oe/package_managers/rpm/package_manager.py
 create mode 100644 meta/lib/oe/package_managers/rpm/rootfs.py
 create mode 100644 meta/lib/oe/package_managers/rpm/sdk.py
meta/classes/populate_sdk_base.bbclass +10 -1
index 990505e89b..d31cda4714 100644
@@ -330,9 +330,18 @@ do_populate_sdk[vardeps] += "${@sdk_variables(d)}"
 do_populate_sdk[file-checksums] += "${TOOLCHAIN_SHAR_REL_TMPL}:True \
                                     ${TOOLCHAIN_SHAR_EXT_TMPL}:True"
 
+def get_package_write_tasks(d):
+    package_managers = d.getVar('PACKAGE_CLASSES').split()
+    tasks = ''
+    for package_manager in package_managers:
+        package_manager = package_manager.replace('package_', '')
+        tasks += ' do_package_write_' + package_manager
+    return tasks
+
 do_populate_sdk[dirs] = "${PKGDATA_DIR} ${TOPDIR}"
 do_populate_sdk[depends] += "${@' '.join([x + ':do_populate_sysroot' for x in d.getVar('SDK_DEPENDS').split()])}  ${@d.getVarFlag('do_rootfs', 'depends', False)}"
 do_populate_sdk[rdepends] = "${@' '.join([x + ':do_package_write_${IMAGE_PKGTYPE} ' + x + ':do_packagedata' for x in d.getVar('SDK_RDEPENDS').split()])}"
-do_populate_sdk[recrdeptask] += "do_packagedata do_package_write_rpm do_package_write_ipk do_package_write_deb"
+do_populate_sdk[recrdeptask] += "do_packagedata "
+do_populate_sdk[recrdeptask] += "${@get_package_write_tasks(d)}"
 do_populate_sdk[file-checksums] += "${POSTINST_INTERCEPT_CHECKSUMS}"
 addtask populate_sdk
meta/lib/oe/manifest.py +2 -147

157 lines not on this page · raw message

meta/lib/oe/package_manager.py +4 -1398

1,424 lines not on this page · raw message

meta/lib/oe/package_managers/deb/manifest.py new file +31 -0
new file mode 100644
index 0000000000..077db13f1c
@@ -0,0 +1,31 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from abc import ABCMeta, abstractmethod
+import os
+import re
+import bb
+from oe.manifest import *
+
+class PkgManifest(Manifest):
+    def create_initial(self):
+        with open(self.initial_manifest, "w+") as manifest:
+            manifest.write(self.initial_manifest_file_header)
+
+            for var in self.var_maps[self.manifest_type]:
+                pkg_list = self.d.getVar(var)
+
+                if pkg_list is None:
+                    continue
+
+                for pkg in pkg_list.split():
+                    manifest.write("%s,%s\n" %
+                                   (self.var_maps[self.manifest_type][var], pkg))
+
+    def create_final(self):
+        pass
+
+    def create_full(self, pm):
+        pass
+
meta/lib/oe/package_managers/deb/package_manager.py new file +719 -0

719 lines not on this page · raw message

meta/lib/oe/package_managers/deb/rootfs.py new file +213 -0

213 lines not on this page · raw message

meta/lib/oe/package_managers/deb/sdk.py new file +97 -0

97 lines not on this page · raw message

meta/lib/oe/package_managers/ipk/.package_manager.py.swo new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..e75431e2caaac37152ef454e776d8a9d53ead35e
GIT binary patch
meta/lib/oe/package_managers/ipk/manifest.py new file +79 -0

79 lines not on this page · raw message

meta/lib/oe/package_managers/ipk/package_manager.py new file +588 -0

261 lines not on this page · raw message

meta/lib/oe/package_managers/ipk/rootfs.py new file +395 -0

395 lines not on this page · raw message

meta/lib/oe/package_managers/ipk/sdk.py new file +104 -0

104 lines not on this page · raw message

meta/lib/oe/package_managers/rpm/manifest.py new file +62 -0

62 lines not on this page · raw message

meta/lib/oe/package_managers/rpm/package_manager.py new file +423 -0

423 lines not on this page · raw message

meta/lib/oe/package_managers/rpm/rootfs.py new file +154 -0

154 lines not on this page · raw message

meta/lib/oe/package_managers/rpm/sdk.py new file +104 -0

104 lines not on this page · raw message

meta/lib/oe/rootfs.py +9 -622

631 lines not on this page · raw message

meta/lib/oe/sdk.py +15 -296

311 lines not on this page · raw message

meta/lib/oeqa/utils/package_manager.py +4 -25
index 2d358f7172..8e31d148c7 100644

Lines of this file are not shown. View the raw message.

meta/recipes-core/meta/package-index.bb +1 -1
index 98c5bcb372..4b063f4298 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.20.1

Part of the diff is not shown here. View the raw message.

#4 lore raw

Re: [OE-core] [PATCH 2/2] lib/oe: Split package manager code to multiple files

Quoted 26 lines · Fredrik Gustafsson

Today OE-Core has support for three package managers, even if there are many more package managers that could be interesting to use. Adding a new package manager to OE-Core would increase the test matrix significantly. In order to let users use their favorite package manager without need for it to have support from upstream, this patch refactors the package manager code so that it's easy to add a new package manager in another layer.

This patch is mostly moving code and duplicating some code (because of the hard coupling between deb and ipk).

---
 meta/classes/populate_sdk_base.bbclass        |   11 +-
 meta/lib/oe/manifest.py                       |  149 +-
 meta/lib/oe/package_manager.py                | 1402 +----------------
 meta/lib/oe/package_managers/deb/manifest.py  |   31 +
 .../package_managers/deb/package_manager.py   |  719 +++++++++
 meta/lib/oe/package_managers/deb/rootfs.py    |  213 +++
 meta/lib/oe/package_managers/deb/sdk.py       |   97 ++
 .../ipk/.package_manager.py.swo               |  Bin 0 -> 45056 bytes
 meta/lib/oe/package_managers/ipk/manifest.py  |   79 +
 .../package_managers/ipk/package_manager.py   |  588 +++++++
 meta/lib/oe/package_managers/ipk/rootfs.py    |  395 +++++
 meta/lib/oe/package_managers/ipk/sdk.py       |  104 ++
 meta/lib/oe/package_managers/rpm/manifest.py  |   62 +
Quoted 8 lines
 .../package_managers/rpm/package_manager.py   |  423 +++++
 meta/lib/oe/package_managers/rpm/rootfs.py    |  154 ++
 meta/lib/oe/package_managers/rpm/sdk.py       |  104 ++
 meta/lib/oe/rootfs.py                         |  631 +-------
 meta/lib/oe/sdk.py                            |  311 +---
 meta/lib/oeqa/utils/package_manager.py        |   29 +-
 meta/recipes-core/meta/package-index.bb       |    2 +-
 20 files changed, 3014 insertions(+), 2490 deletions(-)

I'm afraid this patch is basically impossible to review. There are warning signs like the inclusion of the binary swo file. The hint that code gets duplicated also worries me, even if you abstract this, there shouldn't be reason to do that.

If we do this its going to have to be a more granular series where the patches *just* move code to new locations so we can see/understand what other changes are being made as I simply don't trust this patch, sorry :(.

Cheers,

Richard

#5 lore raw

Re: [OE-core] [PATCH 2/2] lib/oe: Split package manager code to multiple files

________________________________________
From: Richard Purdie <[email protected]>
Sent: Tuesday, June 23, 2020 2:02 PM
To: Fredrik Gustafsson; [email protected]
Cc: tools-cfpbuild-internal; Hugo Cedervall; Fredrik Gustafsson
Subject: Re: [OE-core] [PATCH 2/2] lib/oe: Split package manager code to multiple files

Quoted 26 lines · Fredrik Gustafsson

Today OE-Core has support for three package managers, even if there are many more package managers that could be interesting to use. Adding a new package manager to OE-Core would increase the test matrix significantly. In order to let users use their favorite package manager without need for it to have support from upstream, this patch refactors the package manager code so that it's easy to add a new package manager in another layer.

This patch is mostly moving code and duplicating some code (because of the hard coupling between deb and ipk).

---
 meta/classes/populate_sdk_base.bbclass        |   11 +-
 meta/lib/oe/manifest.py                       |  149 +-
 meta/lib/oe/package_manager.py                | 1402 +----------------
 meta/lib/oe/package_managers/deb/manifest.py  |   31 +
 .../package_managers/deb/package_manager.py   |  719 +++++++++
 meta/lib/oe/package_managers/deb/rootfs.py    |  213 +++
 meta/lib/oe/package_managers/deb/sdk.py       |   97 ++
 .../ipk/.package_manager.py.swo               |  Bin 0 -> 45056 bytes
 meta/lib/oe/package_managers/ipk/manifest.py  |   79 +
 .../package_managers/ipk/package_manager.py   |  588 +++++++
 meta/lib/oe/package_managers/ipk/rootfs.py    |  395 +++++
 meta/lib/oe/package_managers/ipk/sdk.py       |  104 ++
 meta/lib/oe/package_managers/rpm/manifest.py  |   62 +
Quoted 8 lines
 .../package_managers/rpm/package_manager.py   |  423 +++++
 meta/lib/oe/package_managers/rpm/rootfs.py    |  154 ++
 meta/lib/oe/package_managers/rpm/sdk.py       |  104 ++
 meta/lib/oe/rootfs.py                         |  631 +-------
 meta/lib/oe/sdk.py                            |  311 +---
 meta/lib/oeqa/utils/package_manager.py        |   29 +-
 meta/recipes-core/meta/package-index.bb       |    2 +-
 20 files changed, 3014 insertions(+), 2490 deletions(-)

I'm afraid this patch is basically impossible to review. There are warning signs like the inclusion of the binary swo file. The hint that code gets duplicated also worries me, even if you abstract this, there shouldn't be reason to do that.

If we do this its going to have to be a more granular series where the patches *just* move code to new locations so we can see/understand what other changes are being made as I simply don't trust this patch, sorry :(.

Cheers,

Richard

Hi,
sorry about the swo file that's a mistake on my side. The duplicated code is simply so that deb and ipk shares a class. So my choice was to either duplicate that class (and let them diverge which is fine) or in the core package_manager.py file have a class that is not used by all package managers (rpm doesn't use it).

This patch is only code moves and class renames. I know it's very hard to review, that's a real problem. Let me see if I can split it up some more.

BR
Fredrik

#6 lore raw

Re: [OE-core] [PATCH 2/2] lib/oe: Split package manager code to multiple files

Quoted 57 lines · Fredrik Gustafsson

________________________________________
From: Richard Purdie <[email protected]>
Sent: Tuesday, June 23, 2020 2:02 PM
To: Fredrik Gustafsson; [email protected]
Cc: tools-cfpbuild-internal; Hugo Cedervall; Fredrik Gustafsson
Subject: Re: [OE-core] [PATCH 2/2] lib/oe: Split package manager code to multiple files

Quoted 26 lines · Fredrik Gustafsson

Today OE-Core has support for three package managers, even if there are many more package managers that could be interesting to use. Adding a new package manager to OE-Core would increase the test matrix significantly. In order to let users use their favorite package manager without need for it to have support from upstream, this patch refactors the package manager code so that it's easy to add a new package manager in another layer.

This patch is mostly moving code and duplicating some code (because of the hard coupling between deb and ipk).

---
 meta/classes/populate_sdk_base.bbclass        |   11 +-
 meta/lib/oe/manifest.py                       |  149 +-
 meta/lib/oe/package_manager.py                | 1402 +----------------
 meta/lib/oe/package_managers/deb/manifest.py  |   31 +
 .../package_managers/deb/package_manager.py   |  719 +++++++++
 meta/lib/oe/package_managers/deb/rootfs.py    |  213 +++
 meta/lib/oe/package_managers/deb/sdk.py       |   97 ++
 .../ipk/.package_manager.py.swo               |  Bin 0 -> 45056 bytes
 meta/lib/oe/package_managers/ipk/manifest.py  |   79 +
 .../package_managers/ipk/package_manager.py   |  588 +++++++
 meta/lib/oe/package_managers/ipk/rootfs.py    |  395 +++++
 meta/lib/oe/package_managers/ipk/sdk.py       |  104 ++
 meta/lib/oe/package_managers/rpm/manifest.py  |   62 +
Quoted 8 lines
 .../package_managers/rpm/package_manager.py   |  423 +++++
 meta/lib/oe/package_managers/rpm/rootfs.py    |  154 ++
 meta/lib/oe/package_managers/rpm/sdk.py       |  104 ++
 meta/lib/oe/rootfs.py                         |  631 +-------
 meta/lib/oe/sdk.py                            |  311 +---
 meta/lib/oeqa/utils/package_manager.py        |   29 +-
 meta/recipes-core/meta/package-index.bb       |    2 +-
 20 files changed, 3014 insertions(+), 2490 deletions(-)

I'm afraid this patch is basically impossible to review. There are warning signs like the inclusion of the binary swo file. The hint that code gets duplicated also worries me, even if you abstract this, there shouldn't be reason to do that.

If we do this its going to have to be a more granular series where the patches *just* move code to new locations so we can see/understand what other changes are being made as I simply don't trust this patch, sorry :(.

Cheers,

Quoted 7 lines

Richard

Hi,
sorry about the swo file that's a mistake on my side. The duplicated code is simply so that deb and ipk shares a class. So my choice was to either duplicate that class (and let them diverge which is fine) or in the core package_manager.py file have a class that is not used by all package managers (rpm doesn't use it).

This patch is only code moves and class renames. I know it's very hard to review, that's a real problem. Let me see if I can split it up some more.

I really like the idea here but agree with Richard that the patch is impossible to review as is. I'd recommend moving things over in stages and keeping the package_manager.py file around until we're completely finished. Recipes like package-index.bb should not need to change at all - leave a shim layer in place with the old import path.

I see you split the code into package_manager.py, rootfs.py, sdk.py and manifest.py for each package manager. That suggests you could start by just splitting out the rootfs code from each package manager, leaving the rest in package_manager.py. Then on the next commit move the sdk handling, etc. That would make each commit easier to reason about.

I'd also recommend adding a __init__.py file in the ipk, rpm and deb directories so it's easier to import things when needed.

Thanks,

Signature · 2 lines
Paul Barker
Konsulko Group
#7 lore raw

✗ patchtest: failure for "nopackages.bbclass: Move to no..." and 1 more

In reply to Fredrik Gustafsson

== Series Details ==

Series: "nopackages.bbclass: Move to no..." and 1 more
Revision: 1
URL : https://patchwork.openembedded.org/series/24801/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at 878a8561e1)

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
#8 lore raw
In reply to Fredrik Gustafsson

You mentioned Poky few times below - why is it Poky-specific? Since you are not submitting this change to Poky, but to OE-Core instead, can this be used by other OE distros?

Signature · 1 line
Denys
Quoted 45 lines · Fredrik Gustafsson

Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

#9 lore raw

Hi Denys,
you're right. I was confusing poky and OE-core. I'll change the cover letter. Thanks!

I haven't tested it with other distros but I see no reason that it wouldn't work.

BR
Fredrik
________________________________________
From: Denys Dmytriyenko <[email protected]>
Sent: Tuesday, June 23, 2020 8:50 PM
To: Fredrik Gustafsson
Cc: [email protected]; tools-cfpbuild-internal; Hugo Cedervall
Subject: Re: [OE-core] Add package managers as a plugin

You mentioned Poky few times below - why is it Poky-specific? Since you are not submitting this change to Poky, but to OE-Core instead, can this be used by other OE distros?

--
Denys

Quoted 45 lines · Fredrik Gustafsson

Poky today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for Poky since it increases the test matrix. Therefore this patch series refactors the package management code so that it's possible to add more package managers in other own layer. I will send another patch serie that will add apk.

Perfomance metrics below, (note that this includes build times).

APK
===
bitbake core-image-minimal
15.84s user 2.60s system 0% cpu 1:26:19.21 total
16.01s user 2.58s system 0% cpu 1:26:03.72 total
15.69s user 2.61s system 0% cpu 1:26:45.45 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -f -c do_rootfs
0.55s user 0.06s system 4% cpu 14.236 total
0.54s user 0.08s system 4% cpu 15.247 total
0.52s user 0.15s system 4% cpu 15.143 total

RPM
===
bitbake core-image-minimal
18.57s user 3.09s system 0% cpu 1:31:29.09 total
18.58s user 3.08s system 0% cpu 1:30:53.80 total
18.20s user 3.31s system 0% cpu 1:31:06.69 total

bitbake core-image-minimal -c cleansstate
bitbake core-image-minimal -c clean
bitbake core-image-minimal -c do_rootfs -f
0.58s user 0.10s system 3% cpu 19.470 total
0.57s user 0.09s system 3% cpu 19.480 total
0.60s user 0.07s system 3% cpu 20.381 total

#10 lore raw
In reply to Fredrik Gustafsson
Quoted 3 lines · Fredrik Gustafsson

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm.

I was interested enough in your comment here to collect a small sampling of comparative do_rootfs runs on one of my dev machines.

I used a virgin poky clone and tried to replicate the steps you provided in your OP, timing only the execution of bitbake -c do_rootfs core-image-minimal. I used the debian time package for timing. Here's what I got:

# Summary
PM     Avg do_rootfs time (s)
--     ----------------------
deb    33.89
ipk    33.35
rpm    31.85

# Raw Data
deb    00:33.54
deb    00:34.66
deb    00:33.55
deb    00:33.23
deb    00:34.46
ipk    00:33.96
ipk    00:34.43
ipk    00:33.99
ipk    00:32.17
ipk    00:32.22
rpm    00:29.92
rpm    00:32.31
rpm    00:31.90
rpm    00:33.31
rpm    00:31.80

Based on those results, it seems like all the current PMs are similar in terms of execution time. Are there major differences between my test setup and yours? And do you already have a layer with APK available that I could compare against on my hardware?

This is the crude script I used to automate the test runs:

```
#!/bin/bash
set -euxo pipefail

TIME=/usr/bin/time

recipe=core-image-minimal

# Options are:
#  - 'package_deb' for debian style deb files
#  - 'package_ipk' for ipk files are used by opkg (a debian style embedded package manager)
#  - 'package_rpm' for rpm style packages
function set_pkg_man() {
    pkg_man=${1}
    sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/" conf/local.conf
}

function rebuild_recipe() {
    bitbake -c cleanall $recipe
    bitbake $recipe
}

function test_recipe() {
    local name=${1}
    bitbake -c cleansstate $recipe
    bitbake -c clean $recipe
    $TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}

# rpm
set_pkg_man package_rpm
rebuild_recipe

test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5


# ipk
set_pkg_man package_ipk
rebuild_recipe

test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5

# deb
set_pkg_man package_deb
rebuild_recipe

test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```

Thanks,

Signature · 6 lines
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

[email protected]
office: +1(512)683-8522
#11 lore raw

Hi Alex,
Thanks for supplying a script!

I added apk to your script and then ran

```
for f in `ls`; do
printf "$f: "; cat $f | tail -2 | head -1 | awk -F ' ' '{print $3}' | sed -e 's/elapsed//'
done | sort
```

The result is here:
out.apk.1: 0:13.35
out.apk.2: 0:12.89
out.apk.3: 0:12.99
out.apk.4: 0:12.87
out.apk.5: 0:13.02
out.deb.1: 0:20.27
out.deb.2: 0:20.30
out.deb.3: 0:20.33
out.deb.4: 0:20.06
out.deb.5: 0:20.17
out.ipk.1: 0:18.51
out.ipk.2: 0:18.78
out.ipk.3: 0:18.80
out.ipk.4: 0:18.99
out.ipk.5: 0:18.97
out.rpm.1: 0:16.80
out.rpm.2: 0:16.83
out.rpm.3: 0:16.69
out.rpm.4: 0:16.78
out.rpm.5: 0:17.78

That is not supportive of my statement of ipk being the slowest package manager. That statement come from timing tests done with our proprietary version of poky that is somewhat extended when it comes to the do_rootfs task. We're generating a bit more artifacts than poky. However apk is still ~22% faster than rpm and ~40% faster than deb according to the data above.

So I tried your script with core-image-sato-sdk-ptest instead that is a bigger image. The results where more mixed. There is a huge difference between different package managers:
out.apk.1: 1:13.35
out.apk.2: 1:13.51
out.apk.3: 1:13.23
out.apk.4: 1:14.07
out.apk.5: 1:13.00
out.deb.1: 3:49.37
out.deb.2: 3:50.77
out.deb.3: 3:51.39
out.deb.4: 3:53.40
out.deb.5: 3:53.99
out.ipk.1: 2:38.99
out.ipk.2: 2:39.07
out.ipk.3: 2:35.34
out.ipk.4: 2:36.15
out.ipk.5: 2:34.55
out.rpm.1: 1:58.61
out.rpm.2: 1:59.42
out.rpm.3: 1:59.70
out.rpm.4: 1:58.96
out.rpm.5: 1:58.11

But yeah, I should probably rephrase my cover letter. You'll find a layer for APK that will apply on top of my patch serie attached to this email.

Also note that there's a difference between the time to generate a package. Which we completely ignore in this measurements. It's not as important since it's being cached, but for a fresh build it will be a difference between different package managers.

BR
Fredrik

___________________________
From: [email protected] <[email protected]> on behalf of Alex Stewart <[email protected]>
Sent: Wednesday, June 24, 2020 5:18 PM
To: Fredrik Gustafsson; [email protected]
Cc: tools-cfpbuild-internal; Hugo Cedervall; Alejandro Del Castillo
Subject: Re: [OE-core] Add package managers as a plugin

Quoted 3 lines · Fredrik Gustafsson

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. ipkg is way slower than deb or rpm.

I was interested enough in your comment here to collect a small sampling of comparative do_rootfs runs on one of my dev machines.

I used a virgin poky clone and tried to replicate the steps you provided in your OP, timing only the execution of bitbake -c do_rootfs core-image-minimal. I used the debian time package for timing. Here's what I got:

# Summary

PM     Avg do_rootfs time (s)
--     ----------------------
deb    33.89
ipk    33.35
rpm    31.85

# Raw Data

deb    00:33.54
deb    00:34.66
deb    00:33.55
deb    00:33.23
deb    00:34.46
ipk    00:33.96
ipk    00:34.43
ipk    00:33.99
ipk    00:32.17
ipk    00:32.22
rpm    00:29.92
rpm    00:32.31
rpm    00:31.90
rpm    00:33.31
rpm    00:31.80

Based on those results, it seems like all the current PMs are similar in terms of execution time. Are there major differences between my test setup and yours? And do you already have a layer with APK available that I could compare against on my hardware?

This is the crude script I used to automate the test runs:

```
#!/bin/bash
set -euxo pipefail

TIME=/usr/bin/time

recipe=core-image-minimal

# Options are:
#  - 'package_deb' for debian style deb files
#  - 'package_ipk' for ipk files are used by opkg (a debian style
embedded package manager)
#  - 'package_rpm' for rpm style packages
function set_pkg_man() {
     pkg_man=${1}
     sed -i "s/\(^PACKAGE_CLASSES ?= \)\"\(.*\)\"/\1\"$pkg_man\"/"
conf/local.conf
}

function rebuild_recipe() {
     bitbake -c cleanall $recipe
     bitbake $recipe
}

function test_recipe() {
     local name=${1}
     bitbake -c cleansstate $recipe
     bitbake -c clean $recipe
     $TIME bitbake -c do_rootfs -f $recipe 2>&1 | tee outs/out.$name
}

# rpm
set_pkg_man package_rpm
rebuild_recipe

test_recipe rpm.1
test_recipe rpm.2
test_recipe rpm.3
test_recipe rpm.4
test_recipe rpm.5


# ipk
set_pkg_man package_ipk
rebuild_recipe

test_recipe ipk.1
test_recipe ipk.2
test_recipe ipk.3
test_recipe ipk.4
test_recipe ipk.5

# deb
set_pkg_man package_deb
rebuild_recipe

test_recipe deb.1
test_recipe deb.2
test_recipe deb.3
test_recipe deb.4
test_recipe deb.5
```

Thanks,

--
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

[email protected]
office: +1(512)683-8522

#12 lore raw

Hey Fredrik,

Quoted 7 lines · Fredrik Gustafsson

I added apk to your script and then ran
<snip>

That is not supportive of my statement of ipk being the slowest package manager. That statement come from timing tests done with our proprietary version of poky that is somewhat extended when it comes to the do_rootfs task. We're generating a bit more artifacts than poky.

Ah; good to know. Thanks for checking again! Sounds like it could be related to extant opkg bugs which degrade performance when you have many (read: thousands) of packages in the feed.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=13593

Quoted 7 lines

So I tried your script with core-image-sato-sdk-ptest instead that is a bigger image. The results where more mixed. There is a huge difference between different package managers:
<snip>

But yeah, I should probably rephrase my cover letter. You'll find a layer for APK that will apply on top of my patch serie attached to this email.

Thanks for sending that; I'm interested to take a look at it. Those new performance numbers look impressive.

Good luck with the patchset.

Signature · 6 lines
Alex Stewart
Software Engineer - LabVIEW Real-Time OS
National Instruments

[email protected]
office: +1(512)683-8522