Skip to contentSkip to conversation

Add package managers as a plugin

PATCH In review Fredrik Gustafsson 3 participants 9 messages Jul 1, 2020 – Jul 9, 2020 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

OE-core today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested.

However, it might or might not be what's wanted for OE-core since it increases the test matrix. Therefore I will refactor the package management code to allow a layer to add a new package manager without editing the meta layer.

This refactor will be divided into multiple patch series to be easier to review, this first one with a single patch.

#1 lore raw

PATCH v3 1/1 nopackages.bbclass: Get tasks from variable

Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +7 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}
Signature · 1 line
2.20.1
#2 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 18 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +0 -3
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk

meta/classes/nopackages.bbclass line 1

Quoted 10 lines
meta/classes/nopackages.bbclass +1 -9
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 8 lines
meta/classes/nopackages.bbclass +6 -1
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}
--

2.20.1

Signature · 2 lines
Paul Barker
Konsulko Group
#3 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 41 lines · Paul Barker
Quoted 30 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +1 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
Quoted 4 lines

do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 6 lines

meta/classes/nopackages.bbclass line 7

Quoted 6 lines
meta/classes/nopackages.bbclass +6 -0
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

I'm just sad this is having to use anonymous python. Its slow and I'd like to see less of it, not more, given people's complaints about parsing speed.

I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 41 lines · Paul Barker
Quoted 30 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +1 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
Quoted 4 lines

do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 6 lines

meta/classes/nopackages.bbclass line 7

Quoted 6 lines
meta/classes/nopackages.bbclass +6 -0
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

I'm just sad this is having to use anonymous python. Its slow and I'd like to see less of it, not more, given people's complaints about parsing speed.

Quoted 2 lines

I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5

is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#7 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +1 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
Quoted 5 lines

do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just
noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 6 lines

meta/classes/nopackages.bbclass line 7

Quoted 6 lines
meta/classes/nopackages.bbclass +6 -0
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.

Quoted 7 lines
Quoted 2 lines

I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5

is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.

Sorry:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a

Cheers,

Richard (in need of caffeine)

#8 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 70 lines · Richard Purdie
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +1 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
Quoted 5 lines

do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just
noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 6 lines

meta/classes/nopackages.bbclass line 7

Quoted 6 lines
meta/classes/nopackages.bbclass +6 -0
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.

Quoted 7 lines
Quoted 2 lines

I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5

is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.

Quoted 6 lines

Sorry:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a

Cheers,

Richard (in need of caffeine)

I like this approach.

Fredrik, what's the reason for using a variable here again? Is it just to be able to add tasks to the list or do you also need the ability to remove tasks from the list?

Thanks,

Signature · 2 lines
Paul Barker
Konsulko Group
#9 lore raw

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Paul, it's only for adding tasks to do deltask on. Not preventing current deltasks to be done.

Richard, I tried your suggestion but couldn't understand/judge the implications of changing
-__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
+__deltask_regexp__ = re.compile(r"deltask\s+(.+)")

If you're fine with that change. Your solution works fine for my usecase, EXCEPT that +EXTRA_NOPACKAGE_DELTASKS = ""
needs to be
+EXTRA_NOPACKAGE_DELTASKS ?= ""
right? Otherwise that variable would always be empty. Or would something like EXTRA_NOPACKAGE_DELTASKS_append(....) be used?

/Fredrik
________________________________________
From: Paul Barker <[email protected]>
Sent: Wednesday, July 1, 2020 12:22 PM
To: Richard Purdie
Cc: Fredrik Gustafsson; openembedded-core; tools-cfpbuild-internal; Fredrik Gustafsson
Subject: Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

Quoted 70 lines · Richard Purdie
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson

Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +1 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
Quoted 5 lines

do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if += was present in the previous versions, I only just
noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.

Quoted 6 lines

meta/classes/nopackages.bbclass line 7

Quoted 6 lines
meta/classes/nopackages.bbclass +6 -0
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.

Quoted 7 lines
Quoted 2 lines

I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5

is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.

Quoted 6 lines

Sorry:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a

Cheers,

Richard (in need of caffeine)

I like this approach.

Fredrik, what's the reason for using a variable here again? Is it just to be able to add tasks to the list or do you also need the ability to remove tasks from the list?

Thanks,

--
Paul Barker
Konsulko Group

#10 lore raw

Re: [PATCH v3] nopackages.bbclass: Get tasks from variable

In reply to Fredrik Gustafsson

Hi Richard,
is there something I can do to make this move forward?

/Fredrik
________________________________________
From: Fredrik Gustafsson
Sent: Wednesday, July 1, 2020 10:46 AM
To: [email protected]
Cc: tools-cfpbuild-internal; Fredrik Gustafsson
Subject: [PATCH v3] nopackages.bbclass: Get tasks from variable

Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.

---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +7 -12
index 559f5078bd..4c9fe510c5 100644
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}

--
2.20.1