Add package managers as a plugin
OE-core today has three different package managers, the well-known formats deb and rpm is supported as well as ipkg that is good for embedded devices.
When building and having a good cache hit, a significant amount of time is spent in the phase of generating a rootfs, which is really about the performance of the package manager. To save build time and also get a package manager that is suitanle for use on targets where flash memory is a concern, support for apk is suggested.
However, it might or might not be what's wanted for OE-core since it increases the test matrix. Therefore I will refactor the package management code to allow a layer to add a new package manager without editing the meta layer.
This refactor will be divided into multiple patch series to be easier to review, this first one with a single patch.
PATCH v3 1/1 nopackages.bbclass: Get tasks from variable
Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.
--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +7 -12
@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene" + +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}
Signature · 1 line
2.20.1
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 18 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.
Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +0
-3@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk
meta/classes/nopackages.bbclass line 1
Quoted 10 lines
meta/classes/nopackages.bbclass +1
-9-deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"
Not sure if += was present in the previous versions, I only just noticed it now. I think ?= is a better choice so it can be overridden easily. Other than that, this looks good.
Quoted 8 lines
meta/classes/nopackages.bbclass +6
-1+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +} --2.20.1
Signature · 2 lines
Paul Barker Konsulko Group
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 41 lines · Paul Barker
Quoted 30 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +1
-12@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpmQuoted 4 lines
do_package_write_ipkdo_package_write_debdo_package_qado_packagedatado_package_setscenedo_package_write_rpm_setscenedo_package_write_ipk_setscenedo_package_write_deb_setscenedo_package_qa_setscenedo_packagedata_setscene"Not sure if
+=was present in the previous versions, I only just noticed it now. I think?=is a better choice so it can be overridden easily. Other than that, this looks good.
Quoted 6 lines
meta/classes/nopackages.bbclass line 7
Quoted 6 lines
meta/classes/nopackages.bbclass +6
-0+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}
I'm just sad this is having to use anonymous python. Its slow and I'd like to see less of it, not more, given people's complaints about parsing speed.
I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 41 lines · Paul Barker
Quoted 30 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +1
-12@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpmQuoted 4 lines
do_package_write_ipkdo_package_write_debdo_package_qado_packagedatado_package_setscenedo_package_write_rpm_setscenedo_package_write_ipk_setscenedo_package_write_deb_setscenedo_package_qa_setscenedo_packagedata_setscene"Not sure if
+=was present in the previous versions, I only just noticed it now. I think?=is a better choice so it can be overridden easily. Other than that, this looks good.Quoted 6 lines
meta/classes/nopackages.bbclass line 7
Quoted 6 lines
meta/classes/nopackages.bbclass +6
-0+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}I'm just sad this is having to use anonymous python. Its slow and I'd like to see less of it, not more, given people's complaints about parsing speed.
Quoted 2 lines
I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.
is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +1
-12@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpmQuoted 5 lines
do_package_write_ipkdo_package_write_debdo_package_qado_packagedatado_package_setscenedo_package_write_rpm_setscenedo_package_write_ipk_setscenedo_package_write_deb_setscenedo_package_qa_setscenedo_packagedata_setscene"Not sure if
+=was present in the previous versions, I only just
noticed it now. I think?=is a better choice so it can be overridden easily. Other than that, this looks good.Quoted 6 lines
meta/classes/nopackages.bbclass line 7
Quoted 6 lines
meta/classes/nopackages.bbclass +6
-0+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.
Quoted 7 lines
Quoted 2 lines
I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.
is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.
Cheers,
Richard (in need of caffeine)
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 70 lines · Richard Purdie
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +1
-12@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpmQuoted 5 lines
do_package_write_ipkdo_package_write_debdo_package_qado_packagedatado_package_setscenedo_package_write_rpm_setscenedo_package_write_ipk_setscenedo_package_write_deb_setscenedo_package_qa_setscenedo_packagedata_setscene"Not sure if
+=was present in the previous versions, I only just
noticed it now. I think?=is a better choice so it can be overridden easily. Other than that, this looks good.Quoted 6 lines
meta/classes/nopackages.bbclass line 7
Quoted 6 lines
meta/classes/nopackages.bbclass +6
-0+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.Quoted 7 lines
Quoted 2 lines
I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.
is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.
Quoted 6 lines
Cheers,
Richard (in need of caffeine)
I like this approach.
Fredrik, what's the reason for using a variable here again? Is it just to be able to add tasks to the list or do you also need the ability to remove tasks from the list?
Thanks,
Signature · 2 lines
Paul Barker Konsulko Group
Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Paul, it's only for adding tasks to do deltask on. Not preventing current deltasks to be done.
Richard, I tried your suggestion but couldn't understand/judge the implications of changing
-__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
+__deltask_regexp__ = re.compile(r"deltask\s+(.+)")
If you're fine with that change. Your solution works fine for my usecase, EXCEPT that +EXTRA_NOPACKAGE_DELTASKS = ""
needs to be
+EXTRA_NOPACKAGE_DELTASKS ?= ""
right? Otherwise that variable would always be empty. Or would something like EXTRA_NOPACKAGE_DELTASKS_append(....) be used?
/Fredrik
________________________________________
From: Paul Barker <[email protected]>
Sent: Wednesday, July 1, 2020 12:22 PM
To: Richard Purdie
Cc: Fredrik Gustafsson; openembedded-core; tools-cfpbuild-internal; Fredrik Gustafsson
Subject: Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable
Quoted 70 lines · Richard Purdie
Quoted 59 lines · Richard Purdie via lists.openembedded.org
Quoted 57 lines · Richard Purdie via lists.openembedded.org
Quoted 44 lines · Paul Barker
Quoted 31 lines · Fredrik Gustafsson
Get tasks from a variable instead of having them hardcoded. This
will
enable other layers to add tasks that should be deleted when nopackages
is used.Signed-off-by Fredrik Gustafsson <[email protected]>--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)meta/classes/nopackages.bbclass +1
-12@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpmQuoted 5 lines
do_package_write_ipkdo_package_write_debdo_package_qado_packagedatado_package_setscenedo_package_write_rpm_setscenedo_package_write_ipk_setscenedo_package_write_deb_setscenedo_package_qa_setscenedo_packagedata_setscene"Not sure if
+=was present in the previous versions, I only just
noticed it now. I think?=is a better choice so it can be overridden easily. Other than that, this looks good.Quoted 6 lines
meta/classes/nopackages.bbclass line 7
Quoted 6 lines
meta/classes/nopackages.bbclass +6
-0+ +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about parsing speed.Quoted 7 lines
Quoted 2 lines
I did make a suggestion about how this could be done but that isn't an option. I might sort out such a patch.
is a patch which lets deltask become a bit more flexible. I've not tested it much but see it if works/helps.
Quoted 6 lines
Cheers,
Richard (in need of caffeine)
I like this approach.
Fredrik, what's the reason for using a variable here again? Is it just to be able to add tasks to the list or do you also need the ability to remove tasks from the list?
Thanks,
--
Paul Barker
Konsulko Group
Re: [PATCH v3] nopackages.bbclass: Get tasks from variable
In reply to Fredrik GustafssonHi Richard,
is there something I can do to make this move forward?
/Fredrik
________________________________________
From: Fredrik Gustafsson
Sent: Wednesday, July 1, 2020 10:46 AM
To: [email protected]
Cc: tools-cfpbuild-internal; Fredrik Gustafsson
Subject: [PATCH v3] nopackages.bbclass: Get tasks from variable
Get tasks from a variable instead of having them hardcoded. This will enable other layers to add tasks that should be deleted when nopackages is used.
--- meta/classes/nopackages.bbclass | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
meta/classes/nopackages.bbclass +7 -12
@@ -1,12 +1,7 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene" + +python () { + list = d.getVar("NO_PACKAGES_DELTASKS").split() + for task in list: + bb.build.deltask(task, d) +}
--
2.20.1