RFC: OE-Core task rework

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

We need to pay particular attention to task-core-boot and task-base as these are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

Op 15 aug. 2012, om 11:46 heeft Paul Eggleton

Quoted 6 lines

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

Quoted 4 lines

We need to pay particular attention to task-core-boot and task-base as these are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

Have a look at task-boot and task-basic in meta-oe:
https://github.com/openembedded/meta-oe/tree/master/meta-oe/recipes-core/ta
sks

My pet peeve is the bloat oe-core forces onto qemu* builds in the name of "our testing team needs it!!!". Why am I forced to build an nfs server and distcc when setting MACHINE=qemusomething?

Hi all,

There have been a few requests to review the task recipes (i.e. package groups) provided by OE-Core, and indeed these have not really been looked at seriously since OE-Core was created. Ideally I think we want them to be useful to a wide audience and provide useful units of functionality that can be added to an image without the person doing the selection having to manually select too many individual packages. Imagine presenting the list of tasks to someone in a UI for assembling images (such as Hob or Narcissus) and you can start to see that we have some work to do in this area.

I know various distros and users of OE-Core have created their own tasks or resurrected tasks from OE-Classic, and this is an opportunity to perhaps look at bringing some of these (or at least, parts of them) into the core. It is true that tasks will often be an expression of distro policy, and we also can't have any tasks in OE-Core that refer to packages that don't exist in OE- Core; thus distros will always be extending the base tasks or adding their own
- and that's fine. However, with some thought I believe we can come up with a set of tasks that are generally useful to most people using OE-Core.

For reference, I've compiled a list on the wiki of the current tasks in OE- Core with links to the recipes and some notes:

http://www.openembedded.org/wiki/OE-Core_Task_Rework

Some of the things I think we ought to consider/address as part of this exercise:

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

Quoted 5 lines

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

Do the 'tasks' do anything other than specify meta-data groups? If they don't include any glue or additional processes then I strongly agree that the name should be changed.

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

Quoted 5 lines · Paul Eggleton

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

Yeah, I think OE inherited it from Familiar, which in turn got it from Debian. But the meaning of the term has drifted slightly through the generations and, as you say, it is no longer a very accurate reflection of what the packages in question are doing.

It's never been totally obvious to me that there is much need for tasks/package group recipes as such in oe-core itself; they're rather more of a DISTRO policy thing and their presence in the metadata does obviously have a cost in terms of parse time and memory usage. It might perhaps be worth exploring what they're actually being used for in oe-core and whether those things could be better done in a different way that doesn't involve having a .bb file for them.

Quoted 12 lines · On Wednesday 15 August 2012 12:25:23 Phil Blundell wrote:

Quoted 5 lines · Paul Eggleton

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

Yeah, I think OE inherited it from Familiar, which in turn got it from Debian. But the meaning of the term has drifted slightly through the generations and, as you say, it is no longer a very accurate reflection of what the packages in question are doing.

Quoted 7 lines

It's never been totally obvious to me that there is much need for tasks/package group recipes as such in oe-core itself; they're rather more of a DISTRO policy thing and their presence in the metadata does obviously have a cost in terms of parse time and memory usage. It might perhaps be worth exploring what they're actually being used for in oe-core and whether those things could be better done in a different way that doesn't involve having a .bb file for them.

I would say without checking that these days that cost is negligible. Off the top of my head, the benefits we get from using recipes for tasks as opposed to variables or some similar construct:

* The tasks can be used and referred to on the target if desired, not just
when you compose the image (i.e. task packages are produced and thus the package manager knows about them).

* It should be slightly easier to find where tasks are defined within the
metadata. In practice that's not always the case with the way we have named some of the tasks, hopefully that can be improved.

* Related to the above, they can be customised in a visible manner using
bbappends in your own layer.

* If you inherit from task.bbclass you automatically get -dev and -dbg
complements for the tasks defined by the recipe, which can sometimes be useful.

Cheers,
Paul

Quoted 4 lines

* The tasks can be used and referred to on the target if desired, not
just
when you compose the image (i.e. task packages are produced and thus the package manager knows about them).

I think this is a key advantage. Again, if we think of these tasks as logical groups of functionality, it gives an image developer (or installer) the ability to say "I need booting, discrete commands, python, perl, and LSB compliance." and get a system in the end that -should- work.

The image/installer should always be able to specify individual recipes as well, but often inexperienced users won't know that they need two or three recipes (that don't have actual dependencies on each other) to get a functionally complete answer.

Quoted 11 lines · Mark Hatle

Quoted 4 lines

* The tasks can be used and referred to on the target if desired, not
just
when you compose the image (i.e. task packages are produced and thus the package manager knows about them).

I think this is a key advantage. Again, if we think of these tasks as logical groups of functionality, it gives an image developer (or installer) the ability to say "I need booting, discrete commands, python, perl, and LSB compliance." and get a system in the end that -should- work.

Quoted 4 lines

The image/installer should always be able to specify individual recipes as well, but often inexperienced users won't know that they need two or three recipes (that don't have actual dependencies on each other) to get a functionally complete answer.

It seems like you're arguing in favor of the ability to add groups of packages to an image, which no one disagrees with, and isn't really relevant to the bit you're quoting. The bit that tasks add that package groups / image features don't is the ability to add them after the fact at runtime, not image creation time.

Hi all,

There have been a few requests to review the task recipes (i.e. package groups) provided by OE-Core, and indeed these have not really been looked at seriously since OE-Core was created. Ideally I think we want them to be useful to a wide audience and provide useful units of functionality that can be added to an image without the person doing the selection having to manually select too many individual packages. Imagine presenting the list of tasks to someone in a UI for assembling images (such as Hob or Narcissus) and you can start to see that we have some work to do in this area.

I know various distros and users of OE-Core have created their own tasks or resurrected tasks from OE-Classic, and this is an opportunity to perhaps look at bringing some of these (or at least, parts of them) into the core. It is true that tasks will often be an expression of distro policy, and we also can't have any tasks in OE-Core that refer to packages that don't exist in OE- Core; thus distros will always be extending the base tasks or adding their own
- and that's fine. However, with some thought I believe we can come up with a set of tasks that are generally useful to most people using OE-Core.

For reference, I've compiled a list on the wiki of the current tasks in OE- Core with links to the recipes and some notes:

http://www.openembedded.org/wiki/OE-Core_Task_Rework

Some of the things I think we ought to consider/address as part of this exercise:

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

We need to pay particular attention to task-core-boot and task-base as these
are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

3) Ensure all task recipes follow reasonable patterns:

  *  Fix recipe DESCRIPTIONs to make sense (and not contain Poky references)
  *  Ensure all individual packages have a decent SUMMARY/DESCRIPTION
  *  Try to make each task have a reasonable name - there are some current uses

of "core" and "base" that don't actually convey any meaning; LSB specific tasks should be named as such, etc.

* Make all tasks inherit task.bbclass so that they get proper dbg/dev
packages and any other common task functionality

4) Consider the usefulness of the existing PACKAGE_GROUP_ structure which converts some IMAGE_FEATURES into the addition of tasks to be installed (see classes/core-image.bbclass). At the very least we should probably rename this if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g. package-management, debug-tweaks, etc.)?

Please reply with your thoughts. Once we've come to a consensus on the things we want to do (allowing plenty of time for discussion), I'll look at putting together a set of patches that makes the appropriate changes.

Thanks,
Paul

Quoted 7 lines

if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a
select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g.
package-management, debug-tweaks, etc.)?

I'd certainly prefer that images were limited to selecting software from task (group) recipes only, and not providing their own. An image should be able to change the selection based on an "image feature" or similar configuration, but the underlying tasks/groups, recipes, etc should all be 'generic' to a given distribution configuration.

Hi all,

There have been a few requests to review the task recipes (i.e. package groups) provided by OE-Core, and indeed these have not really been looked at seriously since OE-Core was created. Ideally I think we want them to be useful to a wide audience and provide useful units of functionality that can be added to an image without the person doing the selection having to manually select too many individual packages. Imagine presenting the list of tasks to someone in a UI for assembling images (such as Hob or Narcissus) and you can start to see that we have some work to do in this area.

I know various distros and users of OE-Core have created their own tasks or resurrected tasks from OE-Classic, and this is an opportunity to perhaps look at bringing some of these (or at least, parts of them) into the core. It is true that tasks will often be an expression of distro policy, and we also can't have any tasks in OE-Core that refer to packages that don't exist in OE- Core; thus distros will always be extending the base tasks or adding their own
- and that's fine. However, with some thought I believe we can come up with a set of tasks that are generally useful to most people using OE-Core.

For reference, I've compiled a list on the wiki of the current tasks in OE- Core with links to the recipes and some notes:

http://www.openembedded.org/wiki/OE-Core_Task_Rework

Some of the things I think we ought to consider/address as part of this exercise:

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

We need to pay particular attention to task-core-boot and task-base as these
are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

3) Ensure all task recipes follow reasonable patterns:

  *  Fix recipe DESCRIPTIONs to make sense (and not contain Poky references)
  *  Ensure all individual packages have a decent SUMMARY/DESCRIPTION
  *  Try to make each task have a reasonable name - there are some current uses

of "core" and "base" that don't actually convey any meaning; LSB specific tasks should be named as such, etc.

* Make all tasks inherit task.bbclass so that they get proper dbg/dev
packages and any other common task functionality

4) Consider the usefulness of the existing PACKAGE_GROUP_ structure which converts some IMAGE_FEATURES into the addition of tasks to be installed (see classes/core-image.bbclass). At the very least we should probably rename this if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g. package-management, debug-tweaks, etc.)?

Please reply with your thoughts. Once we've come to a consensus on the things we want to do (allowing plenty of time for discussion), I'll look at putting together a set of patches that makes the appropriate changes.

Thanks,
Paul

Quoted 5 lines

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

"group" or "packagegroup" or "package-group" is my suggestion for the existing 'task' recipes. From what I've seen, it should be a rename opportunity -- we can even provide/rprovide the old names....

Quoted 11 lines

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

We need to pay particular attention to task-core-boot and task-base as these>
are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

During the pre-oe-core Yocto Project development, a design was generated to roughly group the packages into functional areas. For the most part, this design (as well as legacy elements) still exist.

I think the boot, base and others need to be evaluated for usefulness... but my feeling is most of them are close to being correct.

Quoted 3 lines

* Try to make each task have a reasonable name - there are some current uses of "core" and "base" that don't actually convey any meaning; LSB specific tasks should be named as such, etc.

Yup, there is a logical grouping for the lsb specific tasks, as for others. The naming needs to be made clear as to why it's there, and what it represents. Same with the summary and descriptions -- they should list the functionality provided by this group of components.

Quoted 8 lines

4) Consider the usefulness of the existing PACKAGE_GROUP_ structure which converts some IMAGE_FEATURES into the addition of tasks to be installed (see classes/core-image.bbclass). At the very least we should probably rename this if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g. package-management, debug-tweaks, etc.)?

I'd certainly prefer that images were limited to selecting software from task (group) recipes only, and not providing their own. An image should be able to change the selection based on an "image feature" or similar configuration, but the underlying tasks/groups, recipes, etc should all be 'generic' to a given distribution configuration.

I think if you go through the images today, a lot of that stuff is either old, or could be simplified by creating appropriate tasks/groups.

The main concern with LSB is that we have something called task-basic, which seems to be intended for LSB but does not state as much, and I know at least one person has tried to use it and then been a little puzzled as to why rpm has been pulled in when ipk packaging has been selected. Just naming some of these tasks more appropriately would help avoid such situations. In the specific case of task-basic there may be a case for creating a similar but not LSB-focused task that pulls in real shell utilities in preference to the light versions provided by busybox.

On Monday 20 August 2012 15:45:07 Mark Hatle wrote:

Quoted 5 lines

Yup, there is a logical grouping for the lsb specific tasks, as for others. The naming needs to be made clear as to why it's there, and what it represents. Same with the summary and descriptions -- they should list the functionality provided by this group of components.

The main concern with LSB is that we have something called task-basic, which seems to be intended for LSB but does not state as much, and I know at least one person has tried to use it and then been a little puzzled as to why rpm has been pulled in when ipk packaging has been selected. Just naming some of these tasks more appropriately would help avoid such situations. In the specific case of task-basic there may be a case for creating a similar but not LSB-focused task that pulls in real shell utilities in preference to the light versions provided by busybox.

Quoted 10 lines

Quoted 9 lines

4) Consider the usefulness of the existing PACKAGE_GROUP_ structure which converts some IMAGE_FEATURES into the addition of tasks to be installed (see classes/core-image.bbclass). At the very least we should probably rename this if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g. package-management, debug-tweaks, etc.)?

Quoted 5 lines

I'd certainly prefer that images were limited to selecting software from task (group) recipes only, and not providing their own. An image should be able to change the selection based on an "image feature" or similar configuration, but the underlying tasks/groups, recipes, etc should all be 'generic' to a given distribution configuration.

The first part seems reasonable to me; but I'm still short of an answer as to what to do with the existing IMAGE_FEATURES code as mentioned above. At the moment aside from the special features such as debug-tweaks, these are just a thin layer on top of task recipes which doesn't add very much at all other than extra maintenance.

Quoted 2 lines

I think if you go through the images today, a lot of that stuff is either old, or could be simplified by creating appropriate tasks/groups.

OK, that's a good point. I have another task on my list to clean up our image recipes and that would be a good segue into that.

Cheers,
Paul

Hi all,

There have been a few requests to review the task recipes (i.e. package groups) provided by OE-Core, and indeed these have not really been looked at seriously since OE-Core was created. Ideally I think we want them to be useful to a wide audience and provide useful units of functionality that can be added to an image without the person doing the selection having to manually select too many individual packages. Imagine presenting the list of tasks to someone in a UI for assembling images (such as Hob or Narcissus) and you can start to see that we have some work to do in this area.

I know various distros and users of OE-Core have created their own tasks or resurrected tasks from OE-Classic, and this is an opportunity to perhaps look at bringing some of these (or at least, parts of them) into the core. It is true that tasks will often be an expression of distro policy, and we also can't have any tasks in OE-Core that refer to packages that don't exist in OE- Core; thus distros will always be extending the base tasks or adding their own - and that's fine. However, with some thought I believe we can come up with a set of tasks that are generally useful to most people using OE-Core.

For reference, I've compiled a list on the wiki of the current tasks in OE- Core with links to the recipes and some notes:

http://www.openembedded.org/wiki/OE-Core_Task_Rework

Some of the things I think we ought to consider/address as part of this exercise:

1) Do we rename "task" to something a little more understandable to the uninitiated, such as "package group"? The word "task" is already used in a much more natural sense within bitbake as a unit of work. Historically I believe we picked up this term from Debian but I'm not aware of significant use by other mainstream distributions.

2) Look at the existing tasks and:
* evaluate their usefulness
* remove any that are obsolete
* adjust existing contents if needed
* look for useful groups of packages that might be added

We need to pay particular attention to task-core-boot and task-base as
these are pulled in by default in any image that inherits from core-image.bbclass - if these are not generally working for people that are creating their own images, we need to change them such that they are.

3) Ensure all task recipes follow reasonable patterns:

 *  Fix recipe DESCRIPTIONs to make sense (and not contain Poky references)
 *  Ensure all individual packages have a decent SUMMARY/DESCRIPTION
 *  Try to make each task have a reasonable name - there are some current

uses of "core" and "base" that don't actually convey any meaning; LSB specific tasks should be named as such, etc.
* Make all tasks inherit task.bbclass so that they get proper dbg/dev
packages and any other common task functionality

4) Consider the usefulness of the existing PACKAGE_GROUP_ structure which converts some IMAGE_FEATURES into the addition of tasks to be installed (see classes/core-image.bbclass). At the very least we should probably rename this if we rename tasks to package groups, and there's a question as to how IMAGE_FEATURES scales - should every task be represented in there or only a select list? Would we be better off not trying to bring any tasks in via IMAGE_FEATURES at all and mostly leave that to control post-processing (e.g. package-management, debug-tweaks, etc.)?

Please reply with your thoughts. Once we've come to a consensus on the things we want to do (allowing plenty of time for discussion), I'll look at putting together a set of patches that makes the appropriate changes.