RFD: Recipe variants, multilib and package handling

Why are variants of recipes important at all? I think we've all seen the benefit of BBCLASSEXTEND massively simplifying native recipes and the nativesdk class gives some glimpse of the generic power. We're in the process of adding multilib handling to OE-Core and this pushes that mechanism. It turns out there are some complexities here to do with the naming conventions and how we namespace various targets which need discussion.

Possible Namespaces
===================

For the purposes of this email lets assume we have the variants "<default target> <native> <nativesdk> <multilibA> <multilibB>". The exact format of the namespace is flexible but there are fours main viable approaches to package namespace of variants:

a) Something gets appended to the end of the name:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo

<native>     : pkgconfig-native
<nativesdk>  : pkgconfig-nativesdk pkgconfig-dev-nativesdk pkgconfig-locale-foo-nativesdk
<multilibA>  : pkgconfig-mlA pkgconfig-dev-mlA pkgconfig-locale-foo-mlA
<multilibB>  : pkgconfig-mlB pkgconfig-dev-mlB pkgconfig-locale-foo-mlB

b) Something gets appended to the end of PN:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo

<native>     : pkgconfig-native
<nativesdk>  : pkgconfig-nativesdk pkgconfig-nativesdk-dev pkgconfig-nativesdk-locale-foo
<multilibA>  : pkgconfig-mlA pkgconfig-mlA-dev pkgconfig-mlA-locale-foo
<multilibB>  : pkgconfig-mlB pkgconfig-mlB-dev pkgconfig-mlB-locale-foo

c) Something gets prepended to the start of the name:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo

<native>     : native-pkgconfig
<nativesdk>  : nativesdk-pkgconfig nativesdk-pkgconfig-dev nativesdk-pkgconfig-locale-foo
<multilibA>  : mlA-pkgconfig mlA-pkgconfig-dev mlA-pkgconfig-locale-foo
<multilibB>  : mlB-pkgconfig mlB-pkgconfig-dev mlB-pkgconfig-locale-foo

Variant d) where something gets prepended to PN works out the same as c) except where we have package names like lib${PN}.

Currently OE operates on b) or some variant thereof. BBCLASSEXTEND automatically adds something to the end of PN.

The two conflicting problems
============================

So what's the problem? It all comes down to how to process a recipe and turn it into one of these variants. There are two classes of problem:

i) Translating DEPENDS/RDEPENDS style variables

Given a general dependency like "pkgconfig-locale-foo", how do you turn it into the variant? For the cases like b) and d) above where some transformation of ${PN} is made, you don't know where to place the variant in the general string. In this example, its obvious to the human eye that it should be after "pkgconfig" but a computer can't do that generically.

That turns out to be hard so then you consider appending to the end of the name. You can search for any reference to the variant in the name, remove it and place it at the end.

ii) Dealing with assumptions about ${PN}

The whole of the metadata does things like:

PACKAGES = "${PN} ${PN}-dev"
FILES_${PN}-dev = "x"

The use of PN like this is why there is the attraction of simply appending to PN. The trouble is that i) above means you can't do this and reliably translate dependencies.

Options
=======

a) We could introduce some kind of suffix variable ${PKGSFX} which would contain the variant, maybe in conjuction with BPN instead of PN:

PACKAGES = "${BPN}${PKGSFX} ${BPN}-dev${PKGSFX}"
FILES_${BPN}-dev${PKGSFX} = "x"

This has the downside of having to touch the whole metadata and it looks ugly.

b) We could try and remap all the variables at BBCLASSEXTEND time. This would mean remapping PACKAGES and renaming all the package specific variables such as FILES, SUMMARY, DESCRIPTION, RDEPENDS, RRECOMMENDS, RSUGGESTS, RPROVIDES, RREPLACES, ALLOW_EMPTY, SECTION and likely others.

I knew were were going to have to read through some of those variables and remap the dependencies but having to totally move them into new namespaces is a scary thought (but still possible).

c) Switch to prepending to PN. This would only break cases where we have packages which don't start with ${PN} of which there are comparatively few. We could make the simplifying assumption that all PACKAGES start with ${PN} meaning that all manipulations would be simple prepends. If we do this, problems i) and ii) above are no longer an issue.

Discussion
==========

I don't think option a) above is viable and the current plan implies we'd do b) but its extremely ugly. I'm therefore tempted to look more seriously at c). The bigger issues would appear to be:

* It breaks with convention/tradition for OE (xxx-native vs native:xxx)
* It would add the constraint of packaging starting with ${PN}
* It would require changes to the likes of debian.bbclass to account for package prefixes when performing auto renaming
* It would break a small set of the metadata where packages don't start with ${PN} (although the class could simply refuse to extend these automatically).

Things to consider:

* Would we just do this for multilibs or would we transition native recipes to the new style of naming? We don't have PACKAGES problems for native recipes.
* Likewise, would nativesdk transition? Is has more PACKAGES problems so likely yes, it would make sense to transition.
* Would we stick with "-" as a delimiter or switch to something like ":"?

Thoughts/suggestions/better ideas welcome...

Cheers,

Richard

In OE-lite, we have tried out this option a. It works out pretty well, but is ugly, and does to some degree make it more complex to manage simple recipe meta-data.

For this reason, next OE-lite version will transition to something more like the option b you suggest.

In next OE-lite version, I have introduce something called "recipe type", which is nothing else than the name from "BBCLASSEXEND", or "machine" for default recipe type.

When resolving dependencies, I have introduced a recipe type context. One for DEPENDS for each recipe type, and one for RDEPENDS for the recipe types that is relevant for run-time dependency handling.

In recipe meta-data, you can fx. say

DEPENDS = "libncurses native:libtool"

Based on the recipe type context, the "libncurses" is prefixed with the recipe type of the context, fx. "machine:libncurses" for machine recipe type.

To further simplify recipe meta-data, I have introduced some generic prefix aliases. I have "host" and "target", and "host-cross" and "target-cross", which is then mapped to the proper recipe type based on the context.

This seems to work out very nicely. I seems effective in reducing the need for recipe type specfic overrides of DEPENDS variables, and makes recipe meta-data much easier to read IMHO.

Of-course, OE-lite does only have package based dependencies, and I haven't given any thought to how this would work out for OE's recipe based build dependencies.

But perhaps it would also be a good choice for OE.

Hehe. In danish, we have a saying: "Hvo intet vover, intet vinder", which I belive is something like "Nothing ventured, nothing gained" in english.

Looking into the future, wouldn't it make most sense to use the solution broadly in OE, to avoid making it even harder to get started with OE?

I decided to switch from "-" to ":", to allow robust translation of non-prefixed dependency names.

Thanks,

Esben

On 06/13/2011 04:52 AM, Richard Purdie wrote:

[snip]

True, but how long do we stick with things that are limiting us when we need a change to fix a real problem? And this is a little easier to deal with, now that we do really have a notion of doing releases so we can more easily explain to folks when the change is.

I know we have some, but do they also really have a good reason for not being ${PN}-foo now, possibly other than we borrowed the notion there from someone else? For example, I know Ubuntu is still 'ssh' for 'openssh', but we don't do that one.

Maybe we need to rename debian.bbclass while we're at it and yes, taking into account multilib is, to me, just a 'yeah, gonna have to' as part of the problem.

I think refusing is a good starting point to encourage someone that needs it to update the recipe, or it can be a janitor project in the end if the set is small enough...

I see a positive here being one less thing to change, but the downside being one more set of logic sitting around. Perhaps as a second pass migrating native over...

I think it'll have to, at least before it's all said and done, otherwise you will run into someone extending for both and puzzling over the different names they get.

Internally, that might make things easier but in terms of writing out the packages, that could be a problem...

I wish it was easier to abstract things away so we could get namespace B and keep that information around to solve problem 'i'. Then problem 'ii' would just be about changing how we define PN.

Tom Rini
Mentor Graphics Corporation