Allarch and packagegroup improvement proposal

As some people are already painfully aware, the current way packagegroup and allarch interact with the sstate signatures is painful.

Some example problem cases:

* An allarch package of scripts with an interpretor dependency (e.g perl)

* A packagegroup with dependencies on something which is debian renamed e.g. build-essentials on libgmp

Currently, packagegroups default to allarch and have their sstate dependencies truncated. This means that when something like gmp rebuilds, it can change package name thanks to debian renaming but the packagegroup referencing the old name remains, causing image creation failures.

So the "obvious" fix is to stop truncating the dependencies? Well, that means that the allarch package is written out every time machine changes to a different arch.

So we need to make packagegroups PACKAGE_ARCH specific? Well, this still doesn't solve the problem of allarch packages having ever greater problems with things like dependencies on perl.

An alternative solution I'm wondering about to address both issues is:

* the sstatesig packagegroup code for dependencies handling should only trigger for allarch
* we add a sanity test that checks if renaming has happened for packagegroup dependencies and warn if it has.

The trouble is the sstatesig code doesn't have the full data store, only a limited cross section so it can't check if PACKAGE_ARCH == all, only if the class is inherited. We therefore probably need to change packagegroup to do conditional class inclusion (which is possible with the modern code base at least). The use of the class can be tested from the sstatesig code and hence indicate allarch.

For the second part, we could add something like:

(quoted diff) +3 -0

+    if bb.data.inherits_class('packagegroup', d) and bb.data.inherits_class('allarch', d):    
+        if set(deps) != set(new_depends):
+            bb.warn("allarch packagegroup %s has renamed dependency %s" % (pkg, str(set(deps) ^ set(new_depends)))) 

to runtime_mapping_rename in package.bbclass which tells us which packagegroups need to move to be PACKAGE_ARCH. Unfortunately, a quick run of OE-Core shows:

WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dbg has renamed dependency set(['libsdl-dbg', 'libsdl-1.2-dbg'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dev has renamed dependency set(['libsdl-dev', 'libsdl-1.2-dev'])
WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])

so we do have the problem in a number of places.

Does anyone have any thoughts?

Quoted 17 lines · Richard Purdie

As some people are already painfully aware, the current way packagegroup and allarch interact with the sstate signatures is painful.

Some example problem cases:

* An allarch package of scripts with an interpretor dependency (e.g perl)

* A packagegroup with dependencies on something which is debian renamed e.g. build-essentials on libgmp

Currently, packagegroups default to allarch and have their sstate dependencies truncated. This means that when something like gmp rebuilds, it can change package name thanks to debian renaming but the packagegroup referencing the old name remains, causing image creation failures.

Quoted 7 lines

So the "obvious" fix is to stop truncating the dependencies? Well, that means that the allarch package is written out every time machine changes to a different arch.

So we need to make packagegroups PACKAGE_ARCH specific? Well, this still doesn't solve the problem of allarch packages having ever greater problems with things like dependencies on perl.

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

Quoted 42 lines

An alternative solution I'm wondering about to address both issues is:

* the sstatesig packagegroup code for dependencies handling should only trigger for allarch
* we add a sanity test that checks if renaming has happened for packagegroup dependencies and warn if it has.

The trouble is the sstatesig code doesn't have the full data store, only a limited cross section so it can't check if PACKAGE_ARCH == all, only if the class is inherited. We therefore probably need to change packagegroup to do conditional class inclusion (which is possible with the modern code base at least). The use of the class can be tested from the sstatesig code and hence indicate allarch.

For the second part, we could add something like:

(quoted diff) +3 -0

+    if bb.data.inherits_class('packagegroup', d) and bb.data.inherits_class('allarch', d):
+        if set(deps) != set(new_depends):
+            bb.warn("allarch packagegroup %s has renamed dependency %s" % (pkg, str(set(deps) ^ set(new_depends))))

to runtime_mapping_rename in package.bbclass which tells us which packagegroups need to move to be PACKAGE_ARCH. Unfortunately, a quick run of OE-Core shows:

WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dbg has renamed dependency set(['libsdl-dbg', 'libsdl-1.2-dbg'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dev has renamed dependency set(['libsdl-dev', 'libsdl-1.2-dev'])
WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])

Quoted 3 lines

so we do have the problem in a number of places.

Does anyone have any thoughts?

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

packagegroups are small and "rebuilt" quickly, so I don't mind "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like we do for couple of them already.

I can even find few changes from me on ML which do exactly that.

Quoted 71 lines · Martin Jansa

Quoted 17 lines · Richard Purdie

As some people are already painfully aware, the current way packagegroup and allarch interact with the sstate signatures is painful.

Some example problem cases:

* An allarch package of scripts with an interpretor dependency (e.g perl)

* A packagegroup with dependencies on something which is debian renamed e.g. build-essentials on libgmp

Currently, packagegroups default to allarch and have their sstate dependencies truncated. This means that when something like gmp rebuilds, it can change package name thanks to debian renaming but the packagegroup referencing the old name remains, causing image creation failures.

Quoted 7 lines

So the "obvious" fix is to stop truncating the dependencies? Well, that means that the allarch package is written out every time machine changes to a different arch.

So we need to make packagegroups PACKAGE_ARCH specific? Well, this still doesn't solve the problem of allarch packages having ever greater problems with things like dependencies on perl.

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

Quoted 42 lines

An alternative solution I'm wondering about to address both issues is:

* the sstatesig packagegroup code for dependencies handling should only trigger for allarch
* we add a sanity test that checks if renaming has happened for packagegroup dependencies and warn if it has.

The trouble is the sstatesig code doesn't have the full data store, only a limited cross section so it can't check if PACKAGE_ARCH == all, only if the class is inherited. We therefore probably need to change packagegroup to do conditional class inclusion (which is possible with the modern code base at least). The use of the class can be tested from the sstatesig code and hence indicate allarch.

For the second part, we could add something like:

(quoted diff) +3 -0

+    if bb.data.inherits_class('packagegroup', d) and bb.data.inherits_class('allarch', d):
+        if set(deps) != set(new_depends):
+            bb.warn("allarch packagegroup %s has renamed dependency %s" % (pkg, str(set(deps) ^ set(new_depends))))

to runtime_mapping_rename in package.bbclass which tells us which packagegroups need to move to be PACKAGE_ARCH. Unfortunately, a quick run of OE-Core shows:

WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dbg has renamed dependency set(['libsdl-dbg', 'libsdl-1.2-dbg'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base-dev has renamed dependency set(['libsdl-dev', 'libsdl-1.2-dev'])
WARNING: allarch packagegroup packagegroup-core-sdk has renamed dependency set(['libgomp', 'libgomp1'])
WARNING: allarch packagegroup packagegroup-core-x11-base has renamed dependency set(['dbus-1', 'dbus'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target has renamed dependency set(['libgcc1', 'libgcc-s-dev', 'libstdc++', 'libc6-dev', 'libgcc-dev', 'libgcc', 'libc6', 'eglibc-dbg', 'libc6-dbg', 'eglibc-dev', 'eglibc-thread-db', 'eglibc', 'libthread-db1', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-x11-sato-base has renamed dependency set(['libsdl-1.2-0', 'libsdl'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dbg has renamed dependency set(['libgcc-s-dbg', 'eglibc-dbg', 'libc6-dbg', 'libgcc-dbg'])
WARNING: allarch packagegroup packagegroup-core-standalone-sdk-target-dev has renamed dependency set(['libgcc-s-dev', 'libc6-dev', 'libgcc-dev', 'eglibc-dev'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])
WARNING: allarch packagegroup packagegroup-core-buildessential has renamed dependency set(['libstdc++', 'libstdc++6'])

Quoted 6 lines

Quoted 3 lines

so we do have the problem in a number of places.

Does anyone have any thoughts?

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

I did a rough find and grep in oe-core, we have 27 packagegroup recipes:

$ find meta -name 'packagegroup-*.bb' | wc -l
27

And 7 ones which set the PACKAGE_ARCH:

$ grep PACKAGE_ARCH meta -r | grep 'packagegroup-.*.bb:' | wc -l
7

It seems that allarch still wins.

I think that we really need such a WARNING not matter whether remove the default allarch or not. (if removed, we have to set the allarch for a few packagegroup recipes, then we still need such a check.)

Quoted 21 lines · Richard Purdie

As some people are already painfully aware, the current way packagegroup and allarch interact with the sstate signatures is painful.

Some example problem cases:

* An allarch package of scripts with an interpretor dependency (e.g perl)

* A packagegroup with dependencies on something which is debian renamed e.g. build-essentials on libgmp

Currently, packagegroups default to allarch and have their sstate dependencies truncated. This means that when something like gmp rebuilds, it can change package name thanks to debian renaming but the packagegroup referencing the old name remains, causing image creation failures.

So the "obvious" fix is to stop truncating the dependencies? Well, that means that the allarch package is written out every time machine changes to a different arch.

Quoted 3 lines

So we need to make packagegroups PACKAGE_ARCH specific? Well, this still doesn't solve the problem of allarch packages having ever greater problems with things like dependencies on perl.

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

packagegroups are small and "rebuilt" quickly, so I don't mind "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like we do for couple of them already.

I can even find few changes from me on ML which do exactly that.

Quoted 2 lines · Martin Jansa

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

Take a step back and think about this from the end user system perspective. The packages are all identical for each architecture, "perl" doesn't change name.

To me that means the correct end result is such a package should be "allarch" in the package feeds.

The question then becomes, how do we generate such things in a sane way.

Quoted 8 lines

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

packagegroups are small and "rebuilt" quickly, so I don't mind "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like we do for couple of them already.

I can even find few changes from me on ML which do exactly that.

It does seem a bit of a cop out to do this on the grounds that its small/fast :/.

I agree there is good reason why some should be PKGARCH but we can probably do better than just marking them all that way IMO. I think we should try and mark them correctly too, i.e. think about whether the packages really are identical and/or make sense as allarch and try and avoid duplication if so.

Quoted 22 lines · Martin Jansa

Quoted 2 lines

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

Take a step back and think about this from the end user system perspective. The packages are all identical for each architecture, "perl" doesn't change name.

To me that means the correct end result is such a package should be "allarch" in the package feeds.

The question then becomes, how do we generate such things in a sane way.

Quoted 8 lines

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

packagegroups are small and "rebuilt" quickly, so I don't mind "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like we do for couple of them already.

I can even find few changes from me on ML which do exactly that.

Quoted 8 lines

It does seem a bit of a cop out to do this on the grounds that its small/fast :/.

I agree there is good reason why some should be PKGARCH but we can probably do better than just marking them all that way IMO. I think we should try and mark them correctly too, i.e. think about whether the packages really are identical and/or make sense as allarch and try and avoid duplication if so.

I also have one other idea. We could adjust debian.bbclass so that it adds an RPROVIDES for the original package name. We could then instruct packagegroups specifically not to adjust the naming for debian renaming.

This would mean that the packagegroups would know the dependencies by their non-debian, unversioned name.

Would that work for people?

I'm torn on the idea right now but thought I'd share it.

Op 18 aug. 2014, om 17:35 heeft Richard Purdie <[email protected]> het volgende geschreven:

Quoted 41 lines · Richard Purdie

Quoted 22 lines · Martin Jansa

Quoted 2 lines

Well then maybe that allarch package with perl dependency shouldn't be marked as allarch.

Take a step back and think about this from the end user system perspective. The packages are all identical for each architecture, "perl" doesn't change name.

To me that means the correct end result is such a package should be "allarch" in the package feeds.

The question then becomes, how do we generate such things in a sane way.

Quoted 8 lines

I'm in favor of removing default allarch and setting correct PACKAGE_ARCH manually in the packagegroup recipes like we do elsewhere.

packagegroups are small and "rebuilt" quickly, so I don't mind "building" them once per TUNE_PKGARCH or even once per MACHINE_ARCH like we do for couple of them already.

I can even find few changes from me on ML which do exactly that.

Quoted 8 lines

It does seem a bit of a cop out to do this on the grounds that its small/fast :/.

I agree there is good reason why some should be PKGARCH but we can probably do better than just marking them all that way IMO. I think we should try and mark them correctly too, i.e. think about whether the packages really are identical and/or make sense as allarch and try and avoid duplication if so.

I also have one other idea. We could adjust debian.bbclass so that it adds an RPROVIDES for the original package name. We could then instruct packagegroups specifically not to adjust the naming for debian renaming.

This would mean that the packagegroups would know the dependencies by their non-debian, unversioned name.

Would that work for people?

Quoted 1 line

I'm torn on the idea right now but thought I'd share it.

What happens in the following situation:

foo.bb generates foo.ipk (/usr/bin/foo) and libfoo5.ipk (/usr/lib/libfoo.so.5), will libfoo5 RPROVIDE 'foo', 'libfoo' or something else?