Skip to contentSkip to conversation

PACKAGE_EXCLUDE_COMPLEMENTARY question

DISCUSSION Rasmus Villemoes 2 participants 2 messages Jul 17

lore
Share & tools
lore
#0 lore raw

Hi

I was playing around with adding ptest packages to our rootfs to see how many of those tests would actually succeed on our targets, and if the rate was high enough, I'd do it permanently and run them as part of our normal test suite.

But it turns out I can't simply turn on ptest-pkgs, as it blows up the rootfs beyond what there's room for on many of our targets. One of the biggest offenders is openssl-ptest, weighing in at +650M (roughly 3x the size of the rootfs before adding any ptest packages...). So I looked around for a knob to say "yes, to install most -ptest packages, but not these...". There's PACKAGE_EXCLUDE_COMPLEMENTARY, which sort-of seems to be intended for this (though undocumented).

However, just adding "openssl" to that variable does not elide openssl-ptest from the image. After adding --debug to the 'oe-pkgdata-util glob' invocation in install_complementary(), I see that yes,

DEBUG: openssl -> !!

the openssl package itself gets ignored, but there are two "aliases" (for lack of the proper word) which do end up pulling in openssl-ptest:

DEBUG: libcrypto3 (*-ptest) -> openssl-ptest
DEBUG: libssl3 (*-ptest) -> openssl-ptest

I suppose adding those as well to PACKAGE_EXCLUDE_COMPLEMENTARY would do the job, but playing that game of whack-a-mole can't be how it's meant to be.

Coding-wise, this seems to come down to skipregex not being used at all in the "# Main processing loop", in particular not after reading the pn which a given package came from. But I have no idea how to modify that code to take skipregex into account or what could/would break if one did that.

Also, had there been for example a openssl-bash-completion package, I didn't really want to exclude that from the complementary package list. IOW, what I'd really want is not a filter on the input to the globbing, but a filter on the output from the "oe-pkgdata-util glob". Perhaps something like a PACKAGE_IGNORE_COMPLEMENTARY which would be subtracted from complementary_pkgs in install_complementary()? Or just use the existing PACKAGE_EXCLUDE and subtract that from the list to be installed? Adding openssl-ptest to PACKAGE_EXCLUDE currently fails hard because we end up asking the backend to both install and exclude the same package.

Rasmus

#1 lore raw

Yocto CI is using a different approach: each ptest package gets its own image, and they're all built with a meta-target core-image-ptest-all. Then they're tested in parallel with bitbake -c testimage core-image-ptest-all. I'd say this scales better and runs the tests faster than shoving all of ptest into a single image, so I would explore in that direction, e.g. you can write a custom version of core-image-ptest-all with exclusion lists.

Alex

Quoted 60 lines · Rasmus Villemoes via lists.openembedded.org

Hi

I was playing around with adding ptest packages to our rootfs to see how many of those tests would actually succeed on our targets, and if the rate was high enough, I'd do it permanently and run them as part of our normal test suite.

But it turns out I can't simply turn on ptest-pkgs, as it blows up the rootfs beyond what there's room for on many of our targets. One of the biggest offenders is openssl-ptest, weighing in at +650M (roughly 3x the size of the rootfs before adding any ptest packages...). So I looked around for a knob to say "yes, to install most -ptest packages, but not these...". There's PACKAGE_EXCLUDE_COMPLEMENTARY, which sort-of seems to be intended for this (though undocumented).

However, just adding "openssl" to that variable does not elide openssl-ptest from the image. After adding --debug to the 'oe-pkgdata-util glob' invocation in install_complementary(), I see that yes,

DEBUG: openssl -> !!

the openssl package itself gets ignored, but there are two "aliases" (for lack of the proper word) which do end up pulling in openssl-ptest:

DEBUG: libcrypto3 (*-ptest) -> openssl-ptest
DEBUG: libssl3 (*-ptest) -> openssl-ptest

I suppose adding those as well to PACKAGE_EXCLUDE_COMPLEMENTARY would do the job, but playing that game of whack-a-mole can't be how it's meant to be.

Coding-wise, this seems to come down to skipregex not being used at all in the "# Main processing loop", in particular not after reading the pn which a given package came from. But I have no idea how to modify that code to take skipregex into account or what could/would break if one did that.

Also, had there been for example a openssl-bash-completion package, I didn't really want to exclude that from the complementary package list. IOW, what I'd really want is not a filter on the input to the globbing, but a filter on the output from the "oe-pkgdata-util glob". Perhaps something like a PACKAGE_IGNORE_COMPLEMENTARY which would be subtracted from complementary_pkgs in install_complementary()? Or just use the existing PACKAGE_EXCLUDE and subtract that from the list to be installed? Adding openssl-ptest to PACKAGE_EXCLUDE currently fails hard because we end up asking the backend to both install and exclude the same package.

Rasmus

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#241162): https://lists.openembedded.org/g/openembedded-core/message/241162
Mute This Topic: https://lists.openembedded.org/mt/120313077/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-