want to clarify proper approach to shared lib (.so) installation
sort of a 2-part soliloquy. in current YP code base i've inherited,
most of the internal (local directory SRC_URI-based) recipes inherit a proprietary class that, among doing other internal, proprietary things, totally redefines PACKAGES as:
PACKAGES = "${PN} ... ${PN}-dev ..."i had never really noticed that before, but it's pretty obvious that that's not a great idea, as it allows what i call the recipe "base" package (${PN}) to gather up everything that matches its standard wildcard pattern before moving on, in effect "stealing" content from subsequent packages they would normally get if the base package was at the end, not the beginning.
somehow, this has worked all this time, but it's clear(?) that what
would be the "normal" contents of the various packages isn't going to be what one would expect; in particular, the base package is going to be what i call "overpackaged", with lots of stuff it doesn't really need so i'm guessing what's going into the image is more than is really necessary. somehow, though, it's worked all this time until recently, when i noticed this quirk was causing some Q/A issues, so i took a deep breath, commented out that line from the class file to use the default packaging approach and re-tried the build, which is when all hell broke loose.
it turns out that these internal recipes use local Makefile-based
source directories, which build, then install their generated artifacts in a temporary (non-YP) staging area per recipe, *then* manually (little by little) install stuff in ${D} via a general do_install() routine, at which point the regular packaging and subsequent steps kick in, but it's what now gets copied into ${D} that is causing grief.
apparently, many of these recipes generate a shared library, and i'm
well aware that the *normal* packaging involving a shared library is like this example from libidn2 (snipped for brevity to show only shared lib stuff):
libidn2/
usr/
lib/
libidn2.so.0 -> libidn2.so.0.3.7
libidn2.so.0.3.7 [actual library file]
libidn2-dev/
usr/
lib/
libidn2.so -> libidn2.so.0.3.7
so the *normal* packaging for a shared lib is that the lib itself and a symlink to it go into the base package, while another symlink goes into the "-dev" package. i'd never really paid that much attention to that until i reset that PACKAGES variable, as all of these internal recipes end up installing into ${D} nothing but the shared lib file itself under /usr/lib, and why this has worked all this time is a mystery, but having made this change is generating all sorts of Q/A diagnostics as this is what ends up in ${D} using a "fubar" recipe example given the manual installation being done using normal packaging:
fubar/
usr/
bin/
... snip ...
no lib/ directory
fubar-dev/
usr/
lib/
libfnvcma.so [actual shared lib]
unsurprisingly, there are QA issues with the above:
ERROR: fubar-1.0-r0 do_package_qa: QA Issue: -dev package contains non-symlink .so: fubar-dev path .../packages-split/fubar-dev/usr/lib/libfnvcma.so'[dev-elf]
ERROR: fubar-1.0-r0 do_package_qa: QA Issue: fubar rdepends on fubar-dev [dev-deps]
*sigh*.
in short, because these internal recipes generate only the single
shared lib file itself and that's all that's copied into ${D}, the regular Q/A tests will naturally barf, and i could use INSANE_SKIP all over the place to get around this, but it seems to me that the proper approach is to tell the developers that they need to start generating the appropriate symlinks for all of their recipes so packaging is done properly.
i'm just about to check if there is a switch or class i can invoke
that will "fix" this issue (as in, set up the shared libs in ${D} properly), but apart from that, am i correct in thinking that the developers need to do this correctly from the beginning?
rday
Quoted 89 lines · Robert P. J. Day
sort of a 2-part soliloquy. in current YP code base i've inherited,
most of the internal (local directory SRC_URI-based) recipes inherit a proprietary class that, among doing other internal, proprietary things, totally redefines PACKAGES as:PACKAGES = "${PN} ... ${PN}-dev ..."i had never really noticed that before, but it's pretty obvious that that's not a great idea, as it allows what i call the recipe "base" package (${PN}) to gather up everything that matches its standard wildcard pattern before moving on, in effect "stealing" content from subsequent packages they would normally get if the base package was at the end, not the beginning.
somehow, this has worked all this time, but it's clear(?) that what
would be the "normal" contents of the various packages isn't going to be what one would expect; in particular, the base package is going to be what i call "overpackaged", with lots of stuff it doesn't really need so i'm guessing what's going into the image is more than is really necessary. somehow, though, it's worked all this time until recently, when i noticed this quirk was causing some Q/A issues, so i took a deep breath, commented out that line from the class file to use the default packaging approach and re-tried the build, which is when all hell broke loose.it turns out that these internal recipes use local Makefile-based
source directories, which build, then install their generated artifacts in a temporary (non-YP) staging area per recipe, *then* manually (little by little) install stuff in ${D} via a generaldo_install()routine, at which point the regular packaging and subsequent steps kick in, but it's what now gets copied into ${D} that is causing grief.apparently, many of these recipes generate a shared library, and i'm
well aware that the *normal* packaging involving a shared library is like this example from libidn2 (snipped for brevity to show only shared lib stuff):libidn2/
usr/
lib/
libidn2.so.0 -> libidn2.so.0.3.7
libidn2.so.0.3.7 [actual library file]libidn2-dev/
usr/
lib/libidn2.so -> libidn2.so.0.3.7so the *normal* packaging for a shared lib is that the lib itself and a symlink to it go into the base package, while another symlink goes into the "-dev" package. i'd never really paid that much attention to that until i reset that PACKAGES variable, as all of these internal recipes end up installing into ${D} nothing but the shared lib file itself under
/usr/lib, and why this has worked all this time is a mystery, but having made this change is generating all sorts of Q/A diagnostics as this is what ends up in ${D} using a "fubar" recipe example given the manual installation being done using normal packaging:fubar/
usr/
bin/... snip ...no
lib/directoryfubar-dev/
usr/
lib/
libfnvcma.so [actual shared lib]unsurprisingly, there are QA issues with the above:
ERROR: fubar-1.0-r0
do_package_qa: QA Issue: -dev package contains non-symlink .so: fubar-dev path .../packages-split/fubar-dev/usr/lib/libfnvcma.so'[dev-elf]
ERROR: fubar-1.0-r0do_package_qa: QA Issue: fubar rdepends on fubar-dev [dev-deps]*sigh*.
in short, because these internal recipes generate only the single
shared lib file itself and that's all that's copied into ${D}, the regular Q/A tests will naturally barf, and i could useINSANE_SKIPall over the place to get around this, but it seems to me that the proper approach is to tell the developers that they need to start generating the appropriate symlinks for all of their recipes so packaging is done properly.
Quoted 9 lines
i'm just about to check if there is a switch or class i can invoke
that will "fix" this issue (as in, set up the shared libs in ${D} properly), but apart from that, am i correct in thinking that the developers need to do this correctly from the beginning?rday
Hi,
in the latter example you are packaging an unversioned shlib:
libfnvcma.so [actual shared lib]
In this case you might play with
SOLIBS = ".so"
FILES_SOLIBSDEV = ""
Please have a look at
https://wiki.yoctoproject.org/wiki/TipsAndTricks/Packaging_Prebuilt_Libraries
Cheers
A.A.
Quoted 103 lines · Andrea Adami
Quoted 89 lines · Robert P. J. Day
sort of a 2-part soliloquy. in current YP code base i've inherited,
most of the internal (local directory SRC_URI-based) recipes inherit a proprietary class that, among doing other internal, proprietary things, totally redefines PACKAGES as:PACKAGES = "${PN} ... ${PN}-dev ..."i had never really noticed that before, but it's pretty obvious that that's not a great idea, as it allows what i call the recipe "base" package (${PN}) to gather up everything that matches its standard wildcard pattern before moving on, in effect "stealing" content from subsequent packages they would normally get if the base package was at the end, not the beginning.
somehow, this has worked all this time, but it's clear(?) that what
would be the "normal" contents of the various packages isn't going to be what one would expect; in particular, the base package is going to be what i call "overpackaged", with lots of stuff it doesn't really need so i'm guessing what's going into the image is more than is really necessary. somehow, though, it's worked all this time until recently, when i noticed this quirk was causing some Q/A issues, so i took a deep breath, commented out that line from the class file to use the default packaging approach and re-tried the build, which is when all hell broke loose.it turns out that these internal recipes use local Makefile-based
source directories, which build, then install their generated artifacts in a temporary (non-YP) staging area per recipe, *then* manually (little by little) install stuff in ${D} via a generaldo_install()routine, at which point the regular packaging and subsequent steps kick in, but it's what now gets copied into ${D} that is causing grief.apparently, many of these recipes generate a shared library, and i'm
well aware that the *normal* packaging involving a shared library is like this example from libidn2 (snipped for brevity to show only shared lib stuff):libidn2/
usr/
lib/
libidn2.so.0 -> libidn2.so.0.3.7
libidn2.so.0.3.7 [actual library file]libidn2-dev/
usr/
lib/libidn2.so -> libidn2.so.0.3.7so the *normal* packaging for a shared lib is that the lib itself and a symlink to it go into the base package, while another symlink goes into the "-dev" package. i'd never really paid that much attention to that until i reset that PACKAGES variable, as all of these internal recipes end up installing into ${D} nothing but the shared lib file itself under
/usr/lib, and why this has worked all this time is a mystery, but having made this change is generating all sorts of Q/A diagnostics as this is what ends up in ${D} using a "fubar" recipe example given the manual installation being done using normal packaging:fubar/
usr/
bin/... snip ...no
lib/directoryfubar-dev/
usr/
lib/
libfnvcma.so [actual shared lib]unsurprisingly, there are QA issues with the above:
ERROR: fubar-1.0-r0
do_package_qa: QA Issue: -dev package contains non-symlink .so: fubar-dev path .../packages-split/fubar-dev/usr/lib/libfnvcma.so'[dev-elf]
ERROR: fubar-1.0-r0do_package_qa: QA Issue: fubar rdepends on fubar-dev [dev-deps]*sigh*.
in short, because these internal recipes generate only the single
shared lib file itself and that's all that's copied into ${D}, the regular Q/A tests will naturally barf, and i could useINSANE_SKIPall over the place to get around this, but it seems to me that the proper approach is to tell the developers that they need to start generating the appropriate symlinks for all of their recipes so packaging is done properly.Quoted 9 lines
i'm just about to check if there is a switch or class i can invoke
that will "fix" this issue (as in, set up the shared libs in ${D} properly), but apart from that, am i correct in thinking that the developers need to do this correctly from the beginning?rday
Hi,
in the latter example you are packaging an unversioned shlib:
Quoted 8 lines
libfnvcma.so [actual shared lib]
In this case you might play with
SOLIBS = ".so"
FILES_SOLIBSDEV= ""Please have a look at
https://wiki.yoctoproject.org/wiki/TipsAndTricks/Packaging_Prebuilt_Libraries
excellent, i was sure there was a simple solution to this.
rday
Quoted 108 lines · Robert P. J. Day
Quoted 105 lines · Andrea Adami
Quoted 89 lines · Robert P. J. Day
sort of a 2-part soliloquy. in current YP code base i've inherited,
most of the internal (local directory SRC_URI-based) recipes inherit a proprietary class that, among doing other internal, proprietary things, totally redefines PACKAGES as:PACKAGES = "${PN} ... ${PN}-dev ..."i had never really noticed that before, but it's pretty obvious that that's not a great idea, as it allows what i call the recipe "base" package (${PN}) to gather up everything that matches its standard wildcard pattern before moving on, in effect "stealing" content from subsequent packages they would normally get if the base package was at the end, not the beginning.
somehow, this has worked all this time, but it's clear(?) that what
would be the "normal" contents of the various packages isn't going to be what one would expect; in particular, the base package is going to be what i call "overpackaged", with lots of stuff it doesn't really need so i'm guessing what's going into the image is more than is really necessary. somehow, though, it's worked all this time until recently, when i noticed this quirk was causing some Q/A issues, so i took a deep breath, commented out that line from the class file to use the default packaging approach and re-tried the build, which is when all hell broke loose.it turns out that these internal recipes use local Makefile-based
source directories, which build, then install their generated artifacts in a temporary (non-YP) staging area per recipe, *then* manually (little by little) install stuff in ${D} via a generaldo_install()routine, at which point the regular packaging and subsequent steps kick in, but it's what now gets copied into ${D} that is causing grief.apparently, many of these recipes generate a shared library, and i'm
well aware that the *normal* packaging involving a shared library is like this example from libidn2 (snipped for brevity to show only shared lib stuff):libidn2/
usr/
lib/
libidn2.so.0 -> libidn2.so.0.3.7
libidn2.so.0.3.7 [actual library file]libidn2-dev/
usr/
lib/libidn2.so -> libidn2.so.0.3.7so the *normal* packaging for a shared lib is that the lib itself and a symlink to it go into the base package, while another symlink goes into the "-dev" package. i'd never really paid that much attention to that until i reset that PACKAGES variable, as all of these internal recipes end up installing into ${D} nothing but the shared lib file itself under
/usr/lib, and why this has worked all this time is a mystery, but having made this change is generating all sorts of Q/A diagnostics as this is what ends up in ${D} using a "fubar" recipe example given the manual installation being done using normal packaging:fubar/
usr/
bin/... snip ...no
lib/directoryfubar-dev/
usr/
lib/
libfnvcma.so [actual shared lib]unsurprisingly, there are QA issues with the above:
ERROR: fubar-1.0-r0
do_package_qa: QA Issue: -dev package contains non-symlink .so: fubar-dev path .../packages-split/fubar-dev/usr/lib/libfnvcma.so'[dev-elf]
ERROR: fubar-1.0-r0do_package_qa: QA Issue: fubar rdepends on fubar-dev [dev-deps]*sigh*.
in short, because these internal recipes generate only the single
shared lib file itself and that's all that's copied into ${D}, the regular Q/A tests will naturally barf, and i could useINSANE_SKIPall over the place to get around this, but it seems to me that the proper approach is to tell the developers that they need to start generating the appropriate symlinks for all of their recipes so packaging is done properly.Quoted 9 lines
i'm just about to check if there is a switch or class i can invoke
that will "fix" this issue (as in, set up the shared libs in ${D} properly), but apart from that, am i correct in thinking that the developers need to do this correctly from the beginning?rday
Hi,
in the latter example you are packaging an unversioned shlib:libfnvcma.so [actual shared lib]
Quoted 8 lines
Quoted 6 lines
In this case you might play with
SOLIBS = ".so"
FILES_SOLIBSDEV= ""Please have a look at
https://wiki.yoctoproject.org/wiki/TipsAndTricks/Packaging_Prebuilt_Librariesexcellent, i was sure there was a simple solution to this.
Just a warning, but if your Makefiles are not generating versioned shared libraries then they are probably not setting a soname either... and that really messes up OE's automatic runtime dependency tracking for shared libraries.
If OE is your main build environment for these components it's usually better in the long run to just update your Makefiles to create versioned shared libraries (using simple version .0 and a single symlink is fine) and setting a soname than to use the above trick to force packaging of an unversioned .so and manually defining runtime dependencies everywhere.
On Fri, 4 Jun 2021, Andre McCurdy wrote:
... big ship ...
Quoted 11 lines
Just a warning, but if your Makefiles are not generating versioned shared libraries then they are probably not setting a soname either... and that really messes up OE's automatic runtime dependency tracking for shared libraries.
If OE is your main build environment for these components it's usually better in the long run to just update your Makefiles to create versioned shared libraries (using simple version .0 and a single symlink is fine) and setting a soname than to use the above trick to force packaging of an unversioned .so and manually defining runtime dependencies everywhere.
oh, i realize that ... the current build system is very
legacy-driven and i have a lengthy list of cleanups to do, this just one of them. for now, things just have to work until i can bend things into shape.
rday