is there an easy to prevent *creation* of some recipe's packages?
pretty sure i know the answer to this one, but was asked the other
day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.
i pointed out that, based on default package definition as found in
bitbake.conf:
PACKAGE_BEFORE_PN ?= ""
PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc
${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
PACKAGES_DYNAMIC = "^${PN}-locale-.*"
FILES = ""
FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
${sysconfdir} ${sharedstatedir} ${localstatedir} \
... snip ...for every recipe you build, those *are* the packages that will be built and populated, short of overriding those values. i suggested that, in normal behaviour, if you check the generated WORKDIR for any recipe, you would first find the full package breakdown under packages-split/, where individual package contents can be examined, and you could further check out the rpms under deploy-rpms/, which would be built regardless of whether you chose to install them or not. (that's correct, yes?)
i finally suggested that, sure, if you do some ugly hacking, i
guess you could save time by not even creating some of those packages, but i speculated that that wouldn't be a significant saving and that it wasn't worth trying to cut corners like that and you were more likely to break things than save any meaningful build time.
thoughts?
rday
Quoted 6 lines · Robert P. J. Day
pretty sure i know the answer to this one, but was asked the other
day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.
I pretty much doubt it will speed up anything.
If for instance the -dev package will be removed from PACKAGES it is very likely that the files of that package will end up in the base package ${PN}, which results in larger target packages that need to be installed (and potentially further RDEPENDS)
Furthermore this will break DEPENDS for recipes that do rely on the -dev package - so chances are high that this will break (or magically change) something further down the line.
One thing that you can do to speed up the build process in terms of packaging time is to delete any **really** not needed files in a do_install_append, if no proper configuration option (cmake flag, make flag, PACKAGECONFIG) does exist which prevents the creation of these files in the first place.
I would recommend not to mess with the packaging defaults for reasons such as speeding things up - as this is IMHO addressing the issue at the wrong end - just make sure the files in <workdir>/image won't get created
Quoted 37 lines
i pointed out that, based on default package definition as found in
bitbake.conf:PACKAGE_BEFORE_PN ?= "" PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" PACKAGES_DYNAMIC = "^${PN}-locale-.*" FILES = "" FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \ ${sysconfdir} ${sharedstatedir} ${localstatedir} \ ... snip ...for every recipe you build, those *are* the packages that will be built and populated, short of overriding those values. i suggested that, in normal behaviour, if you check the generated WORKDIR for any recipe, you would first find the full package breakdown under packages-split/, where individual package contents can be examined, and you could further check out the rpms under deploy-rpms/, which would be built regardless of whether you chose to install them or not. (that's correct, yes?)
i finally suggested that, sure, if you do some ugly hacking, i
guess you could save time by not even creating some of those packages, but i speculated that that wouldn't be a significant saving and that it wasn't worth trying to cut corners like that and you were more likely to break things than save any meaningful build time.thoughts?
rday
Quoted 25 lines · Konrad Weihmann
Quoted 7 lines · Robert P. J. Day
pretty sure i know the answer to this one, but was asked the
other day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.I pretty much doubt it will speed up anything. If for instance the -dev package will be removed from PACKAGES it is very likely that the files of that package will end up in the base package ${PN}, which results in larger target packages that need to be installed (and potentially further RDEPENDS)
Furthermore this will break DEPENDS for recipes that do rely on the -dev package - so chances are high that this will break (or magically change) something further down the line.
One thing that you can do to speed up the build process in terms of packaging time is to delete any **really** not needed files in a
do_install_append, if no proper configuration option (cmake flag, make flag, PACKAGECONFIG) does exist which prevents the creation of these files in the first place.
Quoted 4 lines
I would recommend not to mess with the packaging defaults for reasons such as speeding things up - as this is IMHO addressing the issue at the wrong end - just make sure the files in <workdir>/image won't get created
that's the general idea i was trying to get across (so let's confirm
that i was speaking accurately). i suggested that the majority of time spent in processing a recipe was doing all the fetching and unpacking and patching and (mostly) compiling, the end result of which was to populate that recipe's WORKDIR/image directory, at which time the packaging step took over. so unless one wanted to dangerously mangle the package definitions, if it ended up in ${D}, it was going to get packaged, correct?
so unless one wanted to go back and mess with the recipe's build
configuration to, perhaps, not generate some artifacts, those packages were going to be built, whether they were subsequently installed or not. so short of mangling each recipe file to change its configuration, just leave it alone as the payoff just isn't worth it.
sound about right?
rday
Quoted 27 lines · Robert P. J. Day
Quoted 25 lines · Konrad Weihmann
Quoted 7 lines · Robert P. J. Day
pretty sure i know the answer to this one, but was asked the
other day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.I pretty much doubt it will speed up anything. If for instance the -dev package will be removed from PACKAGES it is very likely that the files of that package will end up in the base package ${PN}, which results in larger target packages that need to be installed (and potentially further RDEPENDS)
Furthermore this will break DEPENDS for recipes that do rely on the -dev package - so chances are high that this will break (or magically change) something further down the line.
One thing that you can do to speed up the build process in terms of packaging time is to delete any **really** not needed files in a
do_install_append, if no proper configuration option (cmake flag, make flag, PACKAGECONFIG) does exist which prevents the creation of these files in the first place.
Quoted 13 lines
Quoted 4 lines
I would recommend not to mess with the packaging defaults for reasons such as speeding things up - as this is IMHO addressing the issue at the wrong end - just make sure the files in <workdir>/image won't get created
that's the general idea i was trying to get across (so let's confirm
that i was speaking accurately). i suggested that the majority of time spent in processing a recipe was doing all the fetching and unpacking and patching and (mostly) compiling, the end result of which was to populate that recipe's WORKDIR/image directory, at which time the packaging step took over. so unless one wanted to dangerously mangle the package definitions, if it ended up in ${D}, it was going to get packaged, correct?
yes... *but* what the packaging does it to loop over PACKAGES and for each item pick up the corresponding FILES_<pkg-name> var and "move" (I think it's actually a copy, but that's just details) all matching files from image to packages-split. If files ending up in image that don't match *any* of the FILES-glob-pattern one would get a QA error.
Quoted 8 lines
so unless one wanted to go back and mess with the recipe's build
configuration to, perhaps, not generate some artifacts, those packages were going to be built, whether they were subsequently installed or not. so short of mangling each recipe file to change its configuration, just leave it alone as the payoff just isn't worth it.sound about right?
yes - the results of trimming down the input to /image will much more have a positive impact.
AFAIK packaging speed is directly linked to the number of files in /image (worst example I've seen is a ruby package containing 250k+ files, which takes round about 20min to package on my machine) - concluding from that, reduction of the number of files in /image is will do the trick
Quoted 3 lines
rday
Let me just throw in one idea with respect to Open Source License Compliance and packaging - with "traditional" programming languages.
Let's say we ship a product and are in the phase of an Open Source License Compliance audit.
We have a couple of interesting issues here
1) Actually we just need to audit the "main package", which is installed on the target. Unfortunately we completely lost the connection between what's in the various packages and where those things come from. What I mean is, that we typically audit the whole source repo, which contains e.g. test cases as well, but those test cases don't even end up in the target. Which leads to "over reporting".
1.1) Now we could just say we need some license files which are in the repo plus the -src or -dbg package, depending on how things are built. Ideally the debug sources are those which correspond to what's installed on the target.
2) What else is kind of lost, or not taken into account with licensing is linking. We only see each package/repo individually, but the right thing to do would be to look at the combined work.
If we look at the total project just auditing what's actually needed would help a lot time-wise as well ;)
Regards
(another) Robert ;)
Quoted 59 lines · Konrad Weihmann
Quoted 27 lines · Robert P. J. Day
Quoted 25 lines · Konrad Weihmann
Quoted 7 lines · Robert P. J. Day
pretty sure i know the answer to this one, but was asked the
other day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.I pretty much doubt it will speed up anything. If for instance the -dev package will be removed from PACKAGES it is very likely that the files of that package will end up in the base package ${PN}, which results in larger target packages that need to be installed (and potentially further RDEPENDS)
Furthermore this will break DEPENDS for recipes that do rely on the -dev package - so chances are high that this will break (or magically change) something further down the line.
One thing that you can do to speed up the build process in terms of packaging time is to delete any **really** not needed files in a
do_install_append, if no proper configuration option (cmake flag, make flag, PACKAGECONFIG) does exist which prevents the creation of these files in the first place.Quoted 13 lines
Quoted 4 lines
I would recommend not to mess with the packaging defaults for reasons such as speeding things up - as this is IMHO addressing the issue at the wrong end - just make sure the files in <workdir>/image won't get created
that's the general idea i was trying to get across (so let's confirm
that i was speaking accurately). i suggested that the majority of time spent in processing a recipe was doing all the fetching and unpacking and patching and (mostly) compiling, the end result of which was to populate that recipe's WORKDIR/image directory, at which time the packaging step took over. so unless one wanted to dangerously mangle the package definitions, if it ended up in ${D}, it was going to get packaged, correct?yes... *but* what the packaging does it to loop over PACKAGES and for each item pick up the corresponding FILES_<pkg-name> var and "move" (I think it's actually a copy, but that's just details) all matching files from image to packages-split.
If files ending up in image that don't match *any* of the FILES-glob-pattern one would get a QA error.Quoted 8 lines
so unless one wanted to go back and mess with the recipe's build
configuration to, perhaps, not generate some artifacts, those packages were going to be built, whether they were subsequently installed or not. so short of mangling each recipe file to change its configuration, just leave it alone as the payoff just isn't worth it.sound about right?
Quoted 8 lines
yes - the results of trimming down the input to /image will much more have a positive impact.
AFAIK packaging speed is directly linked to the number of files in /image (worst example I've seen is a ruby package containing 250k+ files, which takes round about 20min to package on my machine) - concluding from that, reduction of the number of files in /image is will do the trick
This thread seems to have covered the main point already (if you want to save time by not packaging unwanted files then don't install the files in the first place).
It's perhaps worth noting one very common case where files are installed and packaged unnecessarily and that's having the ptest distro feature enabled even though you don't intend to install any ptest packages. For most users ptest should be something you enable selectively when you want to prove the functionality of recipes in oe-core (e.g. after upgrading to a new OE release or when bringing up a new machine) and then disable again afterwards.
Quoted 5 lines · Robert P. J. Day
pretty sure i know the answer to this one, but was asked the other
day and wanted to make sure. in order to speed up the normal OE build, someone suggested bypassing the creation of packages that weren't going to be relevant, such as -dev, -doc and so on -- the idea was that that could make a significant difference in build time.
General case: no. Absolute waste of time.
There are common exceptions: if the recipe has e.g. API documentation that takes a long time to build then add a PACKAGECONFIG for it and respect the api-documentation DISTRO_FEATURE.
There are limited edge cases of course: Piglit is 99% auto-generated test cases, the binaries are *huge* and the need to debug them is limited, so the recipe installs striped binaries so there is no -dbg package to generate. We're talking minutes of time spent stripping gigabytes of libraries here and there's no hackery or magic, the recipe just tells the makefile to strip on install.
Ross