Extending images
I've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.
I then tried
CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous) with the new package added.
I can see that the images produced are vastly different:
* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2
* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-20140718155134.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2
* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2
What's the difference and why did the first attempt fail?
Thanks
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 26 lines · Gary Thomas
've always used '
IMAGE_INSTALL+= " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL+= " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.I then tried
CORE_IMAGE_EXTRA_INSTALL+= " strace"
which produced a perfectly working build (just as previous) with the new package added.I can see that the images produced are vastly different:
* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34
core-image-sato-qemuarm-20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2* After '
IMAGE_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55
core-image-sato-qemuarm-20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2
Quoted 8 lines
* After '
CORE_IMAGE_EXTRA_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05
core-image-sato-qemuarm-20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2What's the difference and why did the first attempt fail?
Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 29 lines · Christopher Larson
On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:
've always used '
IMAGE_INSTALL+= " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL+= " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.I then tried
CORE_IMAGE_EXTRA_INSTALL+= " strace"
which produced a perfectly working build (just as previous) with the new package added.I can see that the images produced are vastly different:
* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2* After '
IMAGE_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-__20140718155134.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2* After '
CORE_IMAGE_EXTRA_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2
Quoted 6 lines
What's the difference and why did the first attempt fail?
Most likely the image defined its own
IMAGE_INSTALLusing ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then yourIMAGE_INSTALL+= would have had no effect at all. To append toIMAGE_INSTALLdirectly without usingCORE_IMAGE_EXTRA_INSTALLyou could have usedIMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.
That makes sense, thanks. I'll stick to using CORE_IMAGE_EXTRA_INSTALL from now on!
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 10 lines · Gary Thomas
Most likely the image defined its own
IMAGE_INSTALLusing ?=, so definingQuoted 8 lines
it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If
the recipe didn't use ?=, then yourIMAGE_INSTALL+= would have had no effect at all. To append toIMAGE_INSTALLdirectly without usingCORE_IMAGE_EXTRA_INSTALLyou could have usedIMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.
Quoted 2 lines
That makes sense, thanks. I'll stick to using
CORE_IMAGE_EXTRA_INSTALLfrom now on!
That's best, indeed. This is a case where there's a real difference between variables intended for the user to override from configuration, and those which are not. IMAGE_INSTALL is a detail of how images are constructed, CORE_IMAGE_EXTRA_INSTALL is an explicit hook provided to the user to non-destructively add packages to an image, so we don't have to try to guess at what the recipe will do with it -- it has explicit semantics, so we know this. There might be value in differentiating between cases like this in the documentation, if we don't already.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 49 lines · Gary Thomas
Quoted 39 lines · Christopher Larson
On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:
've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added: IMAGE_INSTALL += " strace" This produced a completely broken image which barely came up to a shell, lots of missing programs, etc. I then tried CORE_IMAGE_EXTRA_INSTALL += " strace" which produced a perfectly working build (just as previous) with the new package added. I can see that the images produced are vastly different: * Original working build -rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2 * After 'IMAGE_INSTALL += ' -rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-__20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2 * After 'CORE_IMAGE_EXTRA_INSTALL += ' -rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2 What's the difference and why did the first attempt fail?Quoted 8 lines
Most likely the image defined its own
IMAGE_INSTALLusing ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then yourIMAGE_INSTALL+= would have had no effect at all. To append toIMAGE_INSTALLdirectly without usingCORE_IMAGE_EXTRA_INSTALLyou could have usedIMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.
Quoted 2 lines
That makes sense, thanks. I'll stick to using
CORE_IMAGE_EXTRA_INSTALLfrom now on!
not to sound repetitively repetitive, but this is related to what i
thought was the weird way core images are defined based off of core-image.bbclass, where core-image.bbclass does *exactly* what you describe above:
CORE_IMAGE_BASE_INSTALL = '\
packagegroup-core-boot \
packagegroup-base-extended \
\
${CORE_IMAGE_EXTRA_INSTALL} \
'CORE_IMAGE_EXTRA_INSTALL ?= ""
IMAGE_INSTALL ?= "${CORE_IMAGE_BASE_INSTALL}"
the danger here is that if someone has been defining/building images
for a while based off of the lower-level image.bbclass, then (i think) using "IMAGE_INSTALL +=" will work just fine, and the poor developer gets lulled into a false sense of security and carries that habit over to her first core-image and ... boom.
i know there's a standard to define variables prefixed with "EXTRA_"
meant for users to add features, and i think that's a great idea. i think the docs should strongly emphasize that users should look for the EXTRA_ variable before messing with anything else.
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 67 lines · Robert P. J. Day
Quoted 50 lines · Christopher Larson
Quoted 39 lines
On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:
've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added: IMAGE_INSTALL += " strace" This produced a completely broken image which barely came up to a shell, lots of missing programs, etc. I then tried CORE_IMAGE_EXTRA_INSTALL += " strace" which produced a perfectly working build (just as previous) with the new package added. I can see that the images produced are vastly different: * Original working build -rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2 * After 'IMAGE_INSTALL += ' -rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-__20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2 * After 'CORE_IMAGE_EXTRA_INSTALL += ' -rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2 What's the difference and why did the first attempt fail?Quoted 8 lines
Most likely the image defined its own
IMAGE_INSTALLusing ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then yourIMAGE_INSTALL+= would have had no effect at all. To append toIMAGE_INSTALLdirectly without usingCORE_IMAGE_EXTRA_INSTALLyou could have usedIMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.Quoted 2 lines
That makes sense, thanks. I'll stick to using
CORE_IMAGE_EXTRA_INSTALLfrom now on!not to sound repetitively repetitive, but this is related to what i
thought was the weird way core images are defined based off of core-image.bbclass, where core-image.bbclass does *exactly* what you describe above:CORE_IMAGE_BASE_INSTALL = '\ packagegroup-core-boot \ packagegroup-base-extended \ \ ${CORE_IMAGE_EXTRA_INSTALL} \ '
CORE_IMAGE_EXTRA_INSTALL?= ""
Quoted 8 lines
IMAGE_INSTALL?= "${CORE_IMAGE_BASE_INSTALL}"the danger here is that if someone has been defining/building images
for a while based off of the lower-level image.bbclass, then (i think) using "IMAGE_INSTALL+=" will work just fine, and the poor developer gets lulled into a false sense of security and carries that habit over to her first core-image and ... boom.
I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to use IMAGE_INSTALL_append if anything, even back in oe classic :)
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 4 lines · Christopher Larson
I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to
IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to useIMAGE_INSTALL_appendif anything, even back in oe classic :) --
a quick grep in openembedded-core/meta shows:
$ grep -r "IMAGE_INSTALL +=" * recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils" recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\ recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL += "kernel-dev" recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\ recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\ recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev" recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect" recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev" recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL += "packagegroup-core-x11-sato-games" $
now i'm scared to look. :-)
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 25 lines · Robert P. J. Day
Quoted 4 lines · Christopher Larson
I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to
IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to useIMAGE_INSTALL_appendif anything, even back in oe classic :) --a quick grep in openembedded-core/meta shows:
$ grep -r "IMAGE_INSTALL +=" * recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils" recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\ recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL += "kernel-dev" recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\ recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\ recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev" recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect" recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev" recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL += "packagegroup-core-x11-sato-games" $
Quoted 1 line
now i'm scared to look. :-)
This is what they should be doing in most cases. The only time you need _append is when you're defining it from local.conf, because the behavior of IMAGE_INSTALL += in local.conf depends on what the recipe is doing (does it IMAGE_INSTALL = (overrides the user's value), ?= (only uses the user's value, not its own), or += (adds to an already set value)). This is where the value lies in providing the user explicit variables with known semantics, so the user doesn't have to know these peculiarities and can just set the variable already :)
That said, this is definitely another case of many you've pointed out where our recipes just aren't as consistent as would be ideal. IMO when a recipe takes a list variable like this and does just an '=' it's basically saying "I must have only these things, throw away anything already set", so the only time it should be set that way is when there's such an explicit need. Whereas most of the time what they really wanted to say was "I need these things", in which case += is more appropriate (and I'm not just talking about IMAGE_INSTALL here, but also variables like RDEPENDS and DEPENDS) :) But that's just my personal view on the subject.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 26 lines · Gary Thomas
I've always used '
IMAGE_INSTALL+= " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL+= " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.I then tried
CORE_IMAGE_EXTRA_INSTALL+= " strace"
which produced a perfectly working build (just as previous) with the new package added.I can see that the images produced are vastly different:
* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34
core-image-sato-qemuarm-20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2* After '
IMAGE_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55
core-image-sato-qemuarm-20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2
Quoted 7 lines
* After '
CORE_IMAGE_EXTRA_INSTALL+= '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05
core-image-sato-qemuarm-20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2What's the difference and why did the first attempt fail?
i'm going to guess this is related to my recent whining about how
weirdly core-image-* recipes are defined. if i use the "bb" command to see what will be installed on a qemuarm in a core-image-sato image, i see:
$ bb show IMAGE_INSTALL -r core-image-sato
Parsing recipes..done.
# IMAGE_INSTALL=${CORE_IMAGE_BASE_INSTALL} packagegroup-core-x11-sato-games
IMAGE_INSTALL="packagegroup-core-boot packagegroup-base-extended \
packagegroup-core-x11-sato-games"
$which looks good. but the instant i add the line:
IMAGE_INSTALL += "strace" to my local.conf ... whoops:
$ bb show IMAGE_INSTALL -r core-image-sato Parsing recipes..done. IMAGE_INSTALL=" strace packagegroup-core-x11-sato-games" $
note the loss of the essential core-boot and base-extended
packagegroups, which is why your image is broken, which is because of the way IMAGE_INSTALL is assigned in core-image.bbclass.
lesson: do not mess with IMAGE_INSTALL directly unless you truly
know what you're doing.
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================