Skip to contentSkip to conversation

Extending images

DISCUSSION Gary Thomas 3 participants 9 messages Jul 18, 2014

lore
Share & tools
lore
#0 lore raw

I've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.

I then tried
CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous) with the new package added.

I can see that the images produced are vastly different:

* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2

* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-20140718155134.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2

* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2

What's the difference and why did the first attempt fail?

Thanks

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 26 lines · Gary Thomas

've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.

I then tried
CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous) with the new package added.

I can see that the images produced are vastly different:

* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34
core-image-sato-qemuarm-20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2

* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55
core-image-sato-qemuarm-20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2

Quoted 8 lines

* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05
core-image-sato-qemuarm-20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2

What's the difference and why did the first attempt fail?

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#8 lore raw
In reply to Gary Thomas
Quoted 26 lines · Gary Thomas

I've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.

I then tried
CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous) with the new package added.

I can see that the images produced are vastly different:

* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34
core-image-sato-qemuarm-20140718124453.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-20140718124453.rootfs.tar.bz2

* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55
core-image-sato-qemuarm-20140718155134.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-20140718155134.rootfs.tar.bz2

Quoted 7 lines

* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05
core-image-sato-qemuarm-20140718160108.rootfs.manifest -rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-20140718160108.rootfs.tar.bz2

What's the difference and why did the first attempt fail?

i'm going to guess this is related to my recent whining about how
weirdly core-image-* recipes are defined. if i use the "bb" command to see what will be installed on a qemuarm in a core-image-sato image, i see:

$ bb show IMAGE_INSTALL -r core-image-sato
Parsing recipes..done.
# IMAGE_INSTALL=${CORE_IMAGE_BASE_INSTALL} packagegroup-core-x11-sato-games
IMAGE_INSTALL="packagegroup-core-boot packagegroup-base-extended \
  packagegroup-core-x11-sato-games"
$

which looks good. but the instant i add the line:

IMAGE_INSTALL += "strace" to my local.conf ... whoops:

$ bb show IMAGE_INSTALL -r core-image-sato
Parsing recipes..done.
IMAGE_INSTALL=" strace packagegroup-core-x11-sato-games"
$

note the loss of the essential core-boot and base-extended
packagegroups, which is why your image is broken, which is because of the way IMAGE_INSTALL is assigned in core-image.bbclass.

lesson: do not mess with IMAGE_INSTALL directly unless you truly
know what you're doing.

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#2 lore raw
In reply to Christopher Larson
Quoted 29 lines · Christopher Larson

On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:

've always used 'IMAGE_INSTALL += " xyz"' in my local.conf to add new packages to a build. That said, I had a working build for qemuarm (probably doesn't matter) and I added:
IMAGE_INSTALL += " strace"
This produced a completely broken image which barely came up to a shell, lots of missing programs, etc.

I then tried
CORE_IMAGE_EXTRA_INSTALL += " strace"
which produced a perfectly working build (just as previous) with the new package added.

I can see that the images produced are vastly different:

* Original working build
-rw-r--r-- 1 gthomas gthomas 11719 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87611203 Jul 18 09:34 core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2

* After 'IMAGE_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 9986 Jul 18 09:55 core-image-sato-qemuarm-__20140718155134.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 37859884 Jul 18 09:56 core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2

* After 'CORE_IMAGE_EXTRA_INSTALL += '
-rw-r--r-- 1 gthomas gthomas 11738 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.manifest
-rw-r--r-- 1 gthomas gthomas 87720106 Jul 18 10:05 core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2

Quoted 6 lines

What's the difference and why did the first attempt fail?

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

That makes sense, thanks. I'll stick to using CORE_IMAGE_EXTRA_INSTALL from now on!

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
Quoted 10 lines · Gary Thomas

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining

Quoted 8 lines

it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If
the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

Quoted 2 lines

That makes sense, thanks. I'll stick to using CORE_IMAGE_EXTRA_INSTALL from now on!

That's best, indeed. This is a case where there's a real difference between variables intended for the user to override from configuration, and those which are not. IMAGE_INSTALL is a detail of how images are constructed, CORE_IMAGE_EXTRA_INSTALL is an explicit hook provided to the user to non-destructively add packages to an image, so we don't have to try to guess at what the recipe will do with it -- it has explicit semantics, so we know this. There might be value in differentiating between cases like this in the documentation, if we don't already.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#4 lore raw
In reply to Gary Thomas
Quoted 49 lines · Gary Thomas
Quoted 39 lines · Christopher Larson

On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:

    've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
    to add new packages to a build.  That said, I had a working
    build for qemuarm (probably doesn't matter) and I added:
       IMAGE_INSTALL += " strace"
    This produced a completely broken image which barely came
    up to a shell, lots of missing programs, etc.

    I then tried
       CORE_IMAGE_EXTRA_INSTALL += " strace"
    which produced a perfectly working build (just as previous)
    with the new package added.

    I can see that the images produced are vastly different:

    * Original working build
    -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
    core-image-sato-qemuarm-__20140718124453.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
    core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2

    * After 'IMAGE_INSTALL += '
    -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
    core-image-sato-qemuarm-__20140718155134.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
    core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2

    * After 'CORE_IMAGE_EXTRA_INSTALL += '
    -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
    core-image-sato-qemuarm-__20140718160108.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
    core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2

    What's the difference and why did the first attempt fail?
Quoted 8 lines

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

Quoted 2 lines

That makes sense, thanks. I'll stick to using
CORE_IMAGE_EXTRA_INSTALL from now on!

not to sound repetitively repetitive, but this is related to what i
thought was the weird way core images are defined based off of core-image.bbclass, where core-image.bbclass does *exactly* what you describe above:

CORE_IMAGE_BASE_INSTALL = '\
    packagegroup-core-boot \
    packagegroup-base-extended \
    \
    ${CORE_IMAGE_EXTRA_INSTALL} \
    '

CORE_IMAGE_EXTRA_INSTALL ?= ""

IMAGE_INSTALL ?= "${CORE_IMAGE_BASE_INSTALL}"

the danger here is that if someone has been defining/building images
for a while based off of the lower-level image.bbclass, then (i think) using "IMAGE_INSTALL +=" will work just fine, and the poor developer gets lulled into a false sense of security and carries that habit over to her first core-image and ... boom.

i know there's a standard to define variables prefixed with "EXTRA_"
meant for users to add features, and i think that's a great idea. i think the docs should strongly emphasize that users should look for the EXTRA_ variable before messing with anything else.

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#5 lore raw
Quoted 67 lines · Robert P. J. Day
Quoted 50 lines · Christopher Larson
Quoted 39 lines

On Fri, Jul 18, 2014 at 9:26 AM, Gary Thomas <[email protected] <mailto:[email protected]>> wrote:

    've always used 'IMAGE_INSTALL += " xyz"' in my local.conf
    to add new packages to a build.  That said, I had a working
    build for qemuarm (probably doesn't matter) and I added:
       IMAGE_INSTALL += " strace"
    This produced a completely broken image which barely came
    up to a shell, lots of missing programs, etc.

    I then tried
       CORE_IMAGE_EXTRA_INSTALL += " strace"
    which produced a perfectly working build (just as previous)
    with the new package added.

    I can see that the images produced are vastly different:

    * Original working build
    -rw-r--r-- 1 gthomas gthomas     11719 Jul 18 09:34
    core-image-sato-qemuarm-__20140718124453.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  87611203 Jul 18 09:34
    core-image-sato-qemuarm-__20140718124453.rootfs.tar.bz2

    * After 'IMAGE_INSTALL += '
    -rw-r--r-- 1 gthomas gthomas      9986 Jul 18 09:55
    core-image-sato-qemuarm-__20140718155134.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  37859884 Jul 18 09:56
    core-image-sato-qemuarm-__20140718155134.rootfs.tar.bz2

    * After 'CORE_IMAGE_EXTRA_INSTALL += '
    -rw-r--r-- 1 gthomas gthomas     11738 Jul 18 10:05
    core-image-sato-qemuarm-__20140718160108.rootfs.manifest
    -rw-r--r-- 1 gthomas gthomas  87720106 Jul 18 10:05
    core-image-sato-qemuarm-__20140718160108.rootfs.tar.bz2

    What's the difference and why did the first attempt fail?
Quoted 8 lines

Most likely the image defined its own IMAGE_INSTALL using ?=, so defining it yourself in the configuration data overrode its default definition, since ?= is "set only if unset". If the recipe didn't use ?=, then your IMAGE_INSTALL += would have had no effect at all. To append to IMAGE_INSTALL directly without using CORE_IMAGE_EXTRA_INSTALL you could have used IMAGE_INSTALL_append, since that's a postponed operation that happens at the end of the recipe parsing.

Quoted 2 lines

That makes sense, thanks. I'll stick to using
CORE_IMAGE_EXTRA_INSTALL from now on!

not to sound repetitively repetitive, but this is related to what i
thought was the weird way core images are defined based off of core-image.bbclass, where core-image.bbclass does *exactly* what you describe above:

CORE_IMAGE_BASE_INSTALL = '\
    packagegroup-core-boot \
    packagegroup-base-extended \
    \
    ${CORE_IMAGE_EXTRA_INSTALL} \
    '

CORE_IMAGE_EXTRA_INSTALL ?= ""

Quoted 8 lines

IMAGE_INSTALL ?= "${CORE_IMAGE_BASE_INSTALL}"

the danger here is that if someone has been defining/building images
for a while based off of the lower-level image.bbclass, then (i think) using "IMAGE_INSTALL +=" will work just fine, and the poor developer gets lulled into a false sense of security and carries that habit over to her first core-image and ... boom.

I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to use IMAGE_INSTALL_append if anything, even back in oe classic :)

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
#6 lore raw
Quoted 4 lines · Christopher Larson

I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to use IMAGE_INSTALL_append if anything, even back in oe classic :) --

a quick grep in openembedded-core/meta shows:

$ grep -r "IMAGE_INSTALL +=" *
recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL += "mtd-utils"
recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL += "kernel-dev"
recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\
recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\
recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests hwlatdetect kernel-dev"
recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev"
recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
$

now i'm scared to look. :-)

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#7 lore raw
Quoted 25 lines · Robert P. J. Day
Quoted 4 lines · Christopher Larson

I really don't think this was ever safe, as it completely depended on how the image recipe added its bits to IMAGE_INSTALL, which could easily vary from image to image. I've *always* told folks to use IMAGE_INSTALL_append if anything, even back in oe classic :) --

a quick grep in openembedded-core/meta shows:

$ grep -r "IMAGE_INSTALL +=" *
recipes-core/images/core-image-minimal-mtdutils.bb:IMAGE_INSTALL +=
"mtd-utils"
recipes-extended/images/core-image-testmaster.bb:IMAGE_INSTALL += "\
recipes-extended/images/core-image-lsb-sdk.bb:IMAGE_INSTALL +=
"kernel-dev"
recipes-graphics/images/core-image-directfb.bb:IMAGE_INSTALL += "\
recipes-qt/images/qt4e-demo-image.bb:IMAGE_INSTALL += "\
recipes-rt/images/core-image-rt-sdk.bb:IMAGE_INSTALL += "rt-tests
hwlatdetect kernel-dev"
recipes-rt/images/core-image-rt.bb:IMAGE_INSTALL += "rt-tests hwlatdetect"
recipes-sato/images/core-image-sato-sdk.bb:IMAGE_INSTALL += "kernel-dev"
recipes-sato/images/core-image-sato.bb:IMAGE_INSTALL +=
"packagegroup-core-x11-sato-games"
$
Quoted 1 line

now i'm scared to look. :-)

This is what they should be doing in most cases. The only time you need _append is when you're defining it from local.conf, because the behavior of IMAGE_INSTALL += in local.conf depends on what the recipe is doing (does it IMAGE_INSTALL = (overrides the user's value), ?= (only uses the user's value, not its own), or += (adds to an already set value)). This is where the value lies in providing the user explicit variables with known semantics, so the user doesn't have to know these peculiarities and can just set the variable already :)

That said, this is definitely another case of many you've pointed out where our recipes just aren't as consistent as would be ideal. IMO when a recipe takes a list variable like this and does just an '=' it's basically saying "I must have only these things, throw away anything already set", so the only time it should be set that way is when there's such an explicit need. Whereas most of the time what they really wanted to say was "I need these things", in which case += is more appropriate (and I'm not just talking about IMAGE_INSTALL here, but also variables like RDEPENDS and DEPENDS) :) But that's just my personal view on the subject.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics