Skip to contentSkip to conversation

SDK question: does "-c populate_sdk" build SDK based on entire image?

DISCUSSION Robert P. J. Day 5 participants 8 messages May 7, 2021 – May 12, 2021

lore
Share & tools
lore
#0 lore raw

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

now, i realize that if i just want to add content to the SDK without
adding to target, i can use one or both of:

TOOLCHAIN_HOST_TASK
TOOLCHAIN_TARGET_TASK

but if someone asks for new content both for the target and the SDK, is it sufficient to just add to target?

and one more question that inspired all of this. i was asked to add
a particular executable to an SDK, call it /usr/bin/fubar. fair enough, i thought, it was short work to determine that that binary came from the recipe fubar_1.0.bb. but i checked and that package was, in fact, being installed in the target, so i was puzzled as to why it wasn't in the SDK as well.

i checked the recipe for fubar and, to my chagrin, it contained the
snippet:

  do_install_append() {
    rm -rf ${D}/usr/bin/fubar
  }

well, that explained it, so as a first test, i'm just going to dump that snippet from the recipe, which will install the executable on the target, and will(?) also install it in the SDK?

rday

#1 lore raw

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

Quoted 5 lines · Robert P. J. Day

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 9 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an eSDK. An SDK just gives you the minimal toolchain and associated tools, to be able to compile something. I would go as far as saying it's basically what a PC distro package for cross-compile with gcc is doing.

Most of headers, shlib and binaries from an image won't make it to the SDK. A much bigger portion of those will be installed in an SDK, though not all of them. I have never used an eSDK so can't comment on what's missing or how to add packages to the eSDK.

Quoted 9 lines

now, i realize that if i just want to add content to the SDK without
adding to target, i can use one or both of:

TOOLCHAIN_HOST_TASK
TOOLCHAIN_TARGET_TASK

but if someone asks for new content both for the target and the SDK, is it sufficient to just add to target?

I'd assume that TOOLCHAIN_HOST_TASK is for native packages and TOOLCHAIN_TARGET_TASK for target packages, for an SDK, TOOLCHAIN_HOST_TASK would contain arm-linux-gcc, make, cmake, whatever common build tools needed and TOOLCHAIN_TARGET_TASK headers/shlib for compiling binaries to be run on your target, compiled via tools from TOOLCHAIN_HOST_TASK.

Not sure this is remotely correct, just some thoughts.

Hope this can help/guide a bit?

Cheers,
Quentin

#2 lore raw
Quoted 20 lines · Quentin Schulz

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

Quoted 5 lines · Robert P. J. Day

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 9 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

Quoted 5 lines

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an eSDK. An SDK just gives you the minimal toolchain and associated tools, to be able to compile something. I would go as far as saying it's basically what a PC distro package for cross-compile with gcc is doing.

... snip ...

i suspect you're right, i've been reading both sections in the SDK manual and have probably just started mixing them up. i swear, once i figure out how it all works, i'll rewrite the manual. :-)

rday

#3 lore raw
In reply to Quentin Schulz
Quoted 20 lines · Quentin Schulz

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

Quoted 5 lines · Robert P. J. Day

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 9 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

Quoted 5 lines

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an eSDK. An SDK just gives you the minimal toolchain and associated tools, to be able to compile something. I would go as far as saying it's basically what a PC distro package for cross-compile with gcc is doing.

... snip ...

it's slowly coming back to me, as i once examined the recipe meta-toolchain.bb for building a toolchain, whose contents are effectively:

inherit populate_sdk

and when you run "bitbake meta-toolchain", you're obviously not supplying a particular image as an argument, so whatever is being done has to be based on fundamental information like MACHINE and so on, with no reference to a particular image so, yes, you'd get a "minimal" toolchain as you describe above independent of any image.

however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
and that latter class file contains:

TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}" TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy" TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"

it seems clear that you can augment that minimal toolchain with exactly the two variables i mentioned earlier. ok, i think i've refreshed my memory on this. now off to figure out the eSDK.

rday

#4 lore raw
Quoted 52 lines · Robert P. J. Day
Quoted 20 lines · Quentin Schulz

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

Quoted 5 lines · Robert P. J. Day

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 9 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

Quoted 5 lines

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an eSDK. An SDK just gives you the minimal toolchain and associated tools, to be able to compile something. I would go as far as saying it's basically what a PC distro package for cross-compile with gcc is doing.

   ... snip ...

it's slowly coming back to me, as i once examined the recipe meta-toolchain.bb for building a toolchain, whose contents are effectively:

inherit populate_sdk

and when you run "bitbake meta-toolchain", you're obviously not supplying a particular image as an argument, so whatever is being done has to be based on fundamental information like MACHINE and so on, with no reference to a particular image so, yes, you'd get a "minimal" toolchain as you describe above independent of any image.

however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
and that latter class file contains:

TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}" TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy" TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"

Quoted 3 lines

it seems clear that you can augment that minimal toolchain with exactly the two variables i mentioned earlier. ok, i think i've refreshed my memory on this. now off to figure out the eSDK.

Didn't we discuss exactly this (ie the difference between a pure SDK recipe such as meta-toolchain and the populate_sdk task of an image recipe) a couple of weeks ago?

Both are valid ways to create an SDK and there are no rules about one or the other being specific to creating a "minimal" toolchain. Basically a pure SDK recipe gives you full control while the populare_sdk task of an image recipe is easier to set up (no new recipe to write) at the expense of the final SDK containing extra junk such as busybox init scripts, etc which are needed in the image but not an SDK.

#5 lore raw
Quoted 3 lines · Andre McCurdy

Didn't we discuss exactly this (ie the difference between a pure SDK recipe such as meta-toolchain and the populate_sdk task of an image recipe) a couple of weeks ago?

yes, i just got confused as i was digging into eSDK and mixed the
two. it's been a long week.

rday

#6 lore raw
In reply to Andre McCurdy
Quoted 61 lines · Andre McCurdy
Quoted 52 lines · Robert P. J. Day
Quoted 20 lines · Quentin Schulz

Hi Robert,

No SDK expert here, so as usual, to be taken with a grain of salt.

Quoted 5 lines · Robert P. J. Day

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 9 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

Quoted 5 lines

I think you're mixing SDK and eSDK. AFAIU so far, you're looking for an eSDK. An SDK just gives you the minimal toolchain and associated tools, to be able to compile something. I would go as far as saying it's basically what a PC distro package for cross-compile with gcc is doing.

   ... snip ...

it's slowly coming back to me, as i once examined the recipe meta-toolchain.bb for building a toolchain, whose contents are effectively:

inherit populate_sdk

and when you run "bitbake meta-toolchain", you're obviously not supplying a particular image as an argument, so whatever is being done has to be based on fundamental information like MACHINE and so on, with no reference to a particular image so, yes, you'd get a "minimal" toolchain as you describe above independent of any image.

however, as populate_sdk.bbclass inherits populate_sdk_base.bbclass,
and that latter class file contains:

TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}" TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy" TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-toolchain-${SDK_VERSION}"

Quoted 3 lines

it seems clear that you can augment that minimal toolchain with exactly the two variables i mentioned earlier. ok, i think i've refreshed my memory on this. now off to figure out the eSDK.

Didn't we discuss exactly this (ie the difference between a pure SDK recipe such as meta-toolchain and the populate_sdk task of an image recipe) a couple of weeks ago?

Quoted 7 lines

Both are valid ways to create an SDK and there are no rules about one or the other being specific to creating a "minimal" toolchain. Basically a pure SDK recipe gives you full control while the populare_sdk task of an image recipe is easier to set up (no new recipe to write) at the expense of the final SDK containing extra junk such as busybox init scripts, etc which are needed in the image but not an SDK.

Exactly! Thanks for this excellent summary!

I've heard so many times before that my meta-toolchain recipe is not the "Yocto" way of doing things and I should instead use populate_sdk. I got tired arguing with people and explaining that meta-toolchain is quite valid and has its merits...

Signature · 4 lines
Regards,
Denys Dmytriyenko <[email protected]>
PGP: 0x420902729A92C964 - https://denix.org/0x420902729A92C964
Fingerprint: 25FC E4A5 8A72 2F69 1186  6D76 4209 0272 9A92 C964
#7 lore raw
In reply to Robert P. J. Day
Quoted 14 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Robert P. J. Day via lists.openembedded.org
Sent: 07 May 2021 15:12
To: OE Core mailing list <[email protected]>
Subject: [OE-core] SDK question: does "-c populate_sdk" build SDK based on entire image?

almost certainly a silly question as i'm still poring over the
mechanics of standard SDK creation, but if i define a perfectly normal image, then build the corresponding SDK with:

Quoted 8 lines
  $ bitbake -c populate_sdk my_image

is the resulting SDK populated based on the entire contents of the target image? that is, if i subsequently add a new package to the target and rebuild the SDK, will the new SDK now contain the corresponding content from that newly-added package? (i'm just about to test this with a build but that's going to take over an hour on this server. *sigh* ...)

We use the SDK a lot to give application developers an easy to use a dev environment which has all the support to build against the libraries included in target images. The SDK of an image recipe will contain all the *-dev packages matching the packages installed in the target for the same image (i.e. if foobar is installed in the target image then foobar-dev is installed in the SDK image, if it exists).

Quoted 9 lines

now, i realize that if i just want to add content to the SDK without
adding to target, i can use one or both of:

TOOLCHAIN_HOST_TASK
TOOLCHAIN_TARGET_TASK

but if someone asks for new content both for the target and the SDK, is it sufficient to just add to target?

Yes, with the exception of recipes for which only the headers are used when compiling a recipe that is included in the target image. As no library is being linked for it then no package is installed into the target, so no corresponding *-dev package is included in the SDK. A typical example of this is boost when the header-only boost libs are used (and I suspect the same happens for static libs). In such a case you can add the recipes to TOOLCHAIN_TARGET_TASK (we actually have an sdk packagegroup that we add to TOOLCHAIN_TARGET_TASK that pulls in all target SDK things that we want in the SDK but not in the target image).

Other things that you may need to add manually to TOOLCHAIN_HOST_TASK are supporting tools to run on the build host which are not included in the SDK by default, like nativesdk-glib-2.0-codegen to get the gdbus-codegen utility.

Quoted 20 lines

and one more question that inspired all of this. i was asked to add
a particular executable to an SDK, call it /usr/bin/fubar. fair enough, i thought, it was short work to determine that that binary came from the recipe fubar_1.0.bb. but i checked and that package was, in fact, being installed in the target, so i was puzzled as to why it wasn't in the SDK as well.

i checked the recipe for fubar and, to my chagrin, it contained the
snippet:

  do_install_append() {
    rm -rf ${D}/usr/bin/fubar
  }

well, that explained it, so as a first test, i'm just going to dump that snippet from the recipe, which will install the executable on the target, and will(?) also install it in the SDK?

rday

Signature · 3 lines
Diego Santa Cruz, PhD
Technology Architect
spinetix.com