Skip to contentSkip to conversation

The role of "distro" and "image"

DISCUSSION Darren Hart 6 participants 11 messages Dec 13, 2011 – Dec 15, 2011

lore
Share & tools
lore
#0 lore raw

I'm trying to wrap up my work on meta-tiny and integrate it into poky proper. I'm having some difficulty drawing a line between the responsibility of the distro definition versus that of the image definition.

For example, if I define a distro which uses tmpdevfs (no udev or mdev) and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of the linux-yocto kernel, and customizes the busybox build to leave out various things - would it make sense for someone to then try to build core-image-sato with it? It doesn't to me, but nothing prevents the user from doing that (other than a likely build failure).

So what does the distro define and what does the image define?

Digging around in conf/distro for meta and meta-yocto, I see things like naming convention, additional dependencies, choice of toolchain and libc, source mirrors, default virtual providers, etc.

The image seems to basically define the package list for the image as well as the format of the rootfs and boot media.

If I understand this correctly, a new "tiny" distro definition would change the preferred linux-yocto provider to a linux-yocto-tiny recipe (which would in turn specify the default KMACHINE/KTYPE as well, the TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

I currently define a new task-core-tiny.bb to reset some MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny) and redefine the REDEPENDS for the image. I believe I could do this instead with assignments in the new distro definition and reuse task-core-boot.bb.

As for images, I might be able to reuse core-image-minimal - but it oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since core-image-minimal.bb is defined in oe-core and POKY is a distro notion of meta-yocto, this contributes to my confusion over distro vs. image.

I'd appreciate some discussion to help me clarify where these lines should be.

Thanks,

Signature · 3 lines
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
#1 lore raw
Quoted 10 lines · Darren Hart

I'm trying to wrap up my work on meta-tiny and integrate it into poky proper. I'm having some difficulty drawing a line between the responsibility of the distro definition versus that of the image definition.

For example, if I define a distro which uses tmpdevfs (no udev or mdev) and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of the linux-yocto kernel, and customizes the busybox build to leave out various things - would it make sense for someone to then try to build core-image-sato with it? It doesn't to me, but nothing prevents the user from doing that (other than a likely build failure).

These all sounds like policy (and therefore distro) decisions to me. As with much of our system there's no hard and fast rule but this certainly sounds like a distro to me.

I think it's perfectly reasonable to assume that you can't build every image you find in the wild for every distro configuration you create.

Quoted 2 lines

So what does the distro define and what does the image define?

Distro defines the build configuration policy, image defines image contents.

You shouldn't set anything in the image definition which affects how things are built as the packages can then be used in another image.

Quoted 4 lines

Digging around in conf/distro for meta and meta-yocto, I see things like naming convention, additional dependencies, choice of toolchain and libc, source mirrors, default virtual providers, etc.

*nod*

Quoted 3 lines

The image seems to basically define the package list for the image as well as the format of the rootfs and boot media.

*nod* though arguably the latter 3 are likely poked elsewhere also.

Quoted 4 lines

If I understand this correctly, a new "tiny" distro definition would change the preferred linux-yocto provider to a linux-yocto-tiny recipe (which would in turn specify the default KMACHINE/KTYPE as well, the TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

Yes!

Quoted 5 lines

I currently define a new task-core-tiny.bb to reset some MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny) and redefine the REDEPENDS for the image. I believe I could do this instead with assignments in the new distro definition and reuse task-core-boot.bb.

Seems like a good approach.

Quoted 7 lines

As for images, I might be able to reuse core-image-minimal - but it oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since core-image-minimal.bb is defined in oe-core and POKY is a distro notion of meta-yocto, this contributes to my confusion over distro vs. image.

I'd appreciate some discussion to help me clarify where these lines should be.

That's just a stray variable that has yet to be properly renamed by the looks of it.

It sounds to me like your understanding is correct and this variable has just thrown you for a loop.

Cheers,
Joshua

Signature · 3 lines
Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre
#2 lore raw
Quoted 11 lines · Joshua Lock
Quoted 8 lines

As for images, I might be able to reuse core-image-minimal - but it oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since core-image-minimal.bb is defined in oe-core and POKY is a distro notion of meta-yocto, this contributes to my confusion over distro vs. image.

I'd appreciate some discussion to help me clarify where these lines should be.

That's just a stray variable that has yet to be properly renamed by the looks of it.

yes there was a patch to rename that to IMAGE_EXTRA_INSTALL or COREIMAGE_EXTRA_INSTALL

#3 lore raw
In reply to Darren Hart
Quoted 4 lines · Darren Hart

I'm trying to wrap up my work on meta-tiny and integrate it into poky proper. I'm having some difficulty drawing a line between the responsibility of the distro definition versus that of the image definition.

Quoted 8 lines

For example, if I define a distro which uses tmpdevfs (no udev or mdev) and specifies KMACHINE=yocto/standard/tiny to build the tiny variant of the linux-yocto kernel, and customizes the busybox build to leave out various things - would it make sense for someone to then try to build core-image-sato with it? It doesn't to me, but nothing prevents the user from doing that (other than a likely build failure).

So what does the distro define and what does the image define?

I think Joshua's response is in line with what I'm thinking. Distro is where all the policy is specified. The image is just a list of pieces of software to include, its not meant to be much more than that.

Quoted 3 lines

Digging around in conf/distro for meta and meta-yocto, I see things like naming convention, additional dependencies, choice of toolchain and libc, source mirrors, default virtual providers, etc.

Right, all of this is "policy" at some level or another.

Quoted 2 lines

The image seems to basically define the package list for the image as well as the format of the rootfs and boot media.

The latter is often machine specific. There are some images which specify those things but they're a minority.

Quoted 4 lines

If I understand this correctly, a new "tiny" distro definition would change the preferred linux-yocto provider to a linux-yocto-tiny recipe (which would in turn specify the default KMACHINE/KTYPE as well, the TCLIBC, DISTRO_FEATURES_LIBC, and some naming rules.

Yes, you'd inherit something for the base config and then customise it.

Quoted 5 lines

I currently define a new task-core-tiny.bb to reset some MACHINE_ESSENTIAL bits (qemu pulls in more than is necessary for tiny) and redefine the REDEPENDS for the image. I believe I could do this instead with assignments in the new distro definition and reuse task-core-boot.bb.

The latter reuse of task-core-boot would be ideal. If that can't work I'd be interested to see why and if we couldn't enhance it so it could work.

Quoted 4 lines

As for images, I might be able to reuse core-image-minimal - but it oddly contains "${POKY_EXTRA_INSTALL}" in the IMAGE_INSTALL. Since core-image-minimal.bb is defined in oe-core and POKY is a distro notion of meta-yocto, this contributes to my confusion over distro vs. image.

This is a bug and there are some patches out there to turn it into COREIMAGE_EXTRA_INSTALL. Its a feature of the core-image.bbclass (formerly poky.bbclass, hence the name confusion).

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Cheers,

Richard

#4 lore raw
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Signature · 1 line
Christopher Larson
#5 lore raw

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

We had a similar problem at work, people were sprinkling COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

regards,

Koen

#6 lore raw
Quoted 19 lines · Koen Kooi

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Quoted 1 line

We had a similar problem at work, people were sprinkling COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Haha, ouch. It's also worth taking a moment to emphasize that the way distro/machine/image is structured is by design, not accidental. Having these pieces be orthogonal buys us a great deal of flexibility and capability, which is why we did it this way. Now, whether a given distro/machine/image combination is *useful* is a different question, and one I think is best addressed via policy / documentation.

Signature · 1 line
Christopher Larson
#7 lore raw
Quoted 25 lines · Chris Larson
Quoted 19 lines · Koen Kooi

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Quoted 1 line

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Quoted 6 lines

Haha, ouch. It's also worth taking a moment to emphasize that the way distro/machine/image is structured is by design, not accidental. Having these pieces be orthogonal buys us a great deal of flexibility and capability, which is why we did it this way. Now, whether a given distro/machine/image combination is *useful* is a different question, and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider usability a little. We have people using the system to whom it might not be immediately obvious why building a sato image with the tiny distro is a bad idea.

If you know enough to be able to make that work, the warning is easily bypassed and isn't going to bother you.

So I think there is a case for warning the user if the combination they're selecting is not one that is likely to work and we can easily predict it. There are many combinations we can't predict which is fine.

Like any mechanism, it can be abused of course and using COMPATIBLE_MACHINE as a sign of recipe testing clearly isn't a good idea or what it was designed for.

Cheers,

Richard

#8 lore raw
Quoted 38 lines · Richard Purdie
Quoted 25 lines · Chris Larson
Quoted 19 lines · Koen Kooi

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Quoted 1 line

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Quoted 6 lines

Haha, ouch. It's also worth taking a moment to emphasize that the way distro/machine/image is structured is by design, not accidental. Having these pieces be orthogonal buys us a great deal of flexibility and capability, which is why we did it this way. Now, whether a given distro/machine/image combination is *useful* is a different question, and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider usability a little. We have people using the system to whom it might not be immediately obvious why building a sato image with the tiny distro is a bad idea.

Quoted 6 lines

If you know enough to be able to make that work, the warning is easily bypassed and isn't going to bother you.

So I think there is a case for warning the user if the combination they're selecting is not one that is likely to work and we can easily predict it. There are many combinations we can't predict which is fine.

There's a rather large difference between *warning* the user and raising SkipPackage, making it not possible to build without modifying the recipe.

Signature · 1 line
Christopher Larson
#9 lore raw
Quoted 43 lines · Chris Larson
Quoted 41 lines · Richard Purdie
Quoted 25 lines · Chris Larson
Quoted 19 lines · Koen Kooi

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Quoted 1 line

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Quoted 6 lines

Haha, ouch. It's also worth taking a moment to emphasize that the way distro/machine/image is structured is by design, not accidental. Having these pieces be orthogonal buys us a great deal of flexibility and capability, which is why we did it this way. Now, whether a given distro/machine/image combination is *useful* is a different question, and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider usability a little. We have people using the system to whom it might not be immediately obvious why building a sato image with the tiny distro is a bad idea.

If you know enough to be able to make that work, the warning is easily bypassed and isn't going to bother you.

Quoted 8 lines
Quoted 3 lines

So I think there is a case for warning the user if the combination they're selecting is not one that is likely to work and we can easily predict it. There are many combinations we can't predict which is fine.

There's a rather large difference between *warning* the user and raising SkipPackage, making it not possible to build without modifying the recipe.

What's needed is something which results in a "stop the build and force the user to change something to continue" so I'm not sure I see the problem with that.

I know exactly how much attention people pay to the WARNING messages :( (those are getting cleaned up so hopefully those WARNINGS that are still present become more meaningful/serious)

Cheers,

Richard

#10 lore raw
Quoted 49 lines · Richard Purdie
Quoted 48 lines · Chris Larson
Quoted 44 lines · Richard Purdie
Quoted 25 lines · Chris Larson
Quoted 19 lines · Koen Kooi

Op 13 dec. 2011, om 22:45 heeft Chris Larson het volgende geschreven:

Quoted 9 lines
Quoted 6 lines · Richard Purdie

Not all images can be built with a given distro, our base config is rather ride ranging though for obvious reasons. You could "enforce" this by adding some anonymous python to the distro which does something like:

if inherits("image") and PN != core-image-minimal:
   raise SkipPackage("Image PN not compatible with DISTRO=XXX")

Quoted 7 lines

This is a case where one is best 'controlling' this via policy, not mechanism, in my opinion. This sort of arbitrary technical limitation tends to be foolish and often bites someone somewhere down the line. I'm sure you just wanted to note how it could be done, not recommend that it should be done, but I thought it should be made clear. I wouldn't recommend that anyone do this unless there is an extremely good reason for it.

Quoted 1 line

We had a similar problem at work, people were sprinkling

COMPATIBLE_MACHINE left and right just to mark "I tested recipe X on machine Y". You can guess what happened when new machines needed to get added.

Quoted 6 lines

Haha, ouch. It's also worth taking a moment to emphasize that the way distro/machine/image is structured is by design, not accidental. Having these pieces be orthogonal buys us a great deal of flexibility and capability, which is why we did it this way. Now, whether a given distro/machine/image combination is *useful* is a different question, and one I think is best addressed via policy / documentation.

I understand the flexibility but I also think we need to consider usability a little. We have people using the system to whom it might not be immediately obvious why building a sato image with the tiny distro is a bad idea.

If you know enough to be able to make that work, the warning is easily bypassed and isn't going to bother you.

So I think there is a case for warning the user if the combination they're selecting is not one that is likely to work and we can easily predict it. There are many combinations we can't predict which is fine.

Quoted 7 lines
Quoted 3 lines

There's a rather large difference between *warning* the user and raising SkipPackage, making it not possible to build without modifying the recipe.

What's needed is something which results in a "stop the build and force the user to change something to continue" so I'm not sure I see the problem with that.

I strongly disagree. That's exactly what we shouldn't have, at least unless there's an easy way to disable the behavior. Even our sanity checks can be opted out, and those are far more critical than building a rootfs that might not behave in a useful fashion on a particular piece of hardware.

Signature · 1 line
Christopher Larson