Skip to contentSkip to conversation

dev-manual: correct note about virtual providers

PATCH In review Michael Opdenacker 3 participants 9 messages Aug 30, 2021 – Sep 1, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dev-manual: correct note about virtual providers

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +4 -3
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 
 .. note::
 
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 
 Properly Versioning Pre-Release Recipes
 ---------------------------------------
Signature · 1 line
2.25.1
#1 lore raw

Hi Michael,

Quoted 14 lines · Michael Opdenacker

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +0 -0
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 

documentation/dev-manual/common-tasks.rst The following lists specific examples of virtual providers: · line 2121

Quoted 10 lines
documentation/dev-manual/common-tasks.rst +4 -3
 .. note::
 
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 

I'm not entirely sure what we wanted to tell initially but I feel like the following could be a bit clearer?

"
Virtual providers can be used not only for build time dependencies (:term:DEPENDS) when specified in :term:PROVIDES, as shown in the example above, but also for runtime dependencies (:term:RDEPENDS) when specified in :term:RPROVIDES.
"

Also wondering if we shouldn't use em dashes instead of commas here.

Cheers,
Quentin

Quoted 9 lines
documentation/dev-manual/common-tasks.rst +0 -0
 Properly Versioning Pre-Release Recipes
 ---------------------------------------
Signature · 1 line
2.25.1
#2 lore raw
In reply to Michael Opdenacker
Quoted 14 lines · Michael Opdenacker

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +0 -0
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 

documentation/dev-manual/common-tasks.rst The following lists specific examples of virtual providers: · line 2121

Quoted 10 lines
documentation/dev-manual/common-tasks.rst +4 -3
 .. note::
 
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 

This isn't correct. virtual providers of the form virtual/ *only* apply to the build namespace, not the runtime one.

Cheers,

Richard

#3 lore raw
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Michael Opdenacker

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +0 -0
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 
 .. note::
 
Quoted 12 lines

documentation/dev-manual/common-tasks.rst The following lists specific examples of virtual providers: · line 2121

Quoted 8 lines
documentation/dev-manual/common-tasks.rst +4 -3
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 

This isn't correct. virtual providers of the form virtual/ *only* apply to the build namespace, not the runtime one.

What is the mechanism behind RPROVIDES:${PN} then? You could technically set it to "virtual/something" right? How Bitbake chooses between two recipes creating a package RPROVIDES'ing the same "virtual/something" is something I have no knowledge in but I feel like it's something that should be explained here, or at least explicit why it does not apply to runtime dependencies or point to where the providers for runtime dependencies are explained in the docs.

Cheers,
Quentin

#4 lore raw
Quoted 31 lines · Quentin Schulz
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Michael Opdenacker

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +0 -0
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 
 .. note::
 
Quoted 12 lines

documentation/dev-manual/common-tasks.rst The following lists specific examples of virtual providers: · line 2121

Quoted 8 lines
documentation/dev-manual/common-tasks.rst +4 -3
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 

This isn't correct. virtual providers of the form virtual/ *only* apply to the build namespace, not the runtime one.

Quoted 2 lines

What is the mechanism behind RPROVIDES:${PN} then? You could technically set it to "virtual/something" right?

There is no mechanism and whilst you could set it to that, it wouldn't do anything useful.

Quoted 2 lines

How Bitbake chooses between two recipes creating a package RPROVIDES'ing the same "virtual/something" is something I have no knowledge in

You don't set that. It doesn't work. It isn't supported. This is what that manual section is trying to say. Making the manual pretend it does doesn't change this.

Quoted 4 lines

but I feel like it's something that
should be explained here, or at least explicit why it does not apply to runtime dependencies or point to where the providers for runtime dependencies are explained in the docs.

I've explained this many times before but let me try once again.

Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple providers and choose between them. We have full control over that piece of the process.

Bitbake picks the combination it thinks the user wants and then bakes it into the task graph. For any given configuration, one task graph should result.

The runtime namespace is quite different. RPROVIDES are written into the packages and they are interpreted by the package managers. We have no way to tell the package manager that when it sees "virtual/shell", it should be bash. It would be something different for each package manager and at best it is badly supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've therefore not gone there.

It is understandable the user still needs to make some choices in OE. These are the VIRTUAL-RUNTIME_xxxx variables. These are the virtual/ equivalent from DEPENDS/PROVIDES. The distro hardcodes values of things for the VIRTUAL- RUNTIME_xxx variables. This is to try and make it clear these are not like virtual/ and do not respect PREFERRED_PROVIDER. The selections are hardcoded into packages as they are built.

VIRTUAL-RUNTIME is ugly and it is an OE thing, not a bitbake one, there is nothing related to it in bitbake. This does make it harder to document.

I'd love something nicer but we've never found a way to make it work reliably with the package managers, deterministic builds and all our other constraints.

Cheers,

Richard

#5 lore raw

Hi Richard,

Quoted 20 lines · Richard Purdie
Quoted 3 lines

How Bitbake chooses between two recipes creating a package RPROVIDES'ing the same "virtual/something" is something I have no knowledge in

You don't set that. It doesn't work. It isn't supported. This is what that manual section is trying to say. Making the manual pretend it does doesn't change this.

Quoted 4 lines

but I feel like it's something that
should be explained here, or at least explicit why it does not apply to runtime dependencies or point to where the providers for runtime dependencies are explained in the docs.

I've explained this many times before but let me try once again.

Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple providers and choose between them. We have full control over that piece of the process.

Bitbake picks the combination it thinks the user wants and then bakes it into the task graph. For any given configuration, one task graph should result.

Quoted 6 lines

The runtime namespace is quite different. RPROVIDES are written into the packages and they are interpreted by the package managers. We have no way to tell the package manager that when it sees "virtual/shell", it should be bash. It would be something different for each package manager and at best it is badly supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've therefore not gone there.

Thank you for the clarification. I got confused by the below definitions in recipes:

poky$ git grep virtual | grep RDEPENDS meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common virtual/grub-bootconf" meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} += "virtual/systemd-bootconf"

poky$ git grep virtual | grep RPROVIDES meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/grub-bootconf" meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev = "eglibc-dev libc6-dev virtual-libc-dev" meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev virtual-libc-dev" meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/systemd-bootconf"

In these cases, if I understand correctly, "virtual/xxx" are just regular strings with no special meaning that the package managers will use, right? Should such packages avoid the use of "virtual/" RPROVIDES?

Thanks again,

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw
Quoted 49 lines · Michael Opdenacker

Hi Richard,

Quoted 20 lines · Richard Purdie
Quoted 3 lines

How Bitbake chooses between two recipes creating a package RPROVIDES'ing the same "virtual/something" is something I have no knowledge in

You don't set that. It doesn't work. It isn't supported. This is what that manual section is trying to say. Making the manual pretend it does doesn't change this.

Quoted 4 lines

but I feel like it's something that
should be explained here, or at least explicit why it does not apply to runtime dependencies or point to where the providers for runtime dependencies are explained in the docs.

I've explained this many times before but let me try once again.

Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple providers and choose between them. We have full control over that piece of the process.

Bitbake picks the combination it thinks the user wants and then bakes it into the task graph. For any given configuration, one task graph should result.

Quoted 6 lines

The runtime namespace is quite different. RPROVIDES are written into the packages and they are interpreted by the package managers. We have no way to tell the package manager that when it sees "virtual/shell", it should be bash. It would be something different for each package manager and at best it is badly supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've therefore not gone there.

Thank you for the clarification. I got confused by the below definitions in recipes:

poky$ git grep virtual | grep RDEPENDS meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common virtual/grub-bootconf" meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} += "virtual/systemd-bootconf"

poky$ git grep virtual | grep RPROVIDES meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/grub-bootconf" meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev = "eglibc-dev libc6-dev virtual-libc-dev" meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev virtual-libc-dev" meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/systemd-bootconf"

Quoted 3 lines

In these cases, if I understand correctly, "virtual/xxx" are just regular strings with no special meaning that the package managers will use, right? Should such packages avoid the use of "virtual/" RPROVIDES?

The virtual/xxx should not be there and yes, are just strings.

Cheers,

Richard

#7 lore raw
Quoted 20 lines · Richard Purdie
Quoted 20 lines

Thank you for the clarification. I got confused by the below definitions in recipes:

poky$ git grep virtual | grep RDEPENDS meta/recipes-bsp/grub/grub-efi_2.06.bb:RDEPENDS:${PN} = "grub-common virtual/grub-bootconf" meta/recipes-core/systemd/systemd-boot_249.3.bb:RDEPENDS:${PN} += "virtual/systemd-bootconf"

poky$ git grep virtual | grep RPROVIDES meta/recipes-bsp/grub/grub-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/grub-bootconf" meta/recipes-core/glibc/glibc-package.inc:RPROVIDES:${PN}-dev = "eglibc-dev libc6-dev virtual-libc-dev" meta/recipes-core/musl/musl_git.bb:RPROVIDES:${PN}-dev += "libc-dev virtual-libc-dev" meta/recipes-core/systemd/systemd-bootconf_1.00.bb:RPROVIDES:${PN} += "virtual/systemd-bootconf"

Quoted 4 lines
Quoted 3 lines

In these cases, if I understand correctly, "virtual/xxx" are just regular strings with no special meaning that the package managers will use, right? Should such packages avoid the use of "virtual/" RPROVIDES?

The virtual/xxx should not be there and yes, are just strings.

Thanks for confirming this. What about replacing then "virtual/" by "virtual-" as already done in "virtual-libc-dev" above?

Cheers

Michael.

Signature · 3 lines
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw
In reply to Richard Purdie
Quoted 73 lines · Richard Purdie
Quoted 31 lines · Quentin Schulz
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Michael Opdenacker

Fixes [YOCTO #14530]

---
 documentation/dev-manual/common-tasks.rst | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
documentation/dev-manual/common-tasks.rst +0 -0
index f3b5084436..dd015a761e 100644
@@ -2114,9 +2114,10 @@ The following lists specific examples of virtual providers:
 
 .. note::
 
Quoted 12 lines

documentation/dev-manual/common-tasks.rst The following lists specific examples of virtual providers: · line 2121

Quoted 8 lines
documentation/dev-manual/common-tasks.rst +4 -3
-   Virtual providers only apply to build time dependencies specified with
-   :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
-   dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
+   Virtual providers can be applied not only to build time dependencies
+   specified with :term:`PROVIDES` and :term:`DEPENDS`, as show in the
+   example above, but also to runtime dependencies specified with
+   :term:`RPROVIDES` and :term:`RDEPENDS`.
 

This isn't correct. virtual providers of the form virtual/ *only* apply to the build namespace, not the runtime one.

Quoted 2 lines

What is the mechanism behind RPROVIDES:${PN} then? You could technically set it to "virtual/something" right?

There is no mechanism and whilst you could set it to that, it wouldn't do anything useful.

Quoted 2 lines

How Bitbake chooses between two recipes creating a package RPROVIDES'ing the same "virtual/something" is something I have no knowledge in

You don't set that. It doesn't work. It isn't supported. This is what that manual section is trying to say. Making the manual pretend it does doesn't change this.

Quoted 4 lines

but I feel like it's something that
should be explained here, or at least explicit why it does not apply to runtime dependencies or point to where the providers for runtime dependencies are explained in the docs.

I've explained this many times before but let me try once again.

Bitbake is the interpreter of DEPENDS and PROVIDES and it can see multiple providers and choose between them. We have full control over that piece of the process.

Bitbake picks the combination it thinks the user wants and then bakes it into the task graph. For any given configuration, one task graph should result.

The runtime namespace is quite different. RPROVIDES are written into the packages and they are interpreted by the package managers. We have no way to tell the package manager that when it sees "virtual/shell", it should be bash. It would be something different for each package manager and at best it is badly supported by the distros and packaging backends (e.g. dpkg-reconfigure). We've therefore not gone there.

It is understandable the user still needs to make some choices in OE. These are the VIRTUAL-RUNTIME_xxxx variables. These are the virtual/ equivalent from DEPENDS/PROVIDES. The distro hardcodes values of things for the VIRTUAL- RUNTIME_xxx variables. This is to try and make it clear these are not like virtual/ and do not respect PREFERRED_PROVIDER. The selections are hardcoded into packages as they are built.

Quoted 6 lines

VIRTUAL-RUNTIME is ugly and it is an OE thing, not a bitbake one, there is nothing related to it in bitbake. This does make it harder to document.

I'd love something nicer but we've never found a way to make it work reliably with the package managers, deterministic builds and all our other constraints.

Thanks so much for having taken the time to answer :)

I want to link to the IRC log here because I continued this discussion on IRC with Richard, please see:
https://www.yoctoproject.org/irc/%23yocto.2021-09-01.log.html#t2021-09-01T11:55:01

Cheers,
Quentin