Skip to contentSkip to conversation

doc: bitbake-user-manual-metadata: clarify inherit_defer documentation

PATCH In review Dawid Bijak 3 participants 8 messages Apr 24 – Apr 29

lore
Share & tools
lore
#0 lore raw

cover doc: bitbake-user-manual-metadata: clarify inherit_defer documentation

Hi,
I've been reading through the inherit_defer documentation, and granted my limited knowledge of bitbake, I think the current documentation is incorrect or misleading in a few places:

1)
The sentence "If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed" seems plainly wrong and defeats the purpose of inherit_defer.
As I understand it, the whole point of deferred evaluation is that assignments made after the directive are taken into account.

2)
The examples using inline Python, e.g. inherit_defer ${@'classname' if condition else ''}, are not specific to inherit_defer - they would work equally well with plain inherit. They probably belong in the inherit section.

3)

    The suggested technique of assigning the class name from an anonymous Python function does not appear to work:

    python () {
        if d.getVar('SOMETHING') == 'value':
            d.setVar('VARIABLE', 'someclass')
        else:
            d.setVar('VARIABLE', '')
    }
    inherit_defer ${VARIABLE}

    From my experiments, inherit_defer is resolved before anonymous Python functions run, so ${VARIABLE} always expands to empty
    and the inherit is a no-op.

    Is that the intended order and the documentation is wrong, or is this a bug in bitbake, or did I just get it wrong?

Anyway, just in case, the attached patch rewords the documentation to address points 1 and 2, and adds a note about point 3. Happy to revise if I've misunderstood any of it.

Thanks,
Dawid

Dawid Bijak (1):
doc: bitbake-user-manual-metadata: fix inherit_defer documentation

 .../bitbake-user-manual-metadata.rst          | 61 ++++++++++---------
 1 file changed, 32 insertions(+), 29 deletions(-)
Signature · 1 line
2.48.1
#1 lore raw

PATCH 1/1 doc: bitbake-user-manual-metadata: fix inherit_defer documentation

The documentation for inherit_defer contained the claim

"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"

which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer

Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives

Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with inherit_defer.

---
 .../bitbake-user-manual-metadata.rst          | 61 ++++++++++---------
 1 file changed, 32 insertions(+), 29 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +32 -29
index 40cae6b05..38efe6b11 100644
@@ -803,7 +803,17 @@ An advantage with the inherit directive as compared to both the
 :ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>`
 directives is that you can inherit class files conditionally. You can
 accomplish this by using a variable expression after the ``inherit``
-statement.
+statement, as in::
+
+   inherit ${@'classname' if condition else ''}
+
+Or::
+
+   inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)}
+
+In both cases, if the expression evaluates to an
+empty string, the statement does not trigger a syntax error because it
+becomes a no-op.
 
 For inheriting classes conditionally, using the :ref:`inherit_defer
 <ref-bitbake-user-manual-metadata-inherit-defer>` directive is advised as
@@ -827,39 +837,32 @@ the variable after the line is parsed will take effect. With the :ref:`inherit
 
 Here is an example::
 
-   inherit_defer ${VARNAME}
-
-If ``VARNAME`` is
-going to be set, it needs to be set before the ``inherit_defer`` statement is
-parsed. One way to achieve a conditional inherit in this case is to use
-overrides::
-
    VARIABLE = ""
-   VARIABLE:someoverride = "myclass"
-
-Another method is by using :ref:`anonymous Python
-<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`.
-Here is an example::
+   inherit_defer ${VARIABLE}
+   VARIABLE:someoverride = "someclass"
+
+:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+defers the evaluation of ``${VARIABLE}`` until the end of
+parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARIABLE}``
+expands to ``someclass``, which is then inherited. Contrast this with a plain
+:ref:`inherit <ref-bitbake-user-manual-metadata-inherit>`, which would evaluate
+``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment
+is parsed. ``${VARIABLE}`` would expand to an empty string and the statement
+would become a no-op.
+
+Note that assigning the variable from an
+:ref:`anonymous Python function <bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`
+does *not* work, because deferred inherits are resolved before anonymous
+Python functions run::
 
    python () {
-       if condition == value:
-           d.setVar('VARIABLE', 'myclass')
-       else:
-           d.setVar('VARIABLE', '')
+       if d.getVar('SOMETHING') == 'value':
+           d.setVar('VARIABLE', 'someotherclass')
    }
 
-Alternatively, you could use an inline Python expression in the
-following form::
-
-   inherit_defer ${@'classname' if condition else ''}
-
-Or::
-
-   inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)}
-
-In all cases, if the expression evaluates to an
-empty string, the statement does not trigger a syntax error because it
-becomes a no-op.
+The conditional assignment of ``someotherclass`` has no effect on the
+``inherit_defer`` statement, which has already been resolved by the time
+the anonymous Python function runs.
 
 See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes
 ``inherit`` calls to ``inherit_defer``.
Signature · 1 line
2.48.1
#2 lore raw

Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

Hi Dawid,

Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

The documentation for inherit_defer contained the claim

"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"

Quoted 9 lines

which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer

Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives

I'm pretty sure the whole inherit_defer thing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit 5c2e840eafeb ("ast/BBHandler: Add inherit_defer support") in BitBake.

So I'm very skeptical this is correct.

Quoted 56 lines

Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with inherit_defer.

---
  .../bitbake-user-manual-metadata.rst          | 61 ++++++++++---------
  1 file changed, 32 insertions(+), 29 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +18 -13
index 40cae6b05..38efe6b11 100644
@@ -803,7 +803,17 @@ An advantage with the inherit directive as compared to both the
  :ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>`
  directives is that you can inherit class files conditionally. You can
  accomplish this by using a variable expression after the ``inherit``
-statement.
+statement, as in::
+
+   inherit ${@'classname' if condition else ''}
+
+Or::
+
+   inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)}
+
+In both cases, if the expression evaluates to an
+empty string, the statement does not trigger a syntax error because it
+becomes a no-op.
 
  For inheriting classes conditionally, using the :ref:`inherit_defer
  <ref-bitbake-user-manual-metadata-inherit-defer>` directive is advised as
@@ -827,39 +837,32 @@ the variable after the line is parsed will take effect. With the :ref:`inherit
 
  Here is an example::
 
-   inherit_defer ${VARNAME}
-
-If ``VARNAME`` is
-going to be set, it needs to be set before the ``inherit_defer`` statement is
-parsed. One way to achieve a conditional inherit in this case is to use
-overrides::
-
     VARIABLE = ""
-   VARIABLE:someoverride = "myclass"
-
-Another method is by using :ref:`anonymous Python
-<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`.
-Here is an example::
+   inherit_defer ${VARIABLE}
+   VARIABLE:someoverride = "someclass"
+
+:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+defers the evaluation of ``${VARIABLE}`` until the end of
+parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARIABLE}``
+expands to ``someclass``, which is then inherited. Contrast this with a plain

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst the variable after the line is parsed will take effect. With the :ref:`inherit · line 857

Quoted 10 lines
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +9 -0
+:ref:`inherit <ref-bitbake-user-manual-metadata-inherit>`, which would evaluate
+``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment
+is parsed. ``${VARIABLE}`` would expand to an empty string and the statement
+would become a no-op.
+
+Note that assigning the variable from an
+:ref:`anonymous Python function <bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`
+does *not* work, because deferred inherits are resolved before anonymous
+Python functions run::
 

I vaguely remember reading on IRC we're having some issues with anonymous Python functions and parsing lately, so that may be related to that. I haven't followed anything in that area so maybe I'm wrong or it has already been fixed. So I'm not sure if you just happen to try this when things were broken in BitBake or if it's really intended to work that way.

Cheers,
Quentin

#3 lore raw

Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

Quoted 95 lines · Quentin Schulz

Hi Dawid,

Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

The documentation for inherit_defer contained the claim

"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"

Quoted 9 lines

which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer

Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives

I'm pretty sure the whole inherit_defer thing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit 5c2e840eafeb ("ast/BBHandler: Add inherit_defer support") in BitBake.

So I'm very skeptical this is correct.

Quoted 71 lines

Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with inherit_defer.

---
  .../bitbake-user-manual-metadata.rst          | 61 ++++++++++---------

  1 file changed, 32 insertions(+), 29 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0 -0
index 40cae6b05..38efe6b11 100644
@@ -803,7 +803,17 @@ An advantage with the inherit directive as 

compared to both the
  :ref:include <bitbake-user-manual/bitbake-user-manual-metadata: \\include\\ directive> and :ref:require <bitbake-user-manual/ bitbake-user-manual-metadata:\\require\\ directive> directives is that you can inherit class files conditionally. You can
  accomplish this by using a variable expression after the ``inherit``

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +7 -1
-statement.
+statement, as in::
+
+   inherit ${@'classname' if condition else ''}
+
+Or::
+
+   inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname', 

'', d)}

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +4 -0
+
+In both cases, if the expression evaluates to an
+empty string, the statement does not trigger a syntax error because it
+becomes a no-op.

  For inheriting classes conditionally, using the :ref:inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer> directive is advised as

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0 -0
@@ -827,39 +837,32 @@ the variable after the line is parsed will take 

effect. With the :ref:`inherit

  Here is an example::

-   inherit_defer ${VARNAME}

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0 -3
-
-If ``VARNAME`` is
-going to be set, it needs to be set before the ``inherit_defer`` 

statement is

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0 -3
-parsed. One way to achieve a conditional inherit in this case is to use
-overrides::
-

     VARIABLE = ""
-   VARIABLE:someoverride = "myclass"

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0 -3
-
-Another method is by using :ref:`anonymous Python
-<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python 

Functions>`.

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +6 -1
-Here is an example::
+   inherit_defer ${VARIABLE}
+   VARIABLE:someoverride = "someclass"
+
+:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>`
+defers the evaluation of ``${VARIABLE}`` until the end of
+parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, 

``${VARIABLE}``
+expands to ``someclass``, which is then inherited. Contrast this with a plain
+:ref:inherit <ref-bitbake-user-manual-metadata-inherit>, which would evaluate
+``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment
+is parsed. ``${VARIABLE}`` would expand to an empty string and the statement

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +2 -0
+would become a no-op.
+
Quoted 9 lines

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst the variable after the line is parsed will take effect. With the :ref:`inherit · line 856

Quoted 6 lines

+Note that assigning the variable from an +:ref:anonymous Python function <bitbake-user-manual/bitbake-user- manual-metadata:Anonymous Python Functions>

doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +2 -0
+does *not* work, because deferred inherits are resolved before anonymous
+Python functions run::

I vaguely remember reading on IRC we're having some issues with anonymous Python functions and parsing lately, so that may be related to that. I haven't followed anything in that area so maybe I'm wrong or it has already been fixed. So I'm not sure if you just happen to try this when things were broken in BitBake or if it's really intended to work that way.

Reading my mail backlog, I just stumbled upon https://lore.kernel.org/openembedded-core/CANPvuRk19e3PhcVXxLUJTjaPE9bMXO-04MLytqGNpA0=PEzeeA@mail.gmail.com/ where Jose links to https://git.openembedded.org/openembedded-core/commit/?id=6b553a5042b9d5828a9da675ede16e10f06dae90 which does indeed mention this is a known limitation:

"""
Anonymous python is executed after all inherits including deferred inherits are processed. """

So this part is fine :)

Cheers,
Quentin

#4 lore raw

Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

In reply to Quentin Schulz
Quoted 20 lines · Quentin Schulz

Hi Dawid,

Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

The documentation for inherit_defer contained the claim

"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"

Quoted 9 lines

which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer

Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives

Quoted 8 lines

I'm pretty sure the whole inherit_defer thing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit 5c2e840eafeb ("ast/BBHandler: Add inherit_defer support") in BitBake.

So I'm very skeptical this is correct.

Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.

For example, on current master I tried the following:
COND = ""
COND:append = " x"
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)}
This inherits foo.

However, if I move the :append below the inherit directive:
COND = ""
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.

Greets,
Dawid

#5 lore raw

Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

Quoted 43 lines · Dawid Bijak via lists.openembedded.org
Quoted 20 lines · Quentin Schulz

Hi Dawid,

Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

The documentation for inherit_defer contained the claim

"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"

Quoted 9 lines

which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer

Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives

Quoted 8 lines

I'm pretty sure the whole inherit_defer thing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit 5c2e840eafeb ("ast/BBHandler: Add inherit_defer support") in BitBake.

So I'm very skeptical this is correct.

Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.

For example, on current master I tried the following:
 COND = ""
 COND:append = " x"
 inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} This inherits foo.

Quoted 5 lines

However, if I move the :append below the inherit directive:
 COND = ""
 inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.

It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.

Cheers,

Richard

#6 lore raw

Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

Hi,

Quoted 20 lines · Richard Purdie
Quoted 13 lines · Dawid Bijak via lists.openembedded.org

Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.

For example, on current master I tried the following:
�COND = ""
�COND:append = " x"
�inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} This inherits foo.

Quoted 5 lines

However, if I move the :append below the inherit directive:
�COND = ""
�inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} �COND:append = " x"
it inherits bar.

Quoted 5 lines

It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.

Richard, thanks for chiming in. Your explanation reassures me that my understanding of inherit_defer was correct.

In summary, I believe the 3 issues with the current documentation that I identified in the cover letter are still valid, and my patch still applies.

Greets,
Dawid

#7 lore raw

Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation

Hi Dawid,

Quoted 29 lines · Dawid Bijak

Hi,

Quoted 20 lines · Richard Purdie
Quoted 13 lines · Dawid Bijak via lists.openembedded.org

Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.

For example, on current master I tried the following:
 COND = ""
 COND:append = " x"
 inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)}
This inherits foo.

Quoted 5 lines

However, if I move the :append below the inherit directive:
 COND = ""
 inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.

Quoted 5 lines

It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.

Quoted 6 lines

Richard, thanks for chiming in. Your explanation reassures me that my understanding of inherit_defer was correct.

In summary, I believe the 3 issues with the current documentation that I identified in the cover letter are still valid, and my patch still applies.

1) The sentence should only apply to the inherit statement, and not inherit_defer as that's the point of inherit_defer. So this is fine.

2) NACK. We're deliberately not documenting footguns. You can provide an example if you really want to in the inherit section, and add a big fat warning after it that it'll break in some scenario so you really shouldn't do that and instead use inherit_defer.

3) This is fine, thanks for the fix.

I would suggest to make three separate commits, with 2) last, that way we can pick 1) and 3) (which will become 1) and 2)) and argue on 2) (which will become 3)) without hindering merging of the other commits.

Cheers,
Quentin