doc: bitbake-user-manual-metadata: clarify inherit_defer documentation
cover doc: bitbake-user-manual-metadata: clarify inherit_defer documentation
Hi,
I've been reading through the inherit_defer documentation, and granted my limited knowledge of bitbake, I think the current documentation is incorrect or misleading in a few places:
1)
The sentence "If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed" seems plainly wrong and defeats the purpose of inherit_defer.
As I understand it, the whole point of deferred evaluation is that assignments made after the directive are taken into account.
2)
The examples using inline Python, e.g. inherit_defer ${@'classname' if condition else ''}, are not specific to inherit_defer - they would work equally well with plain inherit. They probably belong in the inherit section.
3)
The suggested technique of assigning the class name from an anonymous Python function does not appear to work:
python () {
if d.getVar('SOMETHING') == 'value':
d.setVar('VARIABLE', 'someclass')
else:
d.setVar('VARIABLE', '')
}
inherit_defer ${VARIABLE}
From my experiments, inherit_defer is resolved before anonymous Python functions run, so ${VARIABLE} always expands to empty
and the inherit is a no-op.
Is that the intended order and the documentation is wrong, or is this a bug in bitbake, or did I just get it wrong?Anyway, just in case, the attached patch rewords the documentation to address points 1 and 2, and adds a note about point 3. Happy to revise if I've misunderstood any of it.
Thanks,
Dawid
Dawid Bijak (1):
doc: bitbake-user-manual-metadata: fix inherit_defer documentation
.../bitbake-user-manual-metadata.rst | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-)
Signature · 1 line
2.48.1
PATCH 1/1 doc: bitbake-user-manual-metadata: fix inherit_defer documentation
The documentation for inherit_defer contained the claim
"If VARNAME is going to be set, it needs to be set before the inherit_defer statement is parsed"
which is incorrect and contradicts the purpose of inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and the inherit_defer statement by using an override assignment placed after inherit_defer
Additionally the inline python expression examples have been moved from the inherit_defer section up to the inherit section, since they apply to both directives
Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with inherit_defer.
--- .../bitbake-user-manual-metadata.rst | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-)
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +32 -29
@@ -803,7 +803,17 @@ An advantage with the inherit directive as compared to both the :ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>` directives is that you can inherit class files conditionally. You can accomplish this by using a variable expression after the ``inherit`` -statement. +statement, as in:: + + inherit ${@'classname' if condition else ''} + +Or:: + + inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)} + +In both cases, if the expression evaluates to an +empty string, the statement does not trigger a syntax error because it +becomes a no-op. For inheriting classes conditionally, using the :ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` directive is advised as
@@ -827,39 +837,32 @@ the variable after the line is parsed will take effect. With the :ref:`inherit Here is an example:: - inherit_defer ${VARNAME} - -If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit_defer`` statement is -parsed. One way to achieve a conditional inherit in this case is to use -overrides:: - VARIABLE = "" - VARIABLE:someoverride = "myclass" - -Another method is by using :ref:`anonymous Python -<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`. -Here is an example:: + inherit_defer ${VARIABLE} + VARIABLE:someoverride = "someclass" + +:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` +defers the evaluation of ``${VARIABLE}`` until the end of +parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARIABLE}`` +expands to ``someclass``, which is then inherited. Contrast this with a plain +:ref:`inherit <ref-bitbake-user-manual-metadata-inherit>`, which would evaluate +``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment +is parsed. ``${VARIABLE}`` would expand to an empty string and the statement +would become a no-op. + +Note that assigning the variable from an +:ref:`anonymous Python function <bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>` +does *not* work, because deferred inherits are resolved before anonymous +Python functions run:: python () { - if condition == value: - d.setVar('VARIABLE', 'myclass') - else: - d.setVar('VARIABLE', '') + if d.getVar('SOMETHING') == 'value': + d.setVar('VARIABLE', 'someotherclass') } -Alternatively, you could use an inline Python expression in the -following form:: - - inherit_defer ${@'classname' if condition else ''} - -Or:: - - inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)} - -In all cases, if the expression evaluates to an -empty string, the statement does not trigger a syntax error because it -becomes a no-op. +The conditional assignment of ``someotherclass`` has no effect on the +``inherit_defer`` statement, which has already been resolved by the time +the anonymous Python function runs. See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes ``inherit`` calls to ``inherit_defer``.
Signature · 1 line
2.48.1
Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
Hi Dawid,
Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
The documentation for
inherit_defercontained the claim"If VARNAME is going to be set, it needs to be set before the
inherit_deferstatement is parsed"
Quoted 9 lines
which is incorrect and contradicts the purpose of
inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and theinherit_deferstatement by using an override assignment placed afterinherit_deferAdditionally the inline python expression examples have been moved from the
inherit_defersection up to the inherit section, since they apply to both directives
I'm pretty sure the whole inherit_defer thing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit 5c2e840eafeb ("ast/BBHandler: Add inherit_defer support") in BitBake.
So I'm very skeptical this is correct.
Quoted 56 lines
Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with
inherit_defer.Signed-off-by Dawid Bijak <[email protected]>--- .../bitbake-user-manual-metadata.rst | 61 ++++++++++--------- 1 file changed, 32 insertions(+), 29 deletions(-)doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +18
-13@@ -803,7 +803,17 @@ An advantage with the inherit directive as compared to both the :ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`include\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manual-metadata:\`\`require\`\` directive>` directives is that you can inherit class files conditionally. You can accomplish this by using a variable expression after the ``inherit`` -statement. +statement, as in:: + + inherit ${@'classname' if condition else ''} + +Or:: + + inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname', '', d)} + +In both cases, if the expression evaluates to an +empty string, the statement does not trigger a syntax error because it +becomes a no-op. For inheriting classes conditionally, using the :ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` directive is advised as@@ -827,39 +837,32 @@ the variable after the line is parsed will take effect. With the :ref:`inherit Here is an example:: - inherit_defer ${VARNAME} - -If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit_defer`` statement is -parsed. One way to achieve a conditional inherit in this case is to use -overrides:: - VARIABLE = "" - VARIABLE:someoverride = "myclass" - -Another method is by using :ref:`anonymous Python -<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>`. -Here is an example:: + inherit_defer ${VARIABLE} + VARIABLE:someoverride = "someclass" + +:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` +defers the evaluation of ``${VARIABLE}`` until the end of +parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARIABLE}`` +expands to ``someclass``, which is then inherited. Contrast this with a plain
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst the variable after the line is parsed will take effect. With the :ref:`inherit · line 857
Quoted 10 lines
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +9
-0+:ref:`inherit <ref-bitbake-user-manual-metadata-inherit>`, which would evaluate +``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment +is parsed. ``${VARIABLE}`` would expand to an empty string and the statement +would become a no-op. + +Note that assigning the variable from an +:ref:`anonymous Python function <bitbake-user-manual/bitbake-user-manual-metadata:Anonymous Python Functions>` +does *not* work, because deferred inherits are resolved before anonymous +Python functions run::
I vaguely remember reading on IRC we're having some issues with anonymous Python functions and parsing lately, so that may be related to that. I haven't followed anything in that area so maybe I'm wrong or it has already been fixed. So I'm not sure if you just happen to try this when things were broken in BitBake or if it's really intended to work that way.
Cheers,
Quentin
Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
Quoted 95 lines · Quentin Schulz
Hi Dawid,
Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
The documentation for
inherit_defercontained the claim"If VARNAME is going to be set, it needs to be set before the
inherit_deferstatement is parsed"Quoted 9 lines
which is incorrect and contradicts the purpose of
inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and theinherit_deferstatement by using an override assignment placed afterinherit_deferAdditionally the inline python expression examples have been moved from the
inherit_defersection up to the inherit section, since they apply to both directivesI'm pretty sure the whole
inherit_deferthing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit5c2e840eafeb("ast/BBHandler: Addinherit_defersupport") in BitBake.So I'm very skeptical this is correct.
Quoted 71 lines
Also add an anti-example showing that the previously documented pattern of setting the inherited class name from an anonymous Python function does not work with
inherit_defer.Signed-off-by Dawid Bijak <[email protected]>---
.../bitbake-user-manual-metadata.rst | 61 ++++++++++---------1 file changed, 32 insertions(+), 29 deletions(-)doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0
-0@@ -803,7 +803,17 @@ An advantage with the inherit directive ascompared to both the
:ref:include <bitbake-user-manual/bitbake-user-manual-metadata: \\include\\directive>and :ref:require <bitbake-user-manual/ bitbake-user-manual-metadata:\\require\\directive>directives is that you can inherit class files conditionally. You can
accomplish this by using a variable expression after the ``inherit``doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +7
-1-statement. +statement, as in:: + + inherit ${@'classname' if condition else ''} + +Or:: + + inherit ${@bb.utils.contains('VARIABLE', 'something', 'classname','', d)}
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +4
-0+ +In both cases, if the expression evaluates to an +empty string, the statement does not trigger a syntax error because it +becomes a no-op.For inheriting classes conditionally, using the :ref:
inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>directive is advised asdoc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0
-0@@ -827,39 +837,32 @@ the variable after the line is parsed will takeeffect. With the :ref:`inherit
Here is an example::
-
inherit_defer${VARNAME}doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0
-3- -If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit_defer``statement is
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0
-3-parsed. One way to achieve a conditional inherit in this case is to use -overrides:: -VARIABLE = ""
- VARIABLE:someoverride = "myclass"doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +0
-3- -Another method is by using :ref:`anonymous Python -<bitbake-user-manual/bitbake-user-manual-metadata:Anonymous PythonFunctions>`.
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +6
-1-Here is an example:: + inherit_defer ${VARIABLE} + VARIABLE:someoverride = "someclass" + +:ref:`inherit_defer <ref-bitbake-user-manual-metadata-inherit-defer>` +defers the evaluation of ``${VARIABLE}`` until the end of +parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`,``${VARIABLE}``
+expands to ``someclass``, which is then inherited. Contrast this with a plain
+:ref:inherit <ref-bitbake-user-manual-metadata-inherit>, which would evaluate
+``${VARIABLE}`` immediately, before the ``VARIABLE:someoverride`` assignment
+is parsed. ``${VARIABLE}`` would expand to an empty string and the statementdoc/bitbake-user-manual/bitbake-user-manual-metadata.rst +2
-0+would become a no-op. +
Quoted 9 lines
doc/bitbake-user-manual/bitbake-user-manual-metadata.rst the variable after the line is parsed will take effect. With the :ref:`inherit · line 856
Quoted 6 lines
+Note that assigning the variable from an +:ref:
anonymous Python function <bitbake-user-manual/bitbake-user- manual-metadata:Anonymous Python Functions>doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +2
-0+does *not* work, because deferred inherits are resolved before anonymous +Python functions run::I vaguely remember reading on IRC we're having some issues with anonymous Python functions and parsing lately, so that may be related to that. I haven't followed anything in that area so maybe I'm wrong or it has already been fixed. So I'm not sure if you just happen to try this when things were broken in BitBake or if it's really intended to work that way.
Reading my mail backlog, I just stumbled upon https://lore.kernel.org/openembedded-core/CANPvuRk19e3PhcVXxLUJTjaPE9bMXO-04MLytqGNpA0=PEzeeA@mail.gmail.com/ where Jose links to https://git.openembedded.org/openembedded-core/commit/?id=6b553a5042b9d5828a9da675ede16e10f06dae90 which does indeed mention this is a known limitation:
"""
Anonymous python is executed after all inherits including deferred inherits are processed. """
So this part is fine :)
Cheers,
Quentin
Re: [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
In reply to Quentin SchulzQuoted 20 lines · Quentin Schulz
Hi Dawid,
Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
The documentation for
inherit_defercontained the claim"If VARNAME is going to be set, it needs to be set before the
inherit_deferstatement is parsed"Quoted 9 lines
which is incorrect and contradicts the purpose of
inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and theinherit_deferstatement by using an override assignment placed afterinherit_deferAdditionally the inline python expression examples have been moved from the
inherit_defersection up to the inherit section, since they apply to both directives
Quoted 8 lines
I'm pretty sure the whole
inherit_deferthing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit5c2e840eafeb("ast/BBHandler: Addinherit_defersupport") in BitBake.So I'm very skeptical this is correct.
Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.
For example, on current master I tried the following:
COND = ""
COND:append = " x"
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)}
This inherits foo.
However, if I move the :append below the inherit directive:
COND = ""
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.
Greets,
Dawid
Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
Quoted 43 lines · Dawid Bijak via lists.openembedded.org
Quoted 20 lines · Quentin Schulz
Hi Dawid,
Quoted 7 lines · Dawid Bijak via lists.yoctoproject.org
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
The documentation for
inherit_defercontained the claim"If VARNAME is going to be set, it needs to be set before the
inherit_deferstatement is parsed"Quoted 9 lines
which is incorrect and contradicts the purpose of
inherit_defer. The wrong claim is replaced with an example which demonstrates the contrast between a plain defer statement and theinherit_deferstatement by using an override assignment placed afterinherit_deferAdditionally the inline python expression examples have been moved from the
inherit_defersection up to the inherit section, since they apply to both directivesQuoted 8 lines
I'm pretty sure the whole
inherit_deferthing started because an inline Python expression didn't work. I vaguely remember a bbappend either adding this new inherit based on PACKAGECONFIG containing a value, or having the main recipe with this inherit <PACKAGECONFIG.contains> and a bbappend modify PACKAGECONFIG. See commit5c2e840eafeb("ast/BBHandler: Addinherit_defersupport") in BitBake.So I'm very skeptical this is correct.
Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.For example, on current master I tried the following:
COND = ""
COND:append = " x"
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} This inherits foo.
Quoted 5 lines
However, if I move the :append below the inherit directive:
COND = ""
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.
It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.
Cheers,
Richard
Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
Hi,
Quoted 20 lines · Richard Purdie
Quoted 13 lines · Dawid Bijak via lists.openembedded.org
Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.For example, on current master I tried the following:
�COND = ""
�COND:append = " x"
�inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} This inherits foo.Quoted 5 lines
However, if I move the :append below the inherit directive:
�COND = ""
�inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} �COND:append = " x"
it inherits bar.
Quoted 5 lines
It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.
Richard, thanks for chiming in. Your explanation reassures me that my understanding of inherit_defer was correct.
In summary, I believe the 3 issues with the current documentation that I identified in the cover letter are still valid, and my patch still applies.
Greets,
Dawid
Re: [bitbake-devel] [docs] [PATCH 1/1] doc: bitbake-user-manual-metadata: fix inherit_defer documentation
Hi Dawid,
Quoted 29 lines · Dawid Bijak
Hi,
Quoted 20 lines · Richard Purdie
Quoted 13 lines · Dawid Bijak via lists.openembedded.org
Hi Quentin,
Thanks for your response.
I'm afraid I don't quite follow your point about PACKAGECONFIG.contains.
I'm not sure what you're referring to exactly.
But, as far as I can tell, the inherit statement works fine with inline python expressions. I think, it evaluates the expression in place, much like := would.For example, on current master I tried the following:
COND = ""
COND:append = " x"
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)}
This inherits foo.Quoted 5 lines
However, if I move the :append below the inherit directive:
COND = ""
inherit ${@bb.utils.contains('COND', 'x', 'foo', 'bar', d)} COND:append = " x"
it inherits bar.Quoted 5 lines
It works, as long as you are sure that COND won't be changed after the inherit. When mutliple files altering the variables are involved, that is often unclear. I think the doc's intent is therefore to recommend anything with variable accesses is therefore deferred, unless the user is sure they know what they're doing.
Quoted 6 lines
Richard, thanks for chiming in. Your explanation reassures me that my understanding of
inherit_deferwas correct.In summary, I believe the 3 issues with the current documentation that I identified in the cover letter are still valid, and my patch still applies.
1) The sentence should only apply to the inherit statement, and not inherit_defer as that's the point of inherit_defer. So this is fine.
2) NACK. We're deliberately not documenting footguns. You can provide an example if you really want to in the inherit section, and add a big fat warning after it that it'll break in some scenario so you really shouldn't do that and instead use inherit_defer.
3) This is fine, thanks for the fix.
I would suggest to make three separate commits, with 2) last, that way we can pick 1) and 3) (which will become 1) and 2)) and argue on 2) (which will become 3)) without hindering merging of the other commits.
Cheers,
Quentin