bitbake-setup: clarify configuration merging and add a notes property
cover bitbake-setup: clarify configuration merging and add a notes property
This series aims to clarify the configurating merging, since it is not necessarily obvious.
The second patch uses this and adds a "notes" string property that can be used to populate "conf-notes.txt" when bb-layers is used instead of oe-template. For this and the "description" property, it's good to be aware that strings get concatenated when they occur in nested configurations.
---
Ernest Van Hoecke (2):
bitbake-setup: clarify configuration merging
bitbake-setup: write notes for generated build configs
bin/bitbake-setup | 3 ++- .../bitbake-user-manual-environment-setup.rst | 11 +++++++++++ lib/bb/tests/setup.py | 4 ++++ setup-schema/bitbake-setup.schema.json | 4 ++++ 4 files changed, 21 insertions(+), 1 deletion(-)
---
base-commit: e9a06f79d9ec767c9d95470be78b006d6fd0d59c
change-id: 20260529-bb-setup-notes-0663341b0478
Best regards,
Signature · 1 line
Ernest Van Hoecke <[email protected]>
PATCH 1/2 bitbake-setup: clarify configuration merging
From: Ernest Van Hoecke <[email protected]>
When nested configurations are used and the same key is present in both configs being merged, the Python "+" behaviour is used. This leads to lists being appended and strings being concatenated. The first is an intuitive behaviour when specifying a tree of configs, the latter might not be.
Clarify the configuration merging behaviour explicitly in the docs.
--- doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst | 6 ++++++ 1 file changed, 6 insertions(+)
doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst +6 -0
@@ -1068,6 +1068,12 @@ They contain the following sections: configuration choices by putting together information from a leaf configuration and all of its ancestors. + When the same keyword is present in a nested configuration and in one of + its ancestors, the values are merged with Python ``+`` semantics. For + example, lists are appended and strings are concatenated directly. String + values such as ``description`` should include any needed separators in + the configuration data. + - ``bb-env-passthrough-additions`` (*optional*): List of environment variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`.
Signature · 1 line
2.43.0
Re: [bitbake-devel] [PATCH 1/2] bitbake-setup: clarify configuration merging
Thanks, this lgtm!
Alex
Quoted 46 lines · Ernest Van Hoecke via lists.openembedded.org
From: Ernest Van Hoecke <[email protected]>
When nested configurations are used and the same key is present in both configs being merged, the Python "+" behaviour is used. This leads to lists being appended and strings being concatenated. The first is an intuitive behaviour when specifying a tree of configs, the latter might not be.
Clarify the configuration merging behaviour explicitly in the docs.
Signed-off-by Ernest Van Hoecke <[email protected]>--- doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst | 6 ++++++ 1 file changed, 6 insertions(+)doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst +6
-1@@ -1068,6 +1068,12 @@ They contain the following sections: configuration choices by putting together information from a leaf configuration and all of its ancestors. + When the same keyword is present in a nested configuration and in one of + its ancestors, the values are merged with Python ``+`` semantics. For + example, lists are appended and strings are concatenated directly. String + values such as ``description`` should include any needed separators in + the configuration data. + - ``bb-env-passthrough-additions`` (*optional*): List of environment variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. --2.43.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19564): https://lists.openembedded.org/g/bitbake-devel/message/19564
Mute This Topic: https://lists.openembedded.org/mt/119548245/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH 2/2 bitbake-setup: write notes for generated build configs
In reply to Ernest Van HoeckeFrom: Ernest Van Hoecke <[email protected]>
For bb-layers configurations, bitbake-setup currently writes an empty "conf-notes.txt". Configurations using "oe-template" get this file from the template.
Add an optional field called "notes" to configurations that populates this field when "bb-layers" is used instead of "oe-template".
--- bin/bitbake-setup | 3 ++- doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst | 5 +++++ lib/bb/tests/setup.py | 4 ++++ setup-schema/bitbake-setup.schema.json | 4 ++++ 4 files changed, 15 insertions(+), 1 deletion(-)
bin/bitbake-setup +2 -1
@@ -301,7 +301,8 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c f.write(bitbake_config["description"] + "\n") with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: - f.write("") + notes = bitbake_config.get("notes") + f.write(notes + "\n" if notes else "") def _make_init_build_env(builddir, oeinitbuildenvdir): builddir = os.path.realpath(builddir)
doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst +5 -0
@@ -1057,6 +1057,11 @@ They contain the following sections: snippet. This is what is prompted during the :ref:`ref-bbsetup-command-init` command execution. + - ``notes`` (*optional*): additional information written to + ``build/conf/conf-notes.txt`` when ``bitbake-setup`` generates the build + configuration from ``bb-layers``. For ``oe-template`` configurations, + this file is provided by the template. + - ``configurations``: Configurations can recursively contain as many nested configurations as needed. This will create more choices when running the :ref:`ref-bbsetup-command-init` command.
lib/bb/tests/setup.py +4 -0
@@ -136,6 +136,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) { "name": "gadget-notemplate", "description": "Gadget notemplate configuration", + "notes": "Gadget notemplate notes", "bb-layers": ["layerA","layerB/meta-layer"], "oe-fragments": ["test-fragment-1"] },
@@ -249,6 +250,9 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) else: with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: self.assertIn(bitbake_config["description"], f.read()) + with open(os.path.join(bb_conf_path, 'conf-notes.txt')) as f: + expected_notes = bitbake_config.get("notes") + self.assertEqual(f.read(), expected_notes + "\n" if expected_notes else "") with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f: bblayers = f.read() for l in bitbake_config["bb-layers"]:
setup-schema/bitbake-setup.schema.json +4 -0
@@ -41,6 +41,10 @@ "type": "string", "description": "Human-readable description of the configuration" }, + "notes": { + "type": "string", + "description": "Extra notes that populate conf-notes.txt when bb-layers is used" + }, "bb-layers": { "type": "array", "description": "List of BitBake layer paths to include, relative to the layers download directory",
Signature · 1 line
2.43.0
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
Thanks, this lgtm too.
Alex
Quoted 98 lines · Ernest Van Hoecke via lists.openembedded.org
From: Ernest Van Hoecke <[email protected]>
For bb-layers configurations, bitbake-setup currently writes an empty "conf-notes.txt". Configurations using "oe-template" get this file from the template.
Add an optional field called "notes" to configurations that populates this field when "bb-layers" is used instead of "oe-template".
Signed-off-by Ernest Van Hoecke <[email protected]>--- bin/bitbake-setup | 3 ++- doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst | 5 +++++ lib/bb/tests/setup.py | 4 ++++ setup-schema/bitbake-setup.schema.json | 4 ++++ 4 files changed, 15 insertions(+), 1 deletion(-)bin/bitbake-setup +2
-1@@ -301,7 +301,8 @@ def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, update_bb_c f.write(bitbake_config["description"] + "\n") with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as f: - f.write("") + notes = bitbake_config.get("notes") + f.write(notes + "\n" if notes else "") def _make_init_build_env(builddir, oeinitbuildenvdir): builddir = os.path.realpath(builddir)doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst +5
-0@@ -1057,6 +1057,11 @@ They contain the following sections: snippet. This is what is prompted during the :ref:`ref-bbsetup-command-init` command execution. + - ``notes`` (*optional*): additional information written to + ``build/conf/conf-notes.txt`` when ``bitbake-setup`` generates the build + configuration from ``bb-layers``. For ``oe-template`` configurations, + this file is provided by the template. + - ``configurations``: Configurations can recursively contain as many nested configurations as needed. This will create more choices when running the :ref:`ref-bbsetup-command-init` command.lib/bb/tests/setup.py +4
-0@@ -136,6 +136,7 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) { "name": "gadget-notemplate", "description": "Gadget notemplate configuration", + "notes": "Gadget notemplate notes", "bb-layers": ["layerA","layerB/meta-layer"], "oe-fragments": ["test-fragment-1"] },@@ -249,6 +250,9 @@ print("BBPATH is {{}}".format(os.environ["BBPATH"])) else: with open(os.path.join(bb_conf_path, 'conf-summary.txt')) as f: self.assertIn(bitbake_config["description"], f.read()) + with open(os.path.join(bb_conf_path, 'conf-notes.txt')) as f: + expected_notes = bitbake_config.get("notes") + self.assertEqual(f.read(), expected_notes + "\n" if expected_notes else "") with open(os.path.join(bb_conf_path, 'bblayers.conf')) as f: bblayers = f.read() for l in bitbake_config["bb-layers"]:setup-schema/bitbake-setup.schema.json +4
-1@@ -41,6 +41,10 @@ "type": "string", "description": "Human-readable description of the configuration" }, + "notes": { + "type": "string", + "description": "Extra notes that populate conf-notes.txt when bb-layers is used" + }, "bb-layers": { "type": "array", "description": "List of BitBake layer paths to include, relative to the layers download directory", --2.43.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19565): https://lists.openembedded.org/g/bitbake-devel/message/19565
Mute This Topic: https://lists.openembedded.org/mt/119548246/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
In reply to Ernest Van HoeckeQuoted 8 lines · Ernest Van Hoecke via lists.openembedded.org
From: Ernest Van Hoecke <[email protected]>
For bb-layers configurations, bitbake-setup currently writes an empty "conf-notes.txt". Configurations using "oe-template" get this file from the template.
Add an optional field called "notes" to configurations that populates this field when "bb-layers" is used instead of "oe-template".
Suggestion: call the field "conf-notes" so it matches the resulting filename?
Also, I supposed newlines must be explicitly written in the string ("\n")? Are those properly converted, or does it need escaping of some sorts?
Thanks,
Antonin
Re: [docs] [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
Quoted 1 line · Antonin Godard via lists.yoctoproject.org
Suggestion: call the field "conf-notes" so it matches the resulting filename?
I disagree. It's meant for general-purpose freeform notes about the configuration; that they also get written into conf-notes.txt is an implementation detail. It's also more consistent, as there's already "description" which gets written into conf-summary.txt.
Alex
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
In reply to Antonin GodardQuoted 10 lines · Antonin Godard
Quoted 8 lines · Ernest Van Hoecke via lists.openembedded.org
From: Ernest Van Hoecke <[email protected]>
For bb-layers configurations, bitbake-setup currently writes an empty "conf-notes.txt". Configurations using "oe-template" get this file from the template.
Add an optional field called "notes" to configurations that populates this field when "bb-layers" is used instead of "oe-template".
Quoted 8 lines
Suggestion: call the field "conf-notes" so it matches the resulting filename?
Also, I supposed newlines must be explicitly written in the string ("\n")? Are those properly converted, or does it need escaping of some sorts?
Thanks,
Antonin
Hi Antonin,
Yes, I tested with some newlines explicitly written and this worked fine. Since it just comes from a json string and is passed to f.write() I don't believe any special escaping is necessary, just "\n" is fine.
My understanding is that the Python json library should handle this well and converts it to a real newline in the python string.
Thanks,
Ernest
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
Quoted 6 lines · Ernest Van Hoecke via lists.openembedded.org
Yes, I tested with some newlines explicitly written and this worked fine. Since it just comes from a json string and is passed to
f.write()I don't believe any special escaping is necessary, just "\n" is fine.My understanding is that the Python json library should handle this well and converts it to a real newline in the python string.
I need to ask though: how practical and readable is it going to be, when written into the json? Notes are meant as multiple paragraphs of text, and it would be good to be able to write them that way, instead of some specially-formatted single string.
Alex
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
Quoted 11 lines · Alexander Kanavin
Quoted 6 lines · Ernest Van Hoecke via lists.openembedded.org
Yes, I tested with some newlines explicitly written and this worked fine. Since it just comes from a json string and is passed to
f.write()I don't believe any special escaping is necessary, just "\n" is fine.My understanding is that the Python json library should handle this well and converts it to a real newline in the python string.
Quoted 7 lines
I need to ask though: how practical and readable is it going to be, when written into the json? Notes are meant as multiple paragraphs of text, and it would be good to be able to write them that way, instead of some specially-formatted single string.
Alex
I considered this and agree, there's no true beautiful solution here. The description field also has this issue, granted, that's meant to be much shorter than notes.
Since json doesn't support multiline strings, the only alternative is making this field a list of strings and concatenating them in code. I'm certainly open to that and can send a v2 doing it that way (it'd look like an extra "\n".join()).
Ernest
Re: [bitbake-devel] [PATCH 2/2] bitbake-setup: write notes for generated build configs
Quoted 8 lines · Ernest Van Hoecke
I considered this and agree, there's no true beautiful solution here. The description field also has this issue, granted, that's meant to be much shorter than notes.
Since json doesn't support multiline strings, the only alternative is making this field a list of strings and concatenating them in code. I'm certainly open to that and can send a v2 doing it that way (it'd look like an extra "\n".join()).
Yes. We're basically restricted by json's original purpose of being a data transfer format, and not a config file format. (the "no comments" restriction is in the same vein). It's a lot more human friendly than xml, but still not friendly enough, by modern standards.
It's tempting to make it a reference to an external file, but this wouldn't work well when the config is fetched with http:
$ bitbake-setup init https://my.server/some-build.conf.json would have to also fetch the notes file.
A list of strings isn't horrible, perhaps either can be supported?
Alex