support bitbake blocks
cover support bitbake blocks
From: Trevor Woerner <[email protected]>
Out of the box, Pygments does not handle bitbake blocks for syntax highlighting. As a result the numerous bitbake examples and snippets throughout our documentation use a basic, non-bitbake-specific highlighting.
This patchset adds a custom lexer to support bitbake syntax, based on the vim syntax highlighting information in [1], adds it to the sphinx extensions, and then goes through the BSP Developer's Guide identifying and re-formatting any bitbake blocks as "..code-block: bitbake".
If this RFC is acceptable, I can update it to convert all such blocks throughout the entire documentation set.
[1] https://git.openembedded.org/bitbake/tree/contrib/vim
Trevor Woerner (2):
add a "bitbake" pygments lexer
BSP dev guide: update to use bitbake lexer
documentation/bsp-guide/bsp.rst | 40 +++++-- documentation/conf.py | 3 +- documentation/sphinx/bitbake.py | 195 ++++++++++++++++++++++++++++++++ 3 files changed, 227 insertions(+), 11 deletions(-) create mode 100644 documentation/sphinx/bitbake.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
RFC PATCH 1/2 add a "bitbake" pygments lexer
From: Trevor Woerner <[email protected]>
Highlighting renders correctly across the BitBake constructs from the upstream vim syntax:
comments
all assignment operators (=, ?=, +=, =+, :=, ??=, .=, =.) ${VAR} and ${@...} interpolation
OE override chains (FILES:${PN}-doc, :append:class-target) varflags (do_install[depends])
inherit/include/require
addtask ... after ... before ...
EXPORT_FUNCTIONS
shell function bodies (delegated to BashLexer) python/fakeroot python task bodies and top-level def blocks (delegated
to PythonLexer).
Changes
documentation/sphinx/bitbake.py is a new local Sphinx extension that
defines BitbakeLexer (a Pygments RegexLexer) and registers it with Sphinx via sphinx.highlighting.lexers under the aliases bitbake and bb. Token rules mirror contrib/vim/syntax/bitbake.vim from the bitbake repo, and inner shell/python regions are dispatched to the existing BashLexer / PythonLexer using Pygments' using() helper.
documentation/conf.py appends 'bitbake' to extensions so the lexer is
loaded on every build.
Usage
In any .rst file you can now write:
.. code-block:: bitbake
SUMMARY = "Hello"
inherit autotools
do_install:append() {
install -d ${D}${bindir}
}bb works as an alias if shorter is preferred, and the same applies to .. highlight:: bitbake.
AI-Generated: codex/claude-opus 4.7 (xhigh)
--- documentation/conf.py | 3 +- documentation/sphinx/bitbake.py | 195 ++++++++++++++++++++++++++++++++ 2 files changed, 197 insertions(+), 1 deletion(-) create mode 100644 documentation/sphinx/bitbake.py
documentation/conf.py +2 -1
@@ -68,7 +68,8 @@ extensions = [ 'sphinx.ext.intersphinx', 'sphinx_copybutton', 'sphinxcontrib.rsvgconverter', - 'yocto-vars' + 'yocto-vars', + 'bitbake', ] autosectionlabel_prefix_document = True
documentation/sphinx/bitbake.py new file +195 -0
195 lines not on this page · raw message
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
RFC PATCH 2/2 BSP dev guide: update to use bitbake lexer
In reply to Trevor WoernerFrom: Trevor Woerner <[email protected]>
Identify all bitbake blocks, re-categorize them as "code-blocks" and set their language as "bitbake".
10 highlighted BitBake blocks emitted in the rendered HTML, matching the 10 code-block:: bitbake directives now in bsp.rst. Build completed with the same 189 pre-existing warnings (all intersphinx / cross- ref related, none from these changes).
Edits
In documentation/bsp-guide/bsp.rst, I converted every snippet that is actually BitBake metadata from a plain literal block (::) to .. code-block:: bitbake. Shell transcripts, directory listings, and prose- following :: blocks were left alone. The 10 converted blocks are:
1. BBLAYERS ?= ... example near the BSP-layer overview.
2. The expanded BBLAYERS example covering container layers like meta-openembedded/meta-oe.
3. The boilerplate conf/layer.conf (BBPATH, BBFILES, BBFILE_COLLECTIONS, LAYERDEPENDS_bsp).
4. The Raspberry Pi conf/layer.conf excerpt with LICENSE_PATH.
5. The include conf/machine/include/rpi-base.inc directive in raspberrypi3.conf.
6. The PREFERRED_PROVIDER_virtual/kernel / PREFERRED_VERSION_linux-yocto snippet in the kernel-recipe section.
7. The FILESEXTRAPATHS:prepend := "${THISDIR}/files:" example in the customizing-a-recipe walkthrough.
8. The Beaglebone conf/layer.conf shown in "BSP Layer Configuration Example".
9. The Beaglebone PREFERRED_PROVIDER_virtual/kernel / PREFERRED_VERSION_linux-yocto snippet in "BSP Kernel Recipe Example".
10. The linux-yocto_6.1.bbappend body (KBRANCH:*, KMACHINE:*, SRCREV_machine:*, COMPATIBLE_MACHINE:*, LINUX_VERSION:*).
The directory-listing block at line 290 stays as code-block:: none (it is a filesystem tree, not BitBake), and the formfactor machconfig snippet (HAVE_TOUCHSCREEN=0 etc.) is intentionally left as a plain literal block since it is shell-style key=value, not BitBake syntax.
Verification
make (python3 -m sphinx -b html ... . _build/html) completes with exit 0 and build succeeded, 189 warnings, identical to the pre-change baseline; none of those warnings reference bsp.rst line numbers I touched, the new bitbake extension, or Pygments lexing. Spot-checking the rendered HTML shows 10 highlight-bitbake Pygments-highlighted blocks in documentation/_build/html/bsp-guide/bsp.html, matching the 10 directives.
AI-Generated: codex/claude-opus 4.7 (xhigh)
--- documentation/bsp-guide/bsp.rst | 40 ++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-)
documentation/bsp-guide/bsp.rst +30 -10
@@ -84,7 +84,9 @@ established after you run the OpenEmbedded build environment setup script (i.e. :ref:`structure-core-script`). Adding the root directory allows the :term:`OpenEmbedded Build System` to recognize the BSP -layer and from it build an image. Here is an example:: +layer and from it build an image. Here is an example: + +.. code-block:: bitbake BBLAYERS ?= " \ /usr/local/src/yocto/meta \
@@ -112,7 +114,9 @@ are known as ":term:`container layers <Container Layer>`". An example of this type of layer is OpenEmbedded's :oe_git:`meta-openembedded </meta-openembedded>` layer. The ``meta-openembedded`` layer contains many ``meta-*`` layers. In cases like this, you need to include the names of the actual layers -you want to work with, such as:: +you want to work with, such as: + +.. code-block:: bitbake BBLAYERS ?= " \ /usr/local/src/yocto/meta \
@@ -531,7 +535,9 @@ identifies the contents of the layer, and contains information about how the build system should use it. Generally, a standard boilerplate file such as the following works. In the following example, you would replace "bsp" with the actual name of the BSP (i.e. "bsp_root_name" from the example -template). :: +template). + +.. code-block:: bitbake # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}"
@@ -546,7 +552,9 @@ template). :: LAYERDEPENDS_bsp = "intel" To illustrate the string substitutions, here are the corresponding -statements from the Raspberry Pi ``conf/layer.conf`` file:: +statements from the Raspberry Pi ``conf/layer.conf`` file: + +.. code-block:: bitbake # We have a conf and classes directory, append to BBPATH BBPATH .= ":${LAYERDIR}"
@@ -603,7 +611,9 @@ For example, many ``tune-*`` files (e.g. ``tune-arm1136jf-s.inc``, To use an include file, you simply include them in the machine configuration file. For example, the Raspberry Pi BSP -``raspberrypi3.conf`` contains the following statement:: +``raspberrypi3.conf`` contains the following statement: + +.. code-block:: bitbake include conf/machine/include/rpi-base.inc
@@ -674,7 +684,9 @@ Suppose you are using the ``linux-yocto_6.12.bb`` recipe to build the kernel. In other words, you have selected the kernel in your ``"bsp_root_name".conf`` file by adding :term:`PREFERRED_PROVIDER` and :term:`PREFERRED_VERSION` -statements as follows:: +statements as follows: + +.. code-block:: bitbake PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto" PREFERRED_VERSION_linux-yocto ?= "6.12%"
@@ -1027,7 +1039,9 @@ BSP-specific configuration file named ``interfaces`` to the also supports several other machines: #. Edit the ``init-ifupdown_1.0.bbappend`` file so that it contains the - following:: + following: + + .. code-block:: bitbake FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
@@ -1195,7 +1209,9 @@ BSP Layer Configuration Example ------------------------------- The layer's ``conf`` directory contains the ``layer.conf`` configuration -file. In this example, the ``conf/layer.conf`` file is the following:: +file. In this example, the ``conf/layer.conf`` file is the following: + +.. code-block:: bitbake # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}"
@@ -1375,7 +1391,9 @@ BSP Kernel Recipe Example ------------------------- The kernel recipe used to build the kernel image for the BeagleBone -device was established in the machine configuration:: +device was established in the machine configuration: + +.. code-block:: bitbake PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto" PREFERRED_VERSION_linux-yocto ?= "6.1%"
@@ -1386,7 +1404,9 @@ metadata used to build the kernel. In this case, a kernel append file kernel recipe (i.e. ``linux-yocto_6.1.bb``), which is located in :oe_git:`/openembedded-core/tree/meta/recipes-kernel/linux`. -The contents of the append file are:: +The contents of the append file are: + +.. code-block:: bitbake KBRANCH:genericx86 = "v6.1/standard/base" KBRANCH:genericx86-64 = "v6.1/standard/base"
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
Hello docs people!
In today's YP tech call I mentioned that I had been using reST/Sphinx extensively recently and had a couple updates I thought might be useful to the project. I have submitted this RFC patchset to gauge interest in this proposal. If you like it I can submit a full patchset with updates for all document.
Thanks and best regards,
Trevor
Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer
In reply to Trevor WoernerHi,
On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]
Review
Quoted 2 lines
(quoted diff) +2
-0+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides +# one.
Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.
Thanks,
Antonin
Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer
Quoted 9 lines · Antonin Godard
Hi,
On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]Review
Quoted 2 lines
(quoted diff) +2
-0+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides +# one.Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.
All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releases
I will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.
https://pygments.org/docs/contributing/
Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.
It was interesting to me to discover that our own documentation does not know how to handle bitbake syntax.
Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer
Quoted 20 lines · Trevor Woerner
Quoted 9 lines · Antonin Godard
Hi,
On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]Review
Quoted 2 lines
(quoted diff) +2
-0+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides +# one.Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.
All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releasesI will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.
Quoted 7 lines
https://pygments.org/docs/contributing/
Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.
We'll see what they think: https://github.com/pygments/pygments/pull/3103
Quoted 3 lines
It was interesting to me to discover that our own documentation does not know how to handle bitbake syntax.
Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer
Hi,
Quoted 23 lines · Trevor Woerner
Quoted 22 lines · Trevor Woerner
Quoted 9 lines · Antonin Godard
Hi,
On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]Review
Quoted 2 lines
(quoted diff) +2
-0+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides +# one.Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.
All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releasesI will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.
Quoted 7 lines
Quoted 5 lines
Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.
We'll see what they think: https://github.com/pygments/pygments/pull/3103
BitBake does not change syntax often, I would say, so once it's settled in, I think it won't have to change that much.
Thanks for the PR! Let's see where this goes.
Antonin