Skip to contentSkip to conversation

support bitbake blocks

RFC PATCH In review Trevor Woerner 2 participants 8 messages Apr 21 – Apr 23

lore
Share & tools
lore
#0 lore raw

cover support bitbake blocks

From: Trevor Woerner <[email protected]>

Out of the box, Pygments does not handle bitbake blocks for syntax highlighting. As a result the numerous bitbake examples and snippets throughout our documentation use a basic, non-bitbake-specific highlighting.

This patchset adds a custom lexer to support bitbake syntax, based on the vim syntax highlighting information in [1], adds it to the sphinx extensions, and then goes through the BSP Developer's Guide identifying and re-formatting any bitbake blocks as "..code-block: bitbake".

If this RFC is acceptable, I can update it to convert all such blocks throughout the entire documentation set.

[1] https://git.openembedded.org/bitbake/tree/contrib/vim

Trevor Woerner (2):
add a "bitbake" pygments lexer
BSP dev guide: update to use bitbake lexer

 documentation/bsp-guide/bsp.rst |  40 +++++--
 documentation/conf.py           |   3 +-
 documentation/sphinx/bitbake.py | 195 ++++++++++++++++++++++++++++++++
 3 files changed, 227 insertions(+), 11 deletions(-)
 create mode 100644 documentation/sphinx/bitbake.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#1 lore raw

RFC PATCH 1/2 add a "bitbake" pygments lexer

From: Trevor Woerner <[email protected]>

Highlighting renders correctly across the BitBake constructs from the upstream vim syntax:
comments
all assignment operators (=, ?=, +=, =+, :=, ??=, .=, =.) ${VAR} and ${@...} interpolation
OE override chains (FILES:${PN}-doc, :append:class-target) varflags (do_install[depends])
inherit/include/require
addtask ... after ... before ...
EXPORT_FUNCTIONS
shell function bodies (delegated to BashLexer) python/fakeroot python task bodies and top-level def blocks (delegated
to PythonLexer).

Changes

documentation/sphinx/bitbake.py is a new local Sphinx extension that
defines BitbakeLexer (a Pygments RegexLexer) and registers it with Sphinx via sphinx.highlighting.lexers under the aliases bitbake and bb. Token rules mirror contrib/vim/syntax/bitbake.vim from the bitbake repo, and inner shell/python regions are dispatched to the existing BashLexer / PythonLexer using Pygments' using() helper.

documentation/conf.py appends 'bitbake' to extensions so the lexer is
loaded on every build.

Usage

In any .rst file you can now write:

.. code-block:: bitbake

     SUMMARY = "Hello"
     inherit autotools
     do_install:append() {
         install -d ${D}${bindir}
     }

bb works as an alias if shorter is preferred, and the same applies to .. highlight:: bitbake.

AI-Generated: codex/claude-opus 4.7 (xhigh)

---
 documentation/conf.py           |   3 +-
 documentation/sphinx/bitbake.py | 195 ++++++++++++++++++++++++++++++++
 2 files changed, 197 insertions(+), 1 deletion(-)
 create mode 100644 documentation/sphinx/bitbake.py
documentation/conf.py +2 -1
index e5e3a8a89abd..aa2ec8d45ed3 100644
@@ -68,7 +68,8 @@ extensions = [
     'sphinx.ext.intersphinx',
     'sphinx_copybutton',
     'sphinxcontrib.rsvgconverter',
-    'yocto-vars'
+    'yocto-vars',
+    'bitbake',
 ]
 autosectionlabel_prefix_document = True
 
documentation/sphinx/bitbake.py new file +195 -0

195 lines not on this page · raw message

Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#6 lore raw

RFC PATCH 2/2 BSP dev guide: update to use bitbake lexer

In reply to Trevor Woerner

From: Trevor Woerner <[email protected]>

Identify all bitbake blocks, re-categorize them as "code-blocks" and set their language as "bitbake".

10 highlighted BitBake blocks emitted in the rendered HTML, matching the 10 code-block:: bitbake directives now in bsp.rst. Build completed with the same 189 pre-existing warnings (all intersphinx / cross- ref related, none from these changes).

Edits

In documentation/bsp-guide/bsp.rst, I converted every snippet that is actually BitBake metadata from a plain literal block (::) to .. code-block:: bitbake. Shell transcripts, directory listings, and prose- following :: blocks were left alone. The 10 converted blocks are:

1. BBLAYERS ?= ... example near the BSP-layer overview.
2. The expanded BBLAYERS example covering container layers like meta-openembedded/meta-oe.
3. The boilerplate conf/layer.conf (BBPATH, BBFILES, BBFILE_COLLECTIONS, LAYERDEPENDS_bsp).
4. The Raspberry Pi conf/layer.conf excerpt with LICENSE_PATH.
5. The include conf/machine/include/rpi-base.inc directive in raspberrypi3.conf.
6. The PREFERRED_PROVIDER_virtual/kernel / PREFERRED_VERSION_linux-yocto snippet in the kernel-recipe section.
7. The FILESEXTRAPATHS:prepend := "${THISDIR}/files:" example in the customizing-a-recipe walkthrough.
8. The Beaglebone conf/layer.conf shown in "BSP Layer Configuration Example".
9. The Beaglebone PREFERRED_PROVIDER_virtual/kernel / PREFERRED_VERSION_linux-yocto snippet in "BSP Kernel Recipe Example".
10. The linux-yocto_6.1.bbappend body (KBRANCH:*, KMACHINE:*, SRCREV_machine:*, COMPATIBLE_MACHINE:*, LINUX_VERSION:*).

The directory-listing block at line 290 stays as code-block:: none (it is a filesystem tree, not BitBake), and the formfactor machconfig snippet (HAVE_TOUCHSCREEN=0 etc.) is intentionally left as a plain literal block since it is shell-style key=value, not BitBake syntax.

Verification

make (python3 -m sphinx -b html ... . _build/html) completes with exit 0 and build succeeded, 189 warnings, identical to the pre-change baseline; none of those warnings reference bsp.rst line numbers I touched, the new bitbake extension, or Pygments lexing. Spot-checking the rendered HTML shows 10 highlight-bitbake Pygments-highlighted blocks in documentation/_build/html/bsp-guide/bsp.html, matching the 10 directives.

AI-Generated: codex/claude-opus 4.7 (xhigh)

---
 documentation/bsp-guide/bsp.rst | 40 ++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 10 deletions(-)
documentation/bsp-guide/bsp.rst +30 -10
index a3b57d29f9e6..67dd9a67d651 100644
@@ -84,7 +84,9 @@ established after you run the OpenEmbedded build environment setup
 script (i.e. :ref:`structure-core-script`).
 Adding the root directory allows the :term:`OpenEmbedded Build System`
 to recognize the BSP
-layer and from it build an image. Here is an example::
+layer and from it build an image. Here is an example:
+
+.. code-block:: bitbake
 
    BBLAYERS ?= " \
       /usr/local/src/yocto/meta \
@@ -112,7 +114,9 @@ are known as ":term:`container layers <Container Layer>`". An example of
 this type of layer is OpenEmbedded's :oe_git:`meta-openembedded </meta-openembedded>`
 layer. The ``meta-openembedded`` layer contains many ``meta-*`` layers.
 In cases like this, you need to include the names of the actual layers
-you want to work with, such as::
+you want to work with, such as:
+
+.. code-block:: bitbake
 
    BBLAYERS ?= " \
      /usr/local/src/yocto/meta \
@@ -531,7 +535,9 @@ identifies the contents of the layer, and contains information about how
 the build system should use it. Generally, a standard boilerplate file
 such as the following works. In the following example, you would replace
 "bsp" with the actual name of the BSP (i.e. "bsp_root_name" from the example
-template). ::
+template).
+
+.. code-block:: bitbake
 
    # We have a conf and classes directory, add to BBPATH
    BBPATH .= ":${LAYERDIR}"
@@ -546,7 +552,9 @@ template). ::
    LAYERDEPENDS_bsp = "intel"
 
 To illustrate the string substitutions, here are the corresponding
-statements from the Raspberry Pi ``conf/layer.conf`` file::
+statements from the Raspberry Pi ``conf/layer.conf`` file:
+
+.. code-block:: bitbake
 
    # We have a conf and classes directory, append to BBPATH
    BBPATH .= ":${LAYERDIR}"
@@ -603,7 +611,9 @@ For example, many ``tune-*`` files (e.g. ``tune-arm1136jf-s.inc``,
 
 To use an include file, you simply include them in the machine
 configuration file. For example, the Raspberry Pi BSP
-``raspberrypi3.conf`` contains the following statement::
+``raspberrypi3.conf`` contains the following statement:
+
+.. code-block:: bitbake
 
    include conf/machine/include/rpi-base.inc
 
@@ -674,7 +684,9 @@ Suppose you are using the ``linux-yocto_6.12.bb`` recipe to build the
 kernel. In other words, you have selected the kernel in your
 ``"bsp_root_name".conf`` file by adding
 :term:`PREFERRED_PROVIDER` and :term:`PREFERRED_VERSION`
-statements as follows::
+statements as follows:
+
+.. code-block:: bitbake
 
    PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
    PREFERRED_VERSION_linux-yocto ?= "6.12%"
@@ -1027,7 +1039,9 @@ BSP-specific configuration file named ``interfaces`` to the
 also supports several other machines:
 
 #. Edit the ``init-ifupdown_1.0.bbappend`` file so that it contains the
-   following::
+   following:
+
+   .. code-block:: bitbake
 
       FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
 
@@ -1195,7 +1209,9 @@ BSP Layer Configuration Example
 -------------------------------
 
 The layer's ``conf`` directory contains the ``layer.conf`` configuration
-file. In this example, the ``conf/layer.conf`` file is the following::
+file. In this example, the ``conf/layer.conf`` file is the following:
+
+.. code-block:: bitbake
 
    # We have a conf and classes directory, add to BBPATH
    BBPATH .= ":${LAYERDIR}"
@@ -1375,7 +1391,9 @@ BSP Kernel Recipe Example
 -------------------------
 
 The kernel recipe used to build the kernel image for the BeagleBone
-device was established in the machine configuration::
+device was established in the machine configuration:
+
+.. code-block:: bitbake
 
    PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
    PREFERRED_VERSION_linux-yocto ?= "6.1%"
@@ -1386,7 +1404,9 @@ metadata used to build the kernel. In this case, a kernel append file
 kernel recipe (i.e. ``linux-yocto_6.1.bb``), which is located in
 :oe_git:`/openembedded-core/tree/meta/recipes-kernel/linux`.
 
-The contents of the append file are::
+The contents of the append file are:
+
+.. code-block:: bitbake
 
    KBRANCH:genericx86  = "v6.1/standard/base"
    KBRANCH:genericx86-64  = "v6.1/standard/base"
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#7 lore raw
In reply to Trevor Woerner

Hello docs people!

In today's YP tech call I mentioned that I had been using reST/Sphinx extensively recently and had a couple updates I thought might be useful to the project. I have submitted this RFC patchset to gauge interest in this proposal. If you like it I can submit a full patchset with updates for all document.

Thanks and best regards,
Trevor

#2 lore raw

Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer

In reply to Trevor Woerner

Hi,

On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides
+# one.

Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.

Thanks,
Antonin

#3 lore raw

Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer

Quoted 9 lines · Antonin Godard

Hi,

On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides
+# one.

Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.

All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releases

I will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.

https://pygments.org/docs/contributing/

Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.

It was interesting to me to discover that our own documentation does not know how to handle bitbake syntax.

#4 lore raw

Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer

Quoted 20 lines · Trevor Woerner
Quoted 9 lines · Antonin Godard

Hi,

On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides
+# one.

Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.

All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releases

I will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.

Quoted 7 lines

https://pygments.org/docs/contributing/

Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.

We'll see what they think: https://github.com/pygments/pygments/pull/3103

Quoted 3 lines

It was interesting to me to discover that our own documentation does not know how to handle bitbake syntax.

#5 lore raw

Re: [docs] [RFC PATCH 1/2] add a "bitbake" pygments lexer

Hi,

Quoted 23 lines · Trevor Woerner
Quoted 22 lines · Trevor Woerner
Quoted 9 lines · Antonin Godard

Hi,

On Tue Apr 21, 2026 at 7:24 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides
+# one.

Have you considered contributing this lexer to Pygments instead? I would rather have it there than having to maintain it here.

All development on Pygments occurs on github: https://github.com/pygments/pygments
Currently there are 143 open pull requests: https://github.com/pygments/pygments/pulls
(some going back to 2019)
However, the latest release was 3 weeks ago: https://github.com/pygments/pygments/releases

I will create a pull request for this feature and see where it goes. The contributor information for the Pygments project is very clear that they are not interested in adding/maintaining lexers for "pet" languages.

https://pygments.org/docs/contributing/

Quoted 7 lines
Quoted 5 lines

Of course *we* are all aware of how big the community is that uses bitbake, but will we be able to convince them? bitbake's usage is large, but compared to, say, something like c or python... All I can do is submit and see. Realistically maintaining it ourselves is a likely possibility if we're interested in this feature.

We'll see what they think: https://github.com/pygments/pygments/pull/3103

BitBake does not change syntax often, I would say, so once it's settled in, I think it won't have to change that much.

Thanks for the PR! Let's see where this goes.

Antonin