Skip to contentSkip to conversation

Container improvements

PATCH Superseded Tim Orling 3 participants 42 messages May 30 – Jul 29

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

cover Container improvements

This series:
* Adds a class to create/run containers with a non-root user
* Adds new containers:
- app-container-python
- app-containter-mosquitto
- app-container-valkey
- app-container-nginx
* Modifies app-container-curl to be more like the upstream experience (and more like the above containers)
* Allows meta-webserver/recipes-http to be parsed for vcontainer distro so we can build multiarch containers for app-container-nginx, etc.

Each of these containers was built in a MACHINE=qemuarm64 environment as well as mc:container-amd64+mc:container-arm64 multiarch environment.

The resulting containers were tested with simple command line usage compared to Docker provided equivalents to ensure the same expected behavior.

Tim Orling (7):
classes: add container-nonroot-user.bbclass recipes-containers/images: add app-container-python recipes-containers/images: add app-container-mosquitto recipes-containers/images: add app-container-valkey recipes-containers/images: add app-container-nginx vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd
app-container-curl: use multilayer mode; container-nonroot-user

 classes/container-nonroot-user.bbclass        | 68 ++++++++++++++++
 conf/distro/include/vcontainer-bbmask.inc     |  2 +-
 conf/layer.conf                               |  1 +
 .../images/app-container-curl.bb              | 29 ++++++-
 .../images/app-container-mosquitto.bb         | 46 +++++++++++
 .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
 .../images/app-container-python.bb            | 57 ++++++++++++++
 .../images/app-container-valkey.bb            | 61 +++++++++++++++
 8 files changed, 336 insertions(+), 5 deletions(-)
 create mode 100644 classes/container-nonroot-user.bbclass
 rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
 create mode 100644 recipes-containers/images/app-container-nginx.bb
 create mode 100644 recipes-containers/images/app-container-python.bb
 create mode 100644 recipes-containers/images/app-container-valkey.bb
Signature · 1 line
2.54.0
author
#1 lore raw

PATCH 1/7 classes: add container-nonroot-user.bbclass

For secure and production environments, we want to run containers as a non-root user. Some applications, such as Python, require a $HOME directory with proper permissions. Because OCI_LAYERS :directories:
copies with 'cp -a --no-preserve=ownership', we need a fixup function to create the proper permissions and ownership in a new raw layer.

The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)

---
 classes/container-nonroot-user.bbclass | 68 ++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 classes/container-nonroot-user.bbclass
classes/container-nonroot-user.bbclass new file +68 -0
new file mode 100644
index 00000000..5139ce8b
@@ -0,0 +1,68 @@
+# For secure and production environments, we want to run containers as a
+# non-root user. Some applications, such as Python, require a $HOME
+# directory with proper permissions. Because OCI_LAYERS :directories:
+# copies with 'cp -a --no-preserve=ownership', we need a fixup function
+# to create the proper permissions and ownership in a new raw layer.
+
+# The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)
+
+inherit extrausers
+
+NONROOT_USER ?= "nonroot"
+NONROOT_UID ?= "65532"
+NONROOT_GID ?= "65532"
+
+# ---------------------------------------------------------------------------
+# Create the unprivileged "nonroot" user (uid 65532, group 65532)
+# ---------------------------------------------------------------------------
+EXTRA_USERS_PARAMS = "\
+    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
+    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
+"
+
+# Allow a container to choose to run as 'root'
+OCI_IMAGE_RUNTIME_UID ?= "${NONROOT_UID}"
+OCI_IMAGE_ENV_VARS = "HOME=/home/${NONROOT_USER}"
+
+# Make sure we can write to e.g. /home/nonroot/.python_history
+# using :directories: in OCI_LAYERS does not preserve permissions
+fakeroot fix_oci_home_perms() {
+    cd ${IMGDEPLOYDIR}
+    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
+    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
+
+    rm -f "$layer_tar"
+
+    python3 - "$layer_tar" <<'PYEOF'
+import sys, tarfile, time
+
+layer_tar = sys.argv[1]
+mtime = int(time.time())
+
+# (path, mode, uid, gid)
+entries = [
+    ("home",         0o755, 0,     0),
+    ("home/${NONROOT_USER}", 0o700, ${NONROOT_UID}, ${NONROOT_GID}),
+]
+
+with tarfile.open(layer_tar, "w") as tar:
+    for name, mode, uid, gid in entries:
+        info = tarfile.TarInfo(name=name)
+        info.type  = tarfile.DIRTYPE
+        info.mode  = mode
+        info.uid   = uid
+        info.gid   = gid
+        info.uname = ""   # numeric-only; let umoci canonicalize
+        info.gname = ""
+        info.mtime = mtime
+        tar.addfile(info)
+PYEOF
+
+    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
+    rm -f "$layer_tar"
+
+    rm -f "$image_name.tar" "$image_name-dir.tar"
+    ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
+    tar -cf "$image_name-dir.tar" "$image_name"
+}
+do_image_oci[postfuncs] += "fix_oci_home_perms"
Signature · 1 line
2.54.0
author
#2 lore raw

PATCH 2/7 recipes-containers/images: add app-container-python

In reply to Tim Orling

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the REPL:
"Cannot read termcap database;
using dumb terminal settings."

Add coreutils to "python" layer to provide /usr/bin/env needed by python3-idle in python3-modules.

Inherit container-nonroot-user and run a nonroot user by default. Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or distro/image config to run as 'root' and include 'pip'.

---
 .../images/app-container-python.bb            | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 recipes-containers/images/app-container-python.bb
recipes-containers/images/app-container-python.bb new file +57 -0
new file mode 100644
index 00000000..a93f1b0f
@@ -0,0 +1,57 @@
+SUMMARY = "Base python3 container image"
+DESCRIPTION = "OCI container image running Python with non-root user. \
+\
+In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
+developers to simply run 'pip install' on top of this container (Not \
+advised for production/hardened use)."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds python3-pip to the python layer (enables `pip install` at runtime)
+#   - runs the container as root (UID 0) so pip can write to site-packages
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root (required for `pip install`).
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = "python3"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
+# and PM handles installation directly to OCI layers
Signature · 1 line
2.54.0
author
#5 lore raw

PATCH 3/7 recipes-containers/images: add app-container-mosquitto

In reply to Tim Orling

Add OCI container image recipe for the Eclipse Mosquitto MQTT broker. The image uses multi-layer mode with separate base and mosquitto layers, exposes standard MQTT (1883) and WebSocket (9001) ports, and launches mosquitto with its default config file as the entrypoint.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

---
 .../images/app-container-mosquitto.bb         | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
recipes-containers/images/app-container-mosquitto.bb new file +46 -0
new file mode 100644
index 00000000..37e34a47
@@ -0,0 +1,46 @@
+SUMMARY = "Mosquitto MQTT broker container image"
+DESCRIPTION = "OCI container running the Eclipse Mosquitto MQTT broker \
+with standard MQTT (1883) and WebSocket (9001) listeners enabled."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    mosquitto:packages:mosquitto \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    mosquitto \
+"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}
+
+OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"
+OCI_IMAGE_PORTS = "1883/tcp 9001/tcp"
+OCI_IMAGE_TAG = "latest"
Signature · 1 line
2.54.0
author
#6 lore raw

PATCH 4/7 recipes-containers/images: add app-container-valkey

In reply to Tim Orling

Add OCI container image recipe for the Valkey in-memory key-value datastore. The image uses multi-layer mode with separate base and valkey layers, exposes the standard Valkey port (6379), and launches valkey-server with its default config file as the entrypoint.

The stock valkey.conf shipped by meta-oe is tuned for a host install (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at launch so the server stays in the foreground as PID 1, logs to stdout, and is reachable from outside the container.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by default. Can optionally set PACKAGECONFIG:pn-app-container-valkey = "dev" in local.conf or a distro/image config to run as root.

---
 .../images/app-container-valkey.bb            | 61 +++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 recipes-containers/images/app-container-valkey.bb
recipes-containers/images/app-container-valkey.bb new file +61 -0
new file mode 100644
index 00000000..b3da2efd
@@ -0,0 +1,61 @@
+SUMMARY = "Valkey key-value store container image"
+DESCRIPTION = "OCI container running the Valkey in-memory key-value \
+datastore, a flexible distributed datastore that supports both caching \
+and beyond caching workloads."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-valkey = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    valkey:packages:valkey \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    valkey \
+"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}
+
+OCI_IMAGE_ENTRYPOINT = "${bindir}/valkey-server"
+# The stock valkey.conf shipped by meta-oe is tuned for a host install
+# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
+# launch so the server stays in the foreground as PID 1, logs to stdout,
+# and is reachable from outside the container.
+OCI_IMAGE_ENTRYPOINT_ARGS = "'${sysconfdir}/valkey/valkey.conf' \
+    --daemonize no \
+    --syslog-enabled no \
+    --bind '0.0.0.0 -::*' \
+    --protected-mode no"
+OCI_IMAGE_PORTS = "6379/tcp"
+OCI_IMAGE_TAG = "latest"
Signature · 1 line
2.54.0
author
#7 lore raw

PATCH 5/7 recipes-containers/images: add app-container-nginx

In reply to Tim Orling

Add OCI container image recipe for the NGINX web server. The image uses multi-layer mode with separate base, nginx packages, nginx runtime directories, and nginx log file layers. Exposes the standard HTTP port (80) and launches nginx with 'daemon off' so it stays in the foreground as PID 1 and logs to stderr.

Add ROOTFS_POSTPROCESS fixups to create the runtime directories nginx expects: /var/volatile/{tmp,log}, /var/log/nginx (resolved explicitly to guarantee inclusion in the container layer regardless of /var/log symlink ordering), and /run/nginx for nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) which are not created by any package. Also create empty /var/log/nginx/{access,error}.log to avoid do_image_oci warnings.

Inherit container-nonroot-user with NONROOT_USER = "nginx" to run with UID 65532 by default. Set PACKAGECONFIG:pn-app-container-nginx = "dev" in local.conf or distro/image config to run as 'root'.

Add SKIP_RECIPE and comment to layer.conf since nginx requires meta-webserver.

---
 conf/layer.conf                               |  1 +
 .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 recipes-containers/images/app-container-nginx.bb
conf/layer.conf +1 -0
index 2a4a4c91..6ea8ccf8 100644
@@ -33,6 +33,7 @@ LAYERDEPENDS_virtualization-layer = " \
 # webserver:
 # - naigos requires apache2
 # - cockpit-machines requires cockpit
+# - app-container-nginx requires nginx
 LAYERRECOMMENDS_virtualization-layer = " \
     webserver \
     selinux \
recipes-containers/images/app-container-nginx.bb new file +77 -0

77 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#8 lore raw

PATCH 6/7 vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd

In reply to Tim Orling

Allow us to build nginx, apache2, etc. multiarch containers.

---
 conf/distro/include/vcontainer-bbmask.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
conf/distro/include/vcontainer-bbmask.inc +1 -1
index ac74464f..077255cb 100644
@@ -93,6 +93,7 @@ BBMASK += "meta-networking/recipes-(?!filter|support)"
 BBMASK += "meta-openstack/recipes-dbs/postgresql/"
 BBMASK += "meta-oe/dynamic-layers/networking-layer/recipes-core/"
 BBMASK += "meta-openstack/recipes-extended/libvirt/"
+BBMASK += "meta-webserver/recipes-(?!httpd)"
 
 # ---------------------------------------------------------------------------
 # Entire layers with 0 recipes in the container image dependency graph.
@@ -101,7 +102,6 @@ BBMASK += "meta-openstack/recipes-extended/libvirt/"
 # ---------------------------------------------------------------------------
 BBMASK += "meta-filesystems/"
 BBMASK += "meta-python/"
-BBMASK += "meta-webserver/"
 # Warning suppression for these fully-masked layers is in meta-virt-host.conf
 # (BBFILE_PATTERN_IGNORE_EMPTY) because BitBake checks the base datastore,
 # not per-multiconfig datastores.
Signature · 1 line
2.54.0
author
#9 lore raw

PATCH 7/7 app-container-curl: use multilayer mode; container-nonroot-user

In reply to Tim Orling

* Move to recipes-containers/images to show maintenance intent
* Switch to multilayer mode to more like the other "library"/"official" container recipes.
* Change OCI_IMAGE_TAG to "latest" for similar reasons.
* Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream containers.
* Install ca-certificates to enable handling https:// sites
* Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by default. Set PACKAGECONFIG:pn-app-container-curl = "dev" in local.conf or distro/image config to run as root and include a CONTAINER_SHELL.

---
 .../images/app-container-curl.bb              | 29 ++++++++++++++++---
 1 file changed, 25 insertions(+), 4 deletions(-)
 rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)
recipes-containers/images/app-container-curl.bb renamed from recipes-demo/images/app-container-curl.bb +25 -4
similarity index 58%
rename from recipes-demo/images/app-container-curl.bb
rename to recipes-containers/images/app-container-curl.bb
index ddeb3022..34204fb9 100644
@@ -2,9 +2,31 @@ SUMMARY = "Curl Application container image"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-curl = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    curl:packages:curl+ca-certificates \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root.
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
 IMAGE_FSTYPES = "container oci"
 inherit image
 inherit image-oci
+inherit container-nonroot-user
 
 IMAGE_FEATURES = ""
 IMAGE_LINGUAS = ""
@@ -39,8 +61,7 @@ rootfs_fixup_var_volatile () {
 }
 
 OCI_IMAGE_ENTRYPOINT = "curl"
-OCI_IMAGE_TAG = "easy"
-OCI_IMAGE_ENTRYPOINT_ARGS = "http://localhost:80"
-CONTAINER_SHELL = "busybox"
+OCI_IMAGE_TAG = "latest"
+OCI_IMAGE_ENTRYPOINT_ARGS = "--help"
 
-IMAGE_INSTALL:append = " curl"
+IMAGE_INSTALL:append = " curl ca-certificates"
Signature · 1 line
2.54.0
#3 lore raw

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

In reply to Tim Orling
Quoted 41 lines · Tim Orling via lists.yoctoproject.org

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the REPL:
"Cannot read termcap database;
using dumb terminal settings."

Add coreutils to "python" layer to provide /usr/bin/env needed by python3-idle in python3-modules.

Inherit container-nonroot-user and run a nonroot user by default. Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or distro/image config to run as 'root' and include 'pip'.

---
 .../images/app-container-python.bb            | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 recipes-containers/images/app-container-python.bb
recipes-containers/images/app-container-python.bb new file +13 -0
new file mode 100644
index 00000000..a93f1b0f
@@ -0,0 +1,57 @@
+SUMMARY = "Base python3 container image"
+DESCRIPTION = "OCI container image running Python with non-root user. \
+\
+In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
+developers to simply run 'pip install' on top of this container (Not \
+advised for production/hardened use)."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:

recipes-containers/images/app-container-python.bb line 23

Quoted 10 lines
recipes-containers/images/app-container-python.bb new file +10 -0
+#   - adds python3-pip to the python layer (enables `pip install` at runtime)
+#   - runs the container as root (UID 0) so pip can write to site-packages
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \

Hi Tim,

I wonder if we should define the base layer contents in image-oci to ensure that it is consistent across recipes.

E.g. in image-oci.bbclass:

OCI_BASE_LAYER = "base:packages:base-files+base-passwd+netbase"

Then in the recipe:

    OCI_LAYERS = "\
        ${OCI_BASE_LAYER} \
        ...
    "

That gives consistency without forcing all OCI images to use the base layer definition if it isn't relevant to them.

Quoted 15 lines
recipes-containers/images/app-container-python.bb new file +15 -0
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root (required for `pip install`).
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = "python3"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user

recipes-containers/images/app-container-python.bb line 48

Quoted 10 lines
recipes-containers/images/app-container-python.bb new file +10 -0
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"

Maybe we also need an OCI_BASE_PACKAGES to go with OCI_BASE_LAYER. Pretty much every image is going to need these three packages installed.

recipes-containers/images/app-container-python.bb line 57

Quoted 9 lines
recipes-containers/images/app-container-python.bb new file +9 -0
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
+# and PM handles installation directly to OCI layers

Best regards,

Signature · 1 line
Paul Barker
#4 lore raw

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

Quoted 71 lines · Paul Barker via lists.yoctoproject.org
Quoted 16 lines · Tim Orling via lists.yoctoproject.org

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the REPL:
"Cannot read termcap database;
using dumb terminal settings."

Add coreutils to "python" layer to provide /usr/bin/env needed by python3-idle in python3-modules.

Inherit container-nonroot-user and run a nonroot user by default. Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or distro/image config to run as 'root' and include 'pip'.

Review

Quoted 7 lines
---
 .../images/app-container-python.bb            | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 recipes-containers/images/app-container-python.bb
(quoted diff) +0 -0

b/recipes-containers/images/app-container-python.bb

recipes-containers/images/app-container-python.bb line 8

Quoted 13 lines

new file mode 100644
index 00000000..a93f1b0f

recipes-containers/images/app-container-python.bb new file +8 -0
@@ -0,0 +1,57 @@
+SUMMARY = "Base python3 container image"
+DESCRIPTION = "OCI container image running Python with non-root user. \
+\
+In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
+developers to simply run 'pip install' on top of this container (Not \
+advised for production/hardened use)."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM =

"file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"

recipes-containers/images/app-container-python.bb line 10

Quoted 2 lines
recipes-containers/images/app-container-python.bb new file +2 -0
+
+# Multi-layer mode: create explicit layers instead of single rootfs

layer

recipes-containers/images/app-container-python.bb

Quoted 4 lines
recipes-containers/images/app-container-python.bb new file +4 -0
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds python3-pip to the python layer (enables `pip install` at

runtime)

recipes-containers/images/app-container-python.bb line 15

Quoted 1 line
recipes-containers/images/app-container-python.bb new file +1 -0
+#   - runs the container as root (UID 0) so pip can write to

site-packages

recipes-containers/images/app-container-python.bb

Quoted 6 lines
recipes-containers/images/app-container-python.bb new file +6 -0
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for

multiple items

recipes-containers/images/app-container-python.bb line 23

Quoted 2 lines
recipes-containers/images/app-container-python.bb new file +2 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \

Hi Tim,

I wonder if we should define the base layer contents in image-oci to ensure that it is consistent across recipes.

E.g. in image-oci.bbclass:

OCI_BASE_LAYER = "base:packages:base-files+base-passwd+netbase"

Then in the recipe:

Quoted 8 lines
    OCI_LAYERS = "\
        ${OCI_BASE_LAYER} \
        ...
    "

That gives consistency without forcing all OCI images to use the base layer definition if it isn't relevant to them.

We can't (and shouldn't) enforce using packages at the base like that, since specifying layers as Tim is using is simply one way of doing it.

It could just as easily be another image, or one of the other techniques.

Quoted 25 lines

recipes-containers/images/app-container-python.bb line 25

Quoted 3 lines
recipes-containers/images/app-container-python.bb new file +2 -0
+    terminal:packages:ncurses-terminfo-base \
+

python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \

recipes-containers/images/app-container-python.bb

Quoted 3 lines
recipes-containers/images/app-container-python.bb new file +3 -0
+"
+
+# In 'dev' mode, override the nonroot UID inherited from

container-nonroot-user

recipes-containers/images/app-container-python.bb line 30

Quoted 2 lines
recipes-containers/images/app-container-python.bb new file +2 -0
+# so the container runs as root (required for `pip install`).
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev',

'0', '${NONROOT_UID}', d)}"

Quoted 13 lines
recipes-containers/images/app-container-python.bb new file +13 -0
+
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = "python3"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
Quoted 9 lines

recipes-containers/images/app-container-python.bb line 48

Quoted 5 lines
recipes-containers/images/app-container-python.bb new file +5 -0
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"

Maybe we also need an OCI_BASE_PACKAGES to go with OCI_BASE_LAYER. Pretty much every image is going to need these three packages installed.

Again, we shouldn't be overly prescriptive here.

Factoring things out at this point is premature.

Bruce

Quoted 29 lines

recipes-containers/images/app-container-python.bb

Quoted 4 lines
recipes-containers/images/app-container-python.bb new file +3 -0
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev',

'python3-pip', '', d)}"

recipes-containers/images/app-container-python.bb line 57

Quoted 6 lines
recipes-containers/images/app-container-python.bb new file +6 -0
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
+# and PM handles installation directly to OCI layers

Best regards,

--
Paul Barker

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#9842):
https://lists.yoctoproject.org/g/meta-virtualization/message/9842
Mute This Topic: https://lists.yoctoproject.org/mt/119557386/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II
#10 lore raw
In reply to Tim Orling

Just ack'ing the series.

I'm working through issues with the recipe bumps right now, but will get to this shortly and have a closer look.

Bruce

In message: [meta-virtualization][PATCH 0/7] Container improvements

Quoted 59 lines · on 29/05/2026 Tim Orling via lists.yoctoproject.org wrote:

This series:
* Adds a class to create/run containers with a non-root user
* Adds new containers:
- app-container-python
- app-containter-mosquitto
- app-container-valkey
- app-container-nginx
* Modifies app-container-curl to be more like the upstream experience (and more like the above containers)
* Allows meta-webserver/recipes-http to be parsed for vcontainer distro so we can build multiarch containers for app-container-nginx, etc.

Each of these containers was built in a MACHINE=qemuarm64 environment as well as mc:container-amd64+mc:container-arm64 multiarch environment.

The resulting containers were tested with simple command line usage compared to Docker provided equivalents to ensure the same expected behavior.

Tim Orling (7):
classes: add container-nonroot-user.bbclass recipes-containers/images: add app-container-python recipes-containers/images: add app-container-mosquitto recipes-containers/images: add app-container-valkey recipes-containers/images: add app-container-nginx vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd
app-container-curl: use multilayer mode; container-nonroot-user

 classes/container-nonroot-user.bbclass        | 68 ++++++++++++++++
 conf/distro/include/vcontainer-bbmask.inc     |  2 +-
 conf/layer.conf                               |  1 +
 .../images/app-container-curl.bb              | 29 ++++++-
 .../images/app-container-mosquitto.bb         | 46 +++++++++++
 .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
 .../images/app-container-python.bb            | 57 ++++++++++++++
 .../images/app-container-valkey.bb            | 61 +++++++++++++++
 8 files changed, 336 insertions(+), 5 deletions(-)
 create mode 100644 classes/container-nonroot-user.bbclass
 rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
 create mode 100644 recipes-containers/images/app-container-nginx.bb
 create mode 100644 recipes-containers/images/app-container-python.bb
 create mode 100644 recipes-containers/images/app-container-valkey.bb
Signature · 11 lines
2.54.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#9826): https://lists.yoctoproject.org/g/meta-virtualization/message/9826
Mute This Topic: https://lists.yoctoproject.org/mt/119557384/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
#11 lore raw

Re: [meta-virtualization][PATCH 1/7] classes: add container-nonroot-user.bbclass

In reply to Tim Orling

Hi Tim,

A few things on this one — none of them blocking the intent, but worth a small follow-up either as a fixup in this series or a v2.

Quoted 9 lines · Tim Orling

For secure and production environments, we want to run containers as a non-root user. Some applications, such as Python, require a $HOME directory with proper permissions. Because OCI_LAYERS :directories:
copies with 'cp -a --no-preserve=ownership', we need a fixup function to create the proper permissions and ownership in a new raw layer.

The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)

The non-root + DHI-style image is a real gap in what we ship today; glad to see it filled.

Review

Quoted 13 lines
(quoted diff) +13 -0
+inherit extrausers
+
+NONROOT_USER ?= "nonroot"
+NONROOT_UID ?= "65532"
+NONROOT_GID ?= "65532"
+
+# ---------------------------------------------------------------------------
+# Create the unprivileged "nonroot" user (uid 65532, group 65532)
+# ---------------------------------------------------------------------------
+EXTRA_USERS_PARAMS = "\
+    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
+    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
+"

This hard-assigns EXTRA_USERS_PARAMS rather than appending. Any image that inherits container-nonroot-user AND wants its own extrausers (e.g. a recipe that needs an additional service account) loses whatever else was set, and the order of inheritance silently determines who wins.

Suggest:

  EXTRA_USERS_PARAMS += " \
      groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
      useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} \
              -d /home/${NONROOT_USER} ${NONROOT_USER}; \
  "

That way the class composes cleanly with anyone else who sets EXTRA_USERS_PARAMS earlier.

Quoted 10 lines
(quoted diff) +10 -0
+# Make sure we can write to e.g. /home/nonroot/.python_history
+# using :directories: in OCI_LAYERS does not preserve permissions
+fakeroot fix_oci_home_perms() {
+    cd ${IMGDEPLOYDIR}
+    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
+    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
+
+    rm -f "$layer_tar"
+
+    python3 - "$layer_tar" <<'PYEOF'

Review

Quoted 10 lines
(quoted diff) +10 -0
+import sys, tarfile, time
+
+layer_tar = sys.argv[1]
+mtime = int(time.time())
+
+# (path, mode, uid, gid)
+entries = [
+    ("home",         0o755, 0,     0),
+    ("home/${NONROOT_USER}", 0o700, ${NONROOT_UID}, ${NONROOT_GID}),
+]

Two things about the Python heredoc:

1. The 'PYEOF' delimiter uses single quotes, which would normally suppress shell expansion inside the heredoc — but bitbake expands ${NONROOT_USER} etc. at parse time *before* the shell ever sees the body, so the expansion does happen, just via a different path than the reader expects. Worth a one-line comment so the next person doesn't try to "fix" it.

2. While the values are simple here (alphanumeric user, integer uid/gid) it's fine, but if someone ever sets NONROOT_USER to something with a quote or backslash, the Python source is no longer well-formed. Since this is a meta-virt-internal class, low risk — but a comment ("NONROOT_USER must be a bare identifier") would protect it.

Review

Quoted 8 lines
(quoted diff) +8 -0
+    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
+    rm -f "$layer_tar"
+
+    rm -f "$image_name.tar" "$image_name-dir.tar"
+    ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
+    tar -cf "$image_name-dir.tar" "$image_name"
+}
+do_image_oci[postfuncs] += "fix_oci_home_perms"

The repackaging step at the bottom assumes do_image_oci's output layout (image_name.tar / image_name-dir.tar) is stable. If image-oci.bbclass ever changes its packaging naming, this silently produces a broken tarball — there's no error checking against the assumption.

A defensive option: stat the expected files before rm -f / tar -cf, and bbwarn if either is missing rather than recreating from a possibly-empty directory. Or, if there's a helper in image-oci.bbclass that already does the repackaging, call that instead of rebuilding by hand.

Otherwise the series looks good — I'll keep going through 2/7..7/7 and respond per-patch as I have specific notes.

Bruce

#13 lore raw

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

In reply to Tim Orling

Hi Tim,

Just a couple of comments ...

Quoted 3 lines · Tim Orling

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

[...]

Review

Quoted 5 lines
(quoted diff) +5 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"

I like this conditional. Putting the bb.utils.contains() directly inside the layer's package list (instead of duplicating the OCI_LAYERS declaration in two PACKAGECONFIG branches).

Worth calling out in our multi-layer mode docs as the recommended way to do conditional packages-per-layer. I'll do a patch for that.

Review

Quoted 8 lines
(quoted diff) +8 -0
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"

This is correct today but it doubles the source of truth. The packages listed in OCI_LAYERS:packages: and the packages listed in IMAGE_INSTALL have to be kept in sync.

If they drift (change OCI_LAYERS but forget IMAGE_INSTALL or vice versa), the build silently breaks at layer assembly time when the missing package isn't in DEPLOY_DIR_IPK.

Two paths I'd consider, either as part of this series or as a follow-up:

a) Add a "# KEEP IN SYNC WITH OCI_LAYERS" comment so the next maintainer knows.

b) Better: derive IMAGE_INSTALL from OCI_LAYERS:packages: layers inside image-oci.bbclass, so the recipe only sets it once. That fixes the problem for every multi-layer recipe, not just python.

I've staged b) on master-next (shortly), if you can do a) .. or maybe it isn't needed at all now.

We can do the same for all the other similar recipes in the series.

One more, also non-blocking — the other 4 new image recipes in this series (mosquitto, valkey, nginx, curl) all carry the rootfs_fixup_var_volatile postprocess to create /var/volatile/{tmp,log}.

This one doesn't. Intentional because python doesn't touch /var/log? Or worth adding for consistency in case someone runs as 'dev' and pip wants a writable spot?

Bruce

#15 lore raw

Re: [meta-virtualization][PATCH 3/7] recipes-containers/images: add app-container-mosquitto

In reply to Tim Orling

Hi Tim,

A few comments — most are series-level patterns that show up here for the first time. I won't repeat them later to not waste our time

Quoted 11 lines · Tim Orling

Add OCI container image recipe for the Eclipse Mosquitto MQTT broker. The image uses multi-layer mode with separate base and mosquitto layers, exposes standard MQTT (1883) and WebSocket (9001) ports, and launches mosquitto with its default config file as the entrypoint.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

(quoted diff) +4 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    mosquitto:packages:mosquitto \
+"

[...]

Review

Quoted 6 lines
(quoted diff) +6 -0
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    mosquitto \
+"

Same point as 2/7 — image-oci.bbclass now folds packages: layers into IMAGE_INSTALL automatically (I pushed to master-next). When you respin, this block can go.

Review

Quoted 6 lines
(quoted diff) +6 -0
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}

This same function appears in 3/7 (mosquitto), 4/7 (valkey), 5/7 (nginx), and 7/7 (curl) — four near-identical copies. Worth factoring into a small helper bbclass (e.g. container-volatile-fixup.bbclass) that each recipe can inherit, or rolling it into container-nonroot-user.bbclass since every recipe that uses the fixup also inherits the nonroot class. Not blocking — but it's a smell that's easy to silence in v2.

Review

Quoted 2 lines
(quoted diff) +2 -0
+OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"

Two questions about running mosquitto as our nonroot (uid 65532):

1. The mosquitto package usually creates a 'mosquitto' system user (low uid, varies by build) and ships a default config that references paths under /var/lib/mosquitto/ and /var/log/mosquitto/ owned by that user. As nonroot we won't be the package's expected user. Does mosquitto -c on the stock config actually start cleanly for you, or did you need to tweak the conf?

I assume it runs fie, since you've been testing it for a while

2. If persistence is enabled in the stock mosquitto.conf (persistence true; persistence_location /var/lib/mosquitto/), we need a writable /var/lib/mosquitto for our nonroot user — same flavour of gap the rootfs_fixup_var_volatile workaround addresses, just for a different path. If you've already validated that persistence is off in the stock conf (or that mosquitto degrades gracefully when the persistence dir isn't writable), a one-line comment in the recipe explaining the trade-off would save the next person from re-deriving it.

Bruce

Quoted 2 lines
(quoted diff) +2 -0
+OCI_IMAGE_PORTS = "1883/tcp 9001/tcp"
+OCI_IMAGE_TAG = "latest"
#17 lore raw

Re: [meta-virtualization][PATCH 4/7] recipes-containers/images: add app-container-valkey

In reply to Tim Orling

Hi Tim,

Two valkey-specific comments — the series-level points from 2/7 and 3/7 apply here too but I won't re-raise them.

Quoted 4 lines · Tim Orling

Add OCI container image recipe for the Valkey in-memory key-value datastore. The image uses multi-layer mode with separate base and valkey layers, exposes the standard Valkey port (6379), and launches valkey-server with its default config file as the entrypoint.

[...]

Review

Quoted 10 lines
(quoted diff) +10 -0
+OCI_IMAGE_ENTRYPOINT = "${bindir}/valkey-server"
+# The stock valkey.conf shipped by meta-oe is tuned for a host install
+# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
+# launch so the server stays in the foreground as PID 1, logs to stdout,
+# and is reachable from outside the container.
+OCI_IMAGE_ENTRYPOINT_ARGS = "'${sysconfdir}/valkey/valkey.conf' \
+    --daemonize no \
+    --syslog-enabled no \
+    --bind '0.0.0.0 -::*' \
+    --protected-mode no"

The override-the-conf-at-launch trick is nice, I like it, and it's documented in the comments.

What about a description of --protected-mode no ?

I quickly searched and found with protected-mode off and bind on all interfaces, anyone who can reach the container gets unauthenticated access — that's the right default for a base image people will pull and customise, but a deployer who slaps this into production without adding AUTH or namespace isolation gets a surprise. Worth a single sentence in DESCRIPTION saying so explicitly:

Maybe this ?

  DESCRIPTION = "OCI container running the Valkey in-memory key-value \
  datastore, a flexible distributed datastore that supports both caching \
  and beyond caching workloads. This image runs with protected-mode \
  disabled and bound to all interfaces, intended as a base for \
  customisation — production deployments should enable requirepass / \
  ACLs or restrict the network namespace."

The second is the same persistence question I asked on mosquitto. The stock meta-oe valkey.conf may or may not enable AOF / RDB snapshotting (appendonly yes / save <seconds> <writes>). If either is on, valkey-server tries to write to its dir (usually /var/lib/valkey). Running as our nonroot uid 65532 against a /var/lib/valkey owned by the valkey package user will fail-to-write or worse, silently lose data.

Did you confirm during testing whether the stock conf has persistence on? If on, we want a /var/lib/valkey fixup similar to the /var/volatile one. If off, a one-line comment saying so would save the next person from re-deriving it.

Bruce

#19 lore raw

Re: [meta-virtualization][PATCH 5/7] recipes-containers/images: add app-container-nginx

In reply to Tim Orling

Hi Tim,

Most complex recipe in the series, and I like the the runtime-dir handling

A few nginx-specific things, that came up when I was searching up the runtime parts.

Quoted 3 lines · Tim Orling

Add OCI container image recipe for the NGINX web server. The image uses multi-layer mode with separate base, nginx packages, nginx runtime directories, and nginx log file layers.

[...]

Quoted 1 line

+NONROOT_USER = "nginx"

The biggest open question here. The nginx recipe in meta-webserver creates its own 'nginx' user (via useradd in inherit useradd or a postinst), with a uid the package picks. Now we have two paths trying to create user 'nginx':

a) container-nonroot-user.bbclass adds it via extrausers as uid

     65532 / gid 65532 (whatever NONROOT_UID/GID are set to).

b) The nginx package's own user-creation step adds it with the package's chosen uid.

Last one to run wins in /etc/passwd, but RPM/dpkg may also refuse a duplicate-uid useradd outright. Did you see any "user already exists" or uid-mismatch noise in do_rootfs? If not, the order happens to be fine in your build but it's fragile.

Are these options for v2 ?

1) Override NONROOT_UID to whatever the nginx package picks for the nginx user. Looks up the package's uid, set NONROOT_UID to match.

2) Use NONROOT_USER = "nginxapp" or similar — a name that doesn't collide with the package's own — and tell nginx to run as that user via the conf file (user nginxapp;) instead of relying on the implicit uid match.

Not blocking but worth a note in the recipe about which approach you chose and why, so we won't get cut and paste propagation without a reason.

Review

Quoted 6 lines
(quoted diff) +6 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    nginx:packages:nginx \
+    nginx-dirs:directories:${localstatedir}/log/nginx+/run/nginx+${localstatedir}/volatile/tmp+${localstatedir}/volatile/log \
+    nginx-files:files:${localstatedir}/log/nginx/access.log+${localstatedir}/log/nginx/error.log \
+"

Nice mix of packages + directories + files layer types in a single recipe — this is exactly the scenario the multi-layer support was built for, glad to see it landing in a real recipe.

Review

Quoted 10 lines
(quoted diff) +10 -0
+    # nginx opens the compiled-in error_log path before reading -c config.
+    # /var/log is typically a symlink to /var/volatile/log in a Yocto rootfs,
+    # so create the target path explicitly to guarantee the directory lands in
+    # the container layer regardless of symlink resolution order.
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log/nginx
+
+    # nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) all
+    # live under /run/nginx, which is not created by any package.
+    install -m 755 -d ${IMAGE_ROOTFS}/run/nginx

The /run/nginx catch is great — that's the kind of thing that bites you at first-request time and you can't reproduce without the right URL hitting the right module. Good comment too.

Quoted 1 line

+OCI_IMAGE_APP_RECIPE = "nginx"

Glad to see this in use. It's been sitting in image-oci.bbclass as documentation-only / hook-point, and this is the first recipe that actually sets it. Once we wire it up to auto-extract SRCREV/branch for OCI labels (the "future versions may auto-extract" hook comment in the bbclass), recipes that set it correctly will get the provenance labels for free.

Bruce

#21 lore raw

Re: [meta-virtualization][PATCH 6/7] vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd

In reply to Tim Orling

Hi Tim,

Small focused change, intent is clear. Two questions, one cleanup.

Quoted 1 line · Tim Orling

Allow us to build nginx, apache2, etc. multiarch containers.

[...]

Quoted 1 line

+BBMASK += "meta-webserver/recipes-(?!httpd)"

[...]

Quoted 3 lines

BBMASK += "meta-filesystems/"
BBMASK += "meta-python/"
-BBMASK += "meta-webserver/"

The mask is right and the regex correctly leaves recipes-httpd parseable while keeping everything else under meta-webserver out.

A couple of things worth a follow-up:

1. recipes-httpd contains both nginx and apache2 (and a couple of smaller recipes — hiawatha, lighttpd, monkey iirc). With this change apache2 also becomes parseable for the vcontainer distro, even though only nginx is used by 5/7. Is the apache2 inclusion intentional (to enable an app-container-apache2 follow-up later), or accidental?

If accidental, tightening the regex to recipes-httpd/nginx- might be safer — fewer recipes pulled into parse keeps the dep graph smaller and avoids surprise cascades from apache2's deps.

2. The companion entry in meta-virt-host.conf:

BBFILE_PATTERN_IGNORE_EMPTY_<meta-webserver> = "1" (or similar variable name — the exact form is what meta-virt-host.conf uses for the other fully-masked layers)

...was added because meta-webserver used to be a fully-masked layer producing the "no recipes in this layer" warning. With this change, meta-webserver does parse at least one recipe (nginx), so the IGNORE_EMPTY entry is no longer needed and should be removed in this same commit. Otherwise it sits as dead config.

3. Did you run bitbake -g <container-image> after this change to confirm the dep graph didn't pick up anything heavy you didn't expect? recipes-httpd's deps tend to be fairly contained, but it's worth a one-time check so we know the BBMASK fence is doing what we think it is.

The change itself is fine to apply as-is; (2) is the only thing I'd ask be folded in before merge.

Bruce

#23 lore raw

Re: [meta-virtualization][PATCH 7/7] app-container-curl: use multilayer mode; container-nonroot-user

In reply to Tim Orling

Hi Tim,

Wrap-up patch, mostly aligns curl with the new conventions. One real question, one observation.

Quoted 6 lines · Tim Orling

* Move to recipes-containers/images to show maintenance intent
* Switch to multilayer mode to more like the other "library"/"official" container recipes.
* Change OCI_IMAGE_TAG to "latest" for similar reasons.
* Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream containers.

The directory move right. no issues. recipes-demo was always "here's an example", recipes-containers/images is "this is supported". Same for "easy" → "latest"; matches Docker convention.

The "--help" default is a small but a win ... running the image with no args now self-documents instead of trying to hit a probably-absent localhost:80.

Quoted 5 lines
(quoted diff) +5 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    curl:packages:curl+ca-certificates \
+"

[...]

Quoted 1 line

-CONTAINER_SHELL = "busybox"

This worries me. The old recipe had CONTAINER_SHELL = "busybox" at the top level, so the variable always had a value. The new recipe removes that line and only references ${CONTAINER_SHELL} inside the conditional shell layer.

In non-dev mode that's fine because the whole bb.utils.contains expression resolves to the empty string and the layer doesn't appear.

In dev mode the OCI_LAYERS line expands to:

shell:packages:

…because nothing in the recipe (or in container-nonroot-user.bbclass) sets CONTAINER_SHELL. image-oci.bbclass's validator parses this as a three-field entry with empty content, so it doesn't bb.fatal — but no packages get added to the auto-derived IMAGE_INSTALL, and the resulting layer is empty. The container is silently shipped without a shell, which is exactly the thing dev mode is supposed to provide.

Two ways to fix:

a) Add a default in the recipe near the PACKAGECONFIG declaration:

CONTAINER_SHELL ??= "busybox"

Then anyone enabling dev gets busybox by default and the variable remains overridable.

b) Push the default into container-nonroot-user.bbclass so every recipe that inherits it gets a consistent shell value.

I'd lean (a) — the class shouldn't carry a shell concept; that's a per-recipe choice — but either works.

Quoted 1 line

+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"

This dev-mode-overrides-uid pattern also appears in 2/7 (python). It's clean here, and identical there — another candidate for the helper class refactor whenever the rootfs_fixup_var_volatile factor-out happens, since it's the same shape of "PACKAGECONFIG dev means root".

Quoted 1 line

+IMAGE_INSTALL:append = " curl ca-certificates"

Same series-level point as the earlier recipes — image-oci.bbclass now auto-derives this from OCI_LAYERS in v2 / master-next, so the line can go.

Bruce

author
#12 lore raw

Re: [meta-virtualization][PATCH 1/7] classes: add container-nonroot-user.bbclass

In reply to Bruce Ashfield
Quoted 41 lines · Bruce Ashfield

Hi Tim,

A few things on this one — none of them blocking the intent, but worth a small follow-up either as a fixup in this series or a v2.

Quoted 9 lines · Tim Orling

For secure and production environments, we want to run containers as a non-root user. Some applications, such as Python, require a $HOME directory with proper permissions. Because OCI_LAYERS :directories:
copies with 'cp -a --no-preserve=ownership', we need a fixup function to create the proper permissions and ownership in a new raw layer.

The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)

The non-root + DHI-style image is a real gap in what we ship today; glad to see it filled.

Review

Quoted 13 lines
(quoted diff) +13 -0
+inherit extrausers
+
+NONROOT_USER ?= "nonroot"
+NONROOT_UID ?= "65532"
+NONROOT_GID ?= "65532"
+
+# ---------------------------------------------------------------------------
+# Create the unprivileged "nonroot" user (uid 65532, group 65532)
+# ---------------------------------------------------------------------------
+EXTRA_USERS_PARAMS = "\
+    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
+    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
+"

This hard-assigns EXTRA_USERS_PARAMS rather than appending. Any image that inherits container-nonroot-user AND wants its own extrausers (e.g. a recipe that needs an additional service account) loses whatever else was set, and the order of inheritance silently determines who wins.

Suggest:

Quoted 9 lines
  EXTRA_USERS_PARAMS += " \
      groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
      useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} \
              -d /home/${NONROOT_USER} ${NONROOT_USER}; \
  "

That way the class composes cleanly with anyone else who sets EXTRA_USERS_PARAMS earlier.

Good catch and suggestion. Applied in v2.

Quoted 29 lines
Quoted 10 lines
(quoted diff) +10 -0
+# Make sure we can write to e.g. /home/nonroot/.python_history
+# using :directories: in OCI_LAYERS does not preserve permissions
+fakeroot fix_oci_home_perms() {
+    cd ${IMGDEPLOYDIR}
+    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
+    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
+
+    rm -f "$layer_tar"
+
+    python3 - "$layer_tar" <<'PYEOF'

Review

Quoted 10 lines
(quoted diff) +10 -0
+import sys, tarfile, time
+
+layer_tar = sys.argv[1]
+mtime = int(time.time())
+
+# (path, mode, uid, gid)
+entries = [
+    ("home",         0o755, 0,     0),
+    ("home/${NONROOT_USER}", 0o700, ${NONROOT_UID}, ${NONROOT_GID}),
+]

Two things about the Python heredoc:

1. The 'PYEOF' delimiter uses single quotes, which would normally suppress shell expansion inside the heredoc — but bitbake expands ${NONROOT_USER} etc. at parse time *before* the shell ever sees the body, so the expansion does happen, just via a different path than the reader expects. Worth a one-line comment so the next person doesn't try to "fix" it.

Quoted 6 lines

2. While the values are simple here (alphanumeric user, integer uid/gid) it's fine, but if someone ever sets NONROOT_USER to something with a quote or backslash, the Python source is no longer well-formed. Since this is a meta-virt-internal class, low risk — but a comment ("NONROOT_USER must be a bare identifier") would protect it.

Added comments for 1. and 2. in v2.

Quoted 14 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
+    rm -f "$layer_tar"
+
+    rm -f "$image_name.tar" "$image_name-dir.tar"
+    ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
+    tar -cf "$image_name-dir.tar" "$image_name"
+}
+do_image_oci[postfuncs] += "fix_oci_home_perms"

The repackaging step at the bottom assumes do_image_oci's output layout (image_name.tar / image_name-dir.tar) is stable. If image-oci.bbclass ever changes its packaging naming, this silently produces a broken tarball — there's no error checking against the assumption.

Quoted 5 lines

A defensive option: stat the expected files before rm -f / tar -cf, and bbwarn if either is missing rather than recreating from a possibly-empty directory. Or, if there's a helper in image-oci.bbclass that already does the repackaging, call that instead of rebuilding by hand.

There is no helper in image-oci.bbclass, the repackaging is handled in do_image_oci(). Added logic gated on OCI_IMAGE_TAR_OUTPUT to detect the presence of $image_name/index.json and throw bbfatal if it is not found. We could bbwarn, but I think we should fail loudly.

Quoted 4 lines

Otherwise the series looks good — I'll keep going through 2/7..7/7 and respond per-patch as I have specific notes.

Bruce

author
#14 lore raw

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

In reply to Bruce Ashfield
Quoted 18 lines · Bruce Ashfield

Hi Tim,

Just a couple of comments ...

Quoted 3 lines · Tim Orling

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

[...]

Review

Quoted 5 lines
(quoted diff) +5 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"
Quoted 6 lines

I like this conditional. Putting the bb.utils.contains() directly inside the layer's package list (instead of duplicating the OCI_LAYERS declaration in two PACKAGECONFIG branches).

Worth calling out in our multi-layer mode docs as the recommended way to do conditional packages-per-layer. I'll do a patch for that.

Just capturing that conditional packages-per-layer docs merged in:
27e41b91 docs/container-bundling: refresh multi-layer mode section

Quoted 23 lines

Review

Quoted 9 lines
(quoted diff) +8 -0
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"

This is correct today but it doubles the source of truth. The packages listed in OCI_LAYERS:packages: and the packages listed in IMAGE_INSTALL have to be kept in sync.

If they drift (change OCI_LAYERS but forget IMAGE_INSTALL or vice versa), the build silently breaks at layer assembly time when the missing package isn't in DEPLOY_DIR_IPK.

Two paths I'd consider, either as part of this series or as a follow-up:

a) Add a "# KEEP IN SYNC WITH OCI_LAYERS" comment so the next maintainer knows.

Quoted 6 lines

b) Better: derive IMAGE_INSTALL from OCI_LAYERS:packages: layers inside image-oci.bbclass, so the recipe only sets it once. That fixes the problem for every multi-layer recipe, not just python.

I've staged b) on master-next (shortly), if you can do a) .. or maybe it isn't needed at all now.

Option b) was merged in:
5470cd0a image-oci: auto-derive IMAGE_INSTALL from OCI_LAYERS packages layers

I don't think a) is needed anymore, since b) is proven to be working

Refactored to remove the redundant/brittle IMAGE_INSTALL in v2.

Quoted 2 lines

We can do the same for all the other similar recipes in the series.

Other similar recipes addressed in v2.

Quoted 8 lines

One more, also non-blocking — the other 4 new image recipes in this series (mosquitto, valkey, nginx, curl) all carry the rootfs_fixup_var_volatile postprocess to create /var/volatile/{tmp,log}.

This one doesn't. Intentional because python doesn't touch /var/log? Or worth adding for consistency in case someone runs as 'dev' and pip wants a writable spot?

Added rootfs_fixup_var_volatile via 'inherit container-volatile-fixup' in v2.

Quoted 2 lines

Bruce

author
#16 lore raw

Re: [meta-virtualization][PATCH 3/7] recipes-containers/images: add app-container-mosquitto

In reply to Bruce Ashfield
Quoted 30 lines · Bruce Ashfield

Hi Tim,

A few comments — most are series-level patterns that show up here for the first time. I won't repeat them later to not waste our time

Quoted 13 lines · Tim Orling

Add OCI container image recipe for the Eclipse Mosquitto MQTT broker. The image uses multi-layer mode with separate base and mosquitto layers, exposes standard MQTT (1883) and WebSocket (9001) ports, and launches mosquitto with its default config file as the entrypoint.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

(quoted diff) +4 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    mosquitto:packages:mosquitto \
+"

[...]

Review

Quoted 6 lines
(quoted diff) +6 -0
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    mosquitto \
+"
Quoted 3 lines

Same point as 2/7 — image-oci.bbclass now folds packages: layers into IMAGE_INSTALL automatically (I pushed to master-next). When you respin, this block can go.

Removed redundant IMAGE_INSTALL in v2.

Quoted 14 lines

Review

Quoted 7 lines
(quoted diff) +6 -0
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}

This same function appears in 3/7 (mosquitto), 4/7 (valkey), 5/7 (nginx), and 7/7 (curl) — four near-identical copies. Worth factoring into a small helper bbclass (e.g. container-volatile-fixup.bbclass) that each recipe can inherit, or rolling it into container-nonroot-user.bbclass since every recipe that uses the fixup also inherits the nonroot class. Not blocking — but it's a smell that's easy to silence in v2.

Added container-volatile-fixup.bbclass in v2.

Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"

Two questions about running mosquitto as our nonroot (uid 65532):

Quoted 8 lines

1. The mosquitto package usually creates a 'mosquitto' system user (low uid, varies by build) and ships a default config that references paths under /var/lib/mosquitto/ and /var/log/mosquitto/ owned by that user. As nonroot we won't be the package's expected user. Does mosquitto -c on the stock config actually start cleanly for you, or did you need to tweak the conf?

I assume it runs fie, since you've been testing it for a while

It turns out that in v1 we were not really running as the 'nonroot' user, because of image-oci and container-nonroot-user both setting OCI_IMAGE_RUNTIME_UID with ?= (which means the order of inheritance influenced the behavior). Part of v2 changes image-oci to OCI_IMAGE_RUNTIME_UID ??= so container-nonroot-user can 'override' it (while still being a 'soft' ?= assignment).

It runs fine in both 'production' (default) and '-dev' flavors.

$ docker run --rm -it registry.yocto.io/library/mosquitto:2

1783090865: Info: running mosquitto as user: nonroot.
1783090865: mosquitto version 2.1.2 starting
1783090865: Config loaded from /etc/mosquitto/mosquitto.conf.
1783090865: Bridge support available.
1783090865: Persistence support available.
1783090865: TLS support available.
1783090865: TLS-PSK support available.
1783090865: Websockets support available.
1783090865: Starting in local only mode. Connections will only be possible from clients running on this machine. 1783090865: Create a configuration file which defines a listener to allow remote access. 1783090865: For more details see https://mosquitto.org/documentation/authentication-methods/
1783090865: Opening ipv4 listen socket on port 1883.
1783090865: Opening ipv6 listen socket on port 1883.
1783090865: mosquitto version 2.1.2 running

$ docker run --rm -it registry.yocto.io/library/mosquitto:2-dev

1783090933: Info: running mosquitto as user: mosquitto.
1783090933: mosquitto version 2.1.2 starting
1783090933: Config loaded from /etc/mosquitto/mosquitto.conf.
1783090933: Bridge support available.
1783090933: Persistence support available.
1783090933: TLS support available.
1783090933: TLS-PSK support available.
1783090933: Websockets support available.
1783090933: Starting in local only mode. Connections will only be possible from clients running on this machine. 1783090933: Create a configuration file which defines a listener to allow remote access. 1783090933: For more details see https://mosquitto.org/documentation/authentication-methods/
1783090933: Opening ipv4 listen socket on port 1883.
1783090933: Opening ipv6 listen socket on port 1883.
1783090933: mosquitto version 2.1.2 running

For '-dev' mode, the container starts as 'root' and setuids to 'mosquitto' user.

One of the other things I noticed in testing is that the 'nonroot' UID 65532 was never making it into /etc/passwd, because the base-passwd package is what was providing /etc/passwd and the useradd/extrausers mechanism installs in IMAGE_ROOTFS, which was not being brought into the container layers. This has also been fixed in v2 of container-nonroot-user.bbclass.

The mosquitto.conf that is built by the recipe has all lines commented out. In practice, end users will almost always want to provide their own mosquitto.conf. For instance, they will probably want to require a password and access control (ACL).

Many real-world uses on target will probably want a multi-container approach, putting a reverse proxy like nginx in front of mosquitto or running with docker-compose.

Quoted 10 lines

2. If persistence is enabled in the stock mosquitto.conf (persistence true; persistence_location /var/lib/mosquitto/), we need a writable /var/lib/mosquitto for our nonroot user — same flavour of gap the rootfs_fixup_var_volatile workaround addresses, just for a different path. If you've already validated that persistence is off in the stock conf (or that mosquitto degrades gracefully when the persistence dir isn't writable), a one-line comment in the recipe explaining the trade-off would save the next person from re-deriving it.

The as-shipped mosquitto.conf has all lines commented out, so persistence is not enabled.

Quoted 2 lines

Bruce

author
#18 lore raw

Re: [meta-virtualization][PATCH 4/7] recipes-containers/images: add app-container-valkey

In reply to Bruce Ashfield
Quoted 28 lines · Bruce Ashfield

Hi Tim,

Two valkey-specific comments — the series-level points from 2/7 and 3/7 apply here too but I won't re-raise them.

Quoted 4 lines · Tim Orling

Add OCI container image recipe for the Valkey in-memory key-value datastore. The image uses multi-layer mode with separate base and valkey layers, exposes the standard Valkey port (6379), and launches valkey-server with its default config file as the entrypoint.

[...]

Review

Quoted 10 lines
(quoted diff) +10 -0
+OCI_IMAGE_ENTRYPOINT = "${bindir}/valkey-server"
+# The stock valkey.conf shipped by meta-oe is tuned for a host install
+# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
+# launch so the server stays in the foreground as PID 1, logs to stdout,
+# and is reachable from outside the container.
+OCI_IMAGE_ENTRYPOINT_ARGS = "'${sysconfdir}/valkey/valkey.conf' \
+    --daemonize no \
+    --syslog-enabled no \
+    --bind '0.0.0.0 -::*' \
+    --protected-mode no"

The override-the-conf-at-launch trick is nice, I like it, and it's documented in the comments.

Quoted 1 line

What about a description of --protected-mode no ?

Although I also like the override-the-conf-at-launch, on further review of upstream containers, they use tini and an entrypoint.sh script to launch with a .conf if passed in on the command line and otherwise no conf.

The whole reason I did the override-the-conf-at-launch was that the default valkey.conf provided by our valkey recipe was not launching cleanly.

I'm a little concerned that by doing the overrides with OCI_IMAGE_ENTRYPOINT_ARGS like this it will be more confusing for the end user to re-use the container.

Quoted 30 lines

I quickly searched and found with protected-mode off and bind on all interfaces, anyone who can reach the container gets unauthenticated access — that's the right default for a base image people will pull and customise, but a deployer who slaps this into production without adding AUTH or namespace isolation gets a surprise. Worth a single sentence in DESCRIPTION saying so explicitly:

Maybe this ?

  DESCRIPTION = "OCI container running the Valkey in-memory key-value \
  datastore, a flexible distributed datastore that supports both caching \
  and beyond caching workloads. This image runs with protected-mode \
  disabled and bound to all interfaces, intended as a base for \
  customisation — production deployments should enable requirepass / \
  ACLs or restrict the network namespace."

The second is the same persistence question I asked on mosquitto. The stock meta-oe valkey.conf may or may not enable AOF / RDB snapshotting (appendonly yes / save <seconds> <writes>). If either is on, valkey-server tries to write to its dir (usually /var/lib/valkey). Running as our nonroot uid 65532 against a /var/lib/valkey owned by the valkey package user will fail-to-write or worse, silently lose data.

Did you confirm during testing whether the stock conf has persistence on? If on, we want a /var/lib/valkey fixup similar to the /var/volatile one. If off, a one-line comment saying so would save the next person from re-deriving it.

Bruce

author
#20 lore raw

Re: [meta-virtualization][PATCH 5/7] recipes-containers/images: add app-container-nginx

In reply to Bruce Ashfield
Quoted 39 lines · Bruce Ashfield

Hi Tim,

Most complex recipe in the series, and I like the the runtime-dir handling

A few nginx-specific things, that came up when I was searching up the runtime parts.

Quoted 3 lines · Tim Orling

Add OCI container image recipe for the NGINX web server. The image uses multi-layer mode with separate base, nginx packages, nginx runtime directories, and nginx log file layers.

[...]

Quoted 1 line

+NONROOT_USER = "nginx"

The biggest open question here. The nginx recipe in meta-webserver creates its own 'nginx' user (via useradd in inherit useradd or a postinst), with a uid the package picks. Now we have two paths trying to create user 'nginx':

a) container-nonroot-user.bbclass adds it via extrausers as uid

     65532 / gid 65532 (whatever NONROOT_UID/GID are set to).

b) The nginx package's own user-creation step adds it with the package's chosen uid.

Last one to run wins in /etc/passwd, but RPM/dpkg may also refuse a duplicate-uid useradd outright. Did you see any "user already exists" or uid-mismatch noise in do_rootfs? If not, the order happens to be fine in your build but it's fragile.

Are these options for v2 ?

1) Override NONROOT_UID to whatever the nginx package picks for the nginx user. Looks up the package's uid, set NONROOT_UID to match.

Quoted 8 lines

2) Use NONROOT_USER = "nginxapp" or similar — a name that doesn't collide with the package's own — and tell nginx to run as that user via the conf file (user nginxapp;) instead of relying on the implicit uid match.

Not blocking but worth a note in the recipe about which approach you chose and why, so we won't get cut and paste propagation without a reason.

Previously, our container was not even running as 'nonroot' user because of inheritance order of image-oci and container-nonroot-user, this has been fixed.

Now, in container-nonroot-user, the UID:GID of the 'nginx' user is changed to the 65532 'nonroot' user values in the layer which provides /etc/passwd. The reason for keeping the 'nginx' user name is that there are a number of places where upstream nginx uses the 'nginx' user name explicitly. This is why the dhi.io/nginx:1.30 container sets the 'nonroot' user to have the 'nginx' user name. See for instance:
https://github.com/docker-hardened-images/catalog/blob/main/image/nginx/debian-13/stable.yaml#L81

One other issue that presented itself once we are running as uid 65532 was directory ownership. This has been fixed with the introduction of the NONROOT_OWNED_DIRS list variable which fixes the ownership in the same 'raw' layer as the '/home/<nonroot_username>' was being fixed.

Quoted 37 lines

Review

Quoted 7 lines
(quoted diff) +6 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    nginx:packages:nginx \
+    nginx-dirs:directories:${localstatedir}/log/nginx+/run/nginx+${localstatedir}/volatile/tmp+${localstatedir}/volatile/log \
+    nginx-files:files:${localstatedir}/log/nginx/access.log+${localstatedir}/log/nginx/error.log \
+"

Nice mix of packages + directories + files layer types in a single recipe — this is exactly the scenario the multi-layer support was built for, glad to see it landing in a real recipe.

Review

Quoted 10 lines
(quoted diff) +10 -0
+    # nginx opens the compiled-in error_log path before reading -c config.
+    # /var/log is typically a symlink to /var/volatile/log in a Yocto rootfs,
+    # so create the target path explicitly to guarantee the directory lands in
+    # the container layer regardless of symlink resolution order.
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log/nginx
+
+    # nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) all
+    # live under /run/nginx, which is not created by any package.
+    install -m 755 -d ${IMAGE_ROOTFS}/run/nginx

The /run/nginx catch is great — that's the kind of thing that bites you at first-request time and you can't reproduce without the right URL hitting the right module. Good comment too.

Quoted 1 line

+OCI_IMAGE_APP_RECIPE = "nginx"

Glad to see this in use. It's been sitting in image-oci.bbclass as documentation-only / hook-point, and this is the first recipe that actually sets it. Once we wire it up to auto-extract SRCREV/branch for OCI labels (the "future versions may auto-extract" hook comment in the bbclass), recipes that set it correctly will get the provenance labels for free.

Bruce

author
#22 lore raw

Re: [meta-virtualization][PATCH 6/7] vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd

In reply to Bruce Ashfield
Quoted 30 lines · Bruce Ashfield

Hi Tim,

Small focused change, intent is clear. Two questions, one cleanup.

Quoted 1 line · Tim Orling

Allow us to build nginx, apache2, etc. multiarch containers.

[...]

Quoted 1 line

+BBMASK += "meta-webserver/recipes-(?!httpd)"

[...]

Quoted 3 lines

BBMASK += "meta-filesystems/"
BBMASK += "meta-python/"
-BBMASK += "meta-webserver/"

The mask is right and the regex correctly leaves recipes-httpd parseable while keeping everything else under meta-webserver out.

A couple of things worth a follow-up:

1. recipes-httpd contains both nginx and apache2 (and a couple of smaller recipes — hiawatha, lighttpd, monkey iirc). With this change apache2 also becomes parseable for the vcontainer distro, even though only nginx is used by 5/7. Is the apache2 inclusion intentional (to enable an app-container-apache2 follow-up later), or accidental?

Quoted 3 lines

If accidental, tightening the regex to recipes-httpd/nginx- might be safer — fewer recipes pulled into parse keeps the dep graph smaller and avoids surprise cascades from apache2's deps.

I don't currently plan to support apache2 (I'd rather focus our energy on Caddy). Tightened up the filter to (?!httpd/nginx) in v2.

Quoted 7 lines

2. The companion entry in meta-virt-host.conf:

BBFILE_PATTERN_IGNORE_EMPTY_<meta-webserver> = "1" (or similar variable name — the exact form is what meta-virt-host.conf uses for the other fully-masked layers)

Quoted 5 lines

...was added because meta-webserver used to be a fully-masked layer producing the "no recipes in this layer" warning. With this change, meta-webserver does parse at least one recipe (nginx), so the IGNORE_EMPTY entry is no longer needed and should be removed in this same commit. Otherwise it sits as dead config.

Removed that line in v2.
However, warnings are still produced in vruntime-bbmask-meta-oe.inc Move the BBFILE_PATTERN_IGNORE_EMPTY_webserver = "1" there.

Quoted 11 lines

3. Did you run bitbake -g <container-image> after this change to confirm the dep graph didn't pick up anything heavy you didn't expect? recipes-httpd's deps tend to be fairly contained, but it's worth a one-time check so we know the BBMASK fence is doing what we think it is.

The change itself is fine to apply as-is; (2) is the only thing I'd ask be folded in before merge.

Bruce

author
#24 lore raw

Re: [meta-virtualization][PATCH 7/7] app-container-curl: use multilayer mode; container-nonroot-user

In reply to Bruce Ashfield
Quoted 57 lines · Bruce Ashfield

Hi Tim,

Wrap-up patch, mostly aligns curl with the new conventions. One real question, one observation.

Quoted 6 lines · Tim Orling

* Move to recipes-containers/images to show maintenance intent
* Switch to multilayer mode to more like the other "library"/"official" container recipes.
* Change OCI_IMAGE_TAG to "latest" for similar reasons.
* Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream containers.

The directory move right. no issues. recipes-demo was always "here's an example", recipes-containers/images is "this is supported". Same for "easy" → "latest"; matches Docker convention.

The "--help" default is a small but a win ... running the image with no args now self-documents instead of trying to hit a probably-absent localhost:80.

Quoted 5 lines
(quoted diff) +5 -0
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    curl:packages:curl+ca-certificates \
+"

[...]

Quoted 1 line

-CONTAINER_SHELL = "busybox"

This worries me. The old recipe had CONTAINER_SHELL = "busybox" at the top level, so the variable always had a value. The new recipe removes that line and only references ${CONTAINER_SHELL} inside the conditional shell layer.

In non-dev mode that's fine because the whole bb.utils.contains expression resolves to the empty string and the layer doesn't appear.

In dev mode the OCI_LAYERS line expands to:

shell:packages:

…because nothing in the recipe (or in container-nonroot-user.bbclass) sets CONTAINER_SHELL. image-oci.bbclass's validator parses this as a three-field entry with empty content, so it doesn't bb.fatal — but no packages get added to the auto-derived IMAGE_INSTALL, and the resulting layer is empty. The container is silently shipped without a shell, which is exactly the thing dev mode is supposed to provide.

Two ways to fix:

a) Add a default in the recipe near the PACKAGECONFIG declaration:

CONTAINER_SHELL ??= "busybox"

Quoted 9 lines

Then anyone enabling dev gets busybox by default and the variable remains overridable.

b) Push the default into container-nonroot-user.bbclass so every recipe that inherits it gets a consistent shell value.

I'd lean (a) — the class shouldn't carry a shell concept; that's a per-recipe choice — but either works.

Added classes/container-dev-mode.bbclass which sets CONTAINER_SHELL to 'busybox' when PACKAGECONFIG contains 'dev'. For hardened/production mode it is recommended to add 'container-dummy-provides' to PACKAGE_EXTRA_ARCHS and then the class falls back to 'busybox' if that is not set.

Quoted 7 lines
Quoted 1 line

+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"

This dev-mode-overrides-uid pattern also appears in 2/7 (python). It's clean here, and identical there — another candidate for the helper class refactor whenever the rootfs_fixup_var_volatile factor-out happens, since it's the same shape of "PACKAGECONFIG dev means root".

Good point and this was refactored into container-dev-mode.bbclass.

Quoted 5 lines
Quoted 1 line

+IMAGE_INSTALL:append = " curl ca-certificates"

Same series-level point as the earlier recipes — image-oci.bbclass now auto-derives this from OCI_LAYERS in v2 / master-next, so the line can go.

IMAGE_INSTALL lines removed in v2.

Quoted 2 lines

Bruce

author
#25 lore raw

v2 cover Container Improvements

In reply to Tim Orling

This series:
* Sets OCI_IMAGE_RUNTIME_UID with ??= weak assignment in image-oci to allow container-nonroot-user to set a different default
* Adds a class to create/run containers with a non-root user
* Adds a class to create non-volatile var log and tmp directories
* Adds a class to enable 'dev' mode where the container starts as the root user
* Adds new containers:
- app-container-python
- app-containter-mosquitto
- app-container-valkey
- app-container-nginx
* Modifies app-container-curl to be more like the upstream experience (and more like the above containers)
* Allows mosquitto and libwebsockets to be parsed for vcontainer distro so we can build multiarch containers for app-container-mosquitto.
* Allows meta-webserver/recipes-httpd/nginx to be parsed for vcontainer distro so we can build multiarch containers for app-container-nginx.
* Adds a fix for buildbot-venv shadowing for environment-setup-ci script

The resulting containers were tested with simple command line usage compared to Docker provided equivalents to ensure the same expected behavior.

Changes in v2:

There were a fair number of changes in v2 in reponse to review comments and observations during testing.

* image-oci.bbclass:
- set OCI_IMAGE_RUNTIME_UID with ??= to allow other classes to override
* container-nonroot-user.bbclass:
- set EXTRA_USERS_PARAMS with +=
- added comments to clarify the PYEOF Python heredoc behavior
- added logic to detect $image_name/index.json and fail if not found
* container-volatile-fixup.bbclass: avoid repeating the same rootfs postfunc by adding a helper class
* container-dev-mode.bbclass: avoid repeating the same logic to run container as root user in '-dev' mode
* app-container-python (and other container image recipes):
- drop now redundant IMAGE_INSTALL lines
- simplify by inheriting container-volatile-fixup
- inherit container-dev-mode
* app-container-mosquitto:
- verify '-dev' mode switches from root user to mosquitto user
- verify "production" mode runs as 'nonroot' user
- fix bbmask to allow mosquitto and required libwebsockets
* app-container-valkey:
- added tini and container-entrypoint.sh, like upstream containers
* app-container-nginx:
- fixup 'nonroot' user ownership of NONROOT_OWNED_DIRS
- fix bbmask to allow nginx (but not other recipes-httpd)
* fix for older distro hosts where the 'websockets' Python library is not new enough (>10.0) and on the AutoBuilder needs to use 'buildtools'. The PATH in environment-setup-ci now plays properly with such a set up by no longer added /usr/bin and /bin before existing $PATH.

Successful build https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/25

Tim Orling (13):
image-oci: set OCI_IMAGE_RUNTIME_UID with ??=
classes: add container-nonroot-user.bbclass
classes: add container-volatile-fixup.bbclass
classes: add container-dev-mode.bbclass recipes-containers/images: add app-container-python recipes-containers/images: add app-container-mosquitto vcontainer-bbmask.inc: allow 'mosquitto', 'libwebsockets' recipes-containers/images: add app-container-valkey recipes-containers/images: add app-container-nginx vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd/nginx
app-container-curl: use multilayer mode; container-nonroot-user
container-image-multiarch: add helper recipe
vcontainer-tarball: fix buildbot-venv shadowing

 classes/container-dev-mode.bbclass            |  34 ++++
 classes/container-nonroot-user.bbclass        | 155 ++++++++++++++++++
 classes/container-volatile-fixup.bbclass      |   6 +
 classes/image-oci.bbclass                     |   3 +-
 conf/distro/include/vcontainer-bbmask.inc     |   6 +-
 conf/layer.conf                               |   1 +
 .../images/app-container-curl.bb              |  39 +++++
 .../images/app-container-mosquitto.bb         |  43 +++++
 .../images/app-container-nginx.bb             |  62 +++++++
 .../images/app-container-python.bb            |  45 +++++
 .../images/app-container-valkey.bb            |  71 ++++++++
 .../container-entrypoint.sh                   |  18 ++
 .../images/container-image-multiarch.bb       |  31 ++++
 .../vcontainer/vcontainer-tarball.bb          |   4 +-
 recipes-demo/images/app-container-curl.bb     |  46 ------
 15 files changed, 512 insertions(+), 52 deletions(-)
 create mode 100644 classes/container-dev-mode.bbclass
 create mode 100644 classes/container-nonroot-user.bbclass
 create mode 100644 classes/container-volatile-fixup.bbclass
 create mode 100644 recipes-containers/images/app-container-curl.bb
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
 create mode 100644 recipes-containers/images/app-container-nginx.bb
 create mode 100644 recipes-containers/images/app-container-python.bb
 create mode 100644 recipes-containers/images/app-container-valkey.bb
 create mode 100644 recipes-containers/images/app-container-valkey/container-entrypoint.sh
 create mode 100644 recipes-containers/images/container-image-multiarch.bb
 delete mode 100644 recipes-demo/images/app-container-curl.bb

This series can be found at
https://github.com/moto-timo/meta-virtualization/tree/container-improvements-v2

Signature · 1 line
2.54.0
author
#26 lore raw

PATCH v2 1/13 image-oci: set OCI_IMAGE_RUNTIME_UID with ??=

Allow other subsequently inherited classes, such as container-nonroot-user to set OCI_IMAGE_RUNTIME_UID with ?=

---
 classes/image-oci.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
classes/image-oci.bbclass +2 -1
index 1ceadc7f..9914411b 100644
@@ -94,7 +94,8 @@ OCI_IMAGE_AUTHOR ?= "${PATCH_GIT_USER_NAME}"
 OCI_IMAGE_AUTHOR_EMAIL ?= "${PATCH_GIT_USER_EMAIL}"
 
 OCI_IMAGE_TAG ?= "latest"
-OCI_IMAGE_RUNTIME_UID ?= ""
+# Allow other classes e.g. container-nonroot-user to set with ?=
+OCI_IMAGE_RUNTIME_UID ??= ""
 
 OCI_IMAGE_ARCH ?= "${@oe.go.map_arch(d.getVar('TARGET_ARCH'))}"
 OCI_IMAGE_SUBARCH ?= "${@oci_map_subarch(d.getVar('TARGET_ARCH'), d.getVar('TUNE_FEATURES'), d)}"
Signature · 1 line
2.54.0
author
#27 lore raw

PATCH v2 2/13 classes: add container-nonroot-user.bbclass

In reply to Tim Orling

For secure and production environments, we want to run containers as a non-root user. Some applications, such as Python, require a $HOME directory with proper permissions. Because OCI_LAYERS :directories:
copies with 'cp -a --no-preserve=ownership', we need a fixup function to create the proper permissions and ownership in a new raw layer.

Process a list of NONROOT_OWNED_DIRS which will also be added as owned by the NONROOT_USER.

Add oci_nonroot_inject_user() to do_image_oci[prefuncs] to inject 'nonroot' user into '/etc/passwd' when a layer is present which provides '/etc/passwd', since 'extrausers' acts on IMAGE_ROOTFS and not the 'base-passwd' package.

The behavior here is inspired by dhi.io/python:3 and by dhi.io/valkey:9 (Docker Hardened Images).

---
 classes/container-nonroot-user.bbclass | 155 +++++++++++++++++++++++++
 1 file changed, 155 insertions(+)
 create mode 100644 classes/container-nonroot-user.bbclass
classes/container-nonroot-user.bbclass new file +155 -0
new file mode 100644
index 00000000..09826536
@@ -0,0 +1,155 @@
+# For secure and production environments, we want to run containers as a
+# non-root user. Some applications, such as Python, require a $HOME
+# directory with proper permissions. Because OCI_LAYERS :directories:
+# copies with 'cp -a --no-preserve=ownership', we need a fixup function
+# to create the proper permissions and ownership in a new raw layer.
+
+# The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)
+
+inherit extrausers
+
+# NONROOT_USER must be a bare identifier (no quotes or backslash, etc.)
+NONROOT_USER ?= "nonroot"
+NONROOT_UID ?= "65532"
+NONROOT_GID ?= "65532"
+# Space-separated absolute paths to create in the image, owned by nonroot.
+NONROOT_OWNED_DIRS ?= ""
+
+# ---------------------------------------------------------------------------
+# Create the unprivileged "nonroot" user (uid 65532, group 65532)
+# ---------------------------------------------------------------------------
+EXTRA_USERS_PARAMS += "\
+    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
+    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} \
+            -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
+"
+
+# Allow a container to choose to run as 'root'
+OCI_IMAGE_RUNTIME_UID ?= "${NONROOT_UID}"
+OCI_IMAGE_ENV_VARS = "HOME=/home/${NONROOT_USER}"
+
+# In multi-layer OCI mode the image is assembled from per-layer package
+# installs (oci_multilayer_install_packages in image-oci-umoci.inc), not from
+# IMAGE_ROOTFS. extrausers/EXTRA_USERS_PARAMS only edits IMAGE_ROOTFS, so the
+# nonroot account never reaches the image. Inject it into the layer rootfs that
+# ships /etc/passwd, before IMAGE_CMD:oci assembles the layers.
+python oci_nonroot_inject_user() {
+    import os
+
+    if (d.getVar('OCI_LAYER_MODE') or 'single') != 'multi':
+        return  # single-layer mode builds from IMAGE_ROOTFS; extrausers handles it
+
+    user = d.getVar('NONROOT_USER')
+    uid  = d.getVar('NONROOT_UID')
+    gid  = d.getVar('NONROOT_GID')
+    home = '/home/%s' % user
+
+    # shell field is cosmetic (runtime user is pinned numerically via
+    # OCI config.user); /bin/sh matches dhi.io passwd entries.
+    passwd_line = '%s:x:%s:%s:%s:%s:/bin/sh\n' % (user, uid, gid, user, home)
+    group_line  = '%s:x:%s:\n' % (user, gid)
+    shadow_line = '%s:!:::::::\n' % user
+
+    def append_once(path, line, key):
+        if not os.path.exists(path):
+            return False
+        with open(path) as f:
+            if any(l.startswith(key) for l in f):
+                return True            # already present (idempotent rebuild)
+        with open(path, 'a') as f:
+            f.write(line)
+        return True
+
+    key = user + ':'
+    count = int(d.getVar('OCI_LAYER_COUNT') or 0)
+    found = False
+    for i in range(1, count + 1):
+        rootfs = d.getVar('OCI_LAYER_%d_ROOTFS' % i)
+        if not rootfs:
+            continue
+        # add to every layer that carries /etc/passwd so the topmost wins too
+        if append_once(os.path.join(rootfs, 'etc/passwd'), passwd_line, key):
+            append_once(os.path.join(rootfs, 'etc/group'),  group_line,  key)
+            append_once(os.path.join(rootfs, 'etc/shadow'), shadow_line, key)  # optional
+            found = True
+
+    if not found:
+        bb.warn("container-nonroot-user: no layer ships /etc/passwd; '%s' not "
+                "added — is base-passwd in a packages: layer?" % user)
+}
+
+# Must run AFTER oci_multilayer_install_packages populates OCI_LAYER_*_ROOTFS.
+do_image_oci[prefuncs] += "oci_nonroot_inject_user"
+
+# Make sure we can write to e.g. /home/nonroot/.python_history
+# using :directories: in OCI_LAYERS does not preserve permissions.
+fakeroot fix_oci_home_perms() {
+    cd ${IMGDEPLOYDIR}
+    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
+    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
+
+    rm -f "$layer_tar"
+
+    # BitBake expands ${NONROOT_USER} etc. at parse time *before*
+    # shell sees the body, so single quoted 'PYEOF' is okay.
+    python3 - "$layer_tar" <<'PYEOF'
+import sys, tarfile, time
+
+layer_tar = sys.argv[1]
+mtime = int(time.time())
+
+uid = ${NONROOT_UID}
+gid = ${NONROOT_GID}
+
+# (path, mode, uid, gid)  -- paths are tar-relative, no leading slash
+entries = [
+    ("home",         0o755, 0,     0),
+    ("home/${NONROOT_USER}", 0o700, uid, gid),
+]
+
+seen = {entry[0] for entry in entries}
+for nonrootdir in "${NONROOT_OWNED_DIRS}".split():
+    parts = nonrootdir.strip("/").split("/")
+    for i, _ in enumerate(parts):
+        path = "/".join(parts[:i+1])
+        if path in seen:
+            continue
+        seen.add(path)
+        leaf = (i == len(parts) - 1)
+        # leaf -> nonroot-owned; parents -> root:root, just to scaffold the path
+        entries.append((path, 0o755, uid if leaf else 0, gid if leaf else 0))
+
+with tarfile.open(layer_tar, "w") as tar:
+    for name, mode, uid, gid in entries:
+        info = tarfile.TarInfo(name=name)
+        info.type  = tarfile.DIRTYPE
+        info.mode  = mode
+        info.uid   = uid
+        info.gid   = gid
+        info.uname = ""   # numeric-only; let umoci canonicalize
+        info.gname = ""
+        info.mtime = mtime
+        tar.addfile(info)
+PYEOF
+
+    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
+    rm -f "$layer_tar"
+
+    # Adding the raw layer mutates the OCI image directory, so the tar outputs
+    # produced by do_image_oci are now stale and must be rebuilt. The image
+    # *directory* ($image_name) is the source of truth; the .tar files are
+    # derived from it and only exist when OCI_IMAGE_TAR_OUTPUT is set, so mirror
+    # that gating (see image-oci-umoci.inc) instead of rebuilding unconditionally.
+    # Guard the assumption: an OCI layout always has an index.json, so its
+    # absence means the packaging layout in image-oci.bbclass has changed (or
+    # the dir is empty). Fail loudly rather than emit a silently-broken tarball.
+    if [ -n "${OCI_IMAGE_TAR_OUTPUT}" ]; then
+        if [ ! -f "$image_name/index.json" ]; then
+            bbfatal "fix_oci_home_perms: '$image_name' is not an OCI image layout (no index.json) in ${IMGDEPLOYDIR}; image-oci packaging layout may have changed"
+        fi
+        rm -f "$image_name.tar" "$image_name-dir.tar"
+        ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
+        tar -cf "$image_name-dir.tar" "$image_name"
+    fi
+}
+do_image_oci[postfuncs] += "fix_oci_home_perms"
Signature · 1 line
2.54.0
author
#28 lore raw

PATCH v2 3/13 classes: add container-volatile-fixup.bbclass

In reply to Tim Orling

Add a ROOTFS_POSTPROCESS_COMMAND to fixup 'log' and 'tmp' in /var/volatile.

---
 classes/container-volatile-fixup.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 classes/container-volatile-fixup.bbclass
classes/container-volatile-fixup.bbclass new file +6 -0
new file mode 100644
index 00000000..1b8a4c1e
@@ -0,0 +1,6 @@
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}
Signature · 1 line
2.54.0
author
#29 lore raw

PATCH v2 4/13 classes: add container-dev-mode.bbclass

In reply to Tim Orling

Shared 'dev' mode support for containers-library app images.

Enable per recipe with:
PACKAGECONFIG:pn-<recipe> = "dev"

'dev' mode wants a shell; default 'production' intent does not, for security purposes. Recipes still choose for themselves where CONTAINER_SHELL actually gets consumed (e.g. an OCI_LAYERS 'shell' layer gated on PACKAGECONFIG 'dev') and whether 'dev' should also relax OCI_IMAGE_RUNTIME_UID to run as root.

Each recipe should set the following, we most likely do not want to set at the class level.
PACKAGECONFIG ??= ""
PACKAGECONFIG[dev] = ""

Inspired by the -dev tagged Docker Hardened Images on dhi.io

---
 classes/container-dev-mode.bbclass | 34 ++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 classes/container-dev-mode.bbclass
classes/container-dev-mode.bbclass new file +34 -0
new file mode 100644
index 00000000..cb96fa90
@@ -0,0 +1,34 @@
+# Shared 'dev' mode support for containers-library app images.
+#
+# Enable per recipe with:
+#   PACKAGECONFIG:pn-<recipe> = "dev"
+#
+# 'dev' mode wants a shell; default 'production' intent does not, for
+# security purposes. Recipes still choose for themselves where
+# CONTAINER_SHELL actually gets consumed (e.g. an OCI_LAYERS 'shell' layer
+# gated on PACKAGECONFIG 'dev') and whether 'dev' should also relax
+# OCI_IMAGE_RUNTIME_UID to run as root.
+#
+# Each recipe should set the following, we most likely do not want to set
+# at the class level.
+# PACKAGECONFIG ??= ""
+# PACKAGECONFIG[dev] = ""
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root.
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
+# 'dev' always wins and gets a real shell (busybox), regardless of whether
+# production intent (below) was also configured, e.g. in local.conf.
+#
+# For production, if the following is configured in local.conf (or the
+# distro):
+#      PACKAGE_EXTRA_ARCHS:append = " container-dummy-provides"
+#
+# it has been explicitly indicated that we don't want or need a shell, so
+# we'll add the dummy provides instead of busybox.
+#
+# This is required, since there are postinstall scripts in base-files and
+# base-passwd that reference /bin/sh and we'll get a rootfs error if
+# there's no shell or no dummy provider.
+CONTAINER_SHELL ?= "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'busybox', bb.utils.contains('PACKAGE_EXTRA_ARCHS', 'container-dummy-provides', 'container-dummy-provides', 'busybox', d), d)}"
Signature · 1 line
2.54.0
author
#30 lore raw

PATCH v2 5/13 recipes-containers/images: add app-container-python

In reply to Tim Orling

From: Tim Orling <[email protected]>

Add OCI container image recipe for Python to use as a base for other Python app containers. The image uses multi-layer mode with separate base, terminal and python layers.

Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the REPL:
"Cannot read termcap database;
using dumb terminal settings."

Add coreutils to "python" layer to provide /usr/bin/env needed by python3-idle in python3-modules.

Inherit container-nonroot-user and run a nonroot user by default. Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or distro/image config to run as 'root' and include 'pip'.

Inherit container-volatile-fixup to ensure /var/volatile/{tmp,log} are available, if needed.

Inherit container-dev-mode to optionally run as UID '0' root user and provide a shell.

---
 .../images/app-container-python.bb            | 45 +++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100644 recipes-containers/images/app-container-python.bb
recipes-containers/images/app-container-python.bb new file +45 -0
new file mode 100644
index 00000000..19a64814
@@ -0,0 +1,45 @@
+SUMMARY = "Base python3 container image"
+DESCRIPTION = "OCI container image running Python with non-root user. \
+\
+In 'dev' mode, can optionally run as 'root' and add 'pip' to allow \
+developers to simply run 'pip install' on top of this container (Not \
+advised for production/hardened use)."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds a shell to the container
+#   - adds python3-pip to the python layer (enables `pip install` at runtime)
+#   - runs the container as root (UID 0) so pip can write to site-packages
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+inherit container-dev-mode
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"
+
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = "python3"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+inherit container-volatile-fixup
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
Signature · 1 line
2.54.0
author
#31 lore raw

PATCH v2 6/13 recipes-containers/images: add app-container-mosquitto

In reply to Tim Orling

Add OCI container image recipe for the Eclipse Mosquitto MQTT broker. The image uses multi-layer mode with separate base and mosquitto layers, exposes standard MQTT (1883) and WebSocket (9001) ports, and launches mosquitto with its default config file as the entrypoint.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

Inherit container-volatile-fixup to fix 'log' and 'tmp' in /var/volatile.

Inherit container-dev-mode to optionally run as UID '0' root user and provide a shell.

Relies on image-oci to auto-derive IMAGE_INSTALL from OCI_LAYERS
packages: layers.

---
 .../images/app-container-mosquitto.bb         | 43 +++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
recipes-containers/images/app-container-mosquitto.bb new file +43 -0
new file mode 100644
index 00000000..9a74dcd8
@@ -0,0 +1,43 @@
+SUMMARY = "Mosquitto MQTT broker container image"
+DESCRIPTION = "OCI container running the Eclipse Mosquitto MQTT broker \
+with standard MQTT (1883) and WebSocket (9001) listeners enabled."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds a shell to the container
+#   - adds python3-pip to the python layer (enables `pip install` at runtime)
+#   - runs the container as root (UID 0) so pip can write to site-packages
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+inherit container-dev-mode
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    mosquitto:packages:mosquitto \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+inherit container-volatile-fixup
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"
+OCI_IMAGE_PORTS = "1883/tcp 9001/tcp"
+OCI_IMAGE_TAG = "latest"
Signature · 1 line
2.54.0
author
#32 lore raw

PATCH v2 7/13 vcontainer-bbmask.inc: allow 'mosquitto', 'libwebsockets'

In reply to Tim Orling

Allow meta-networking/recipes-connectivity/mosquitto and required dependency meta-oe/recipes-connectivity/libwebsockets

---
 conf/distro/include/vcontainer-bbmask.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
conf/distro/include/vcontainer-bbmask.inc +2 -2
index ac74464f..f79ea9b9 100644
@@ -88,8 +88,8 @@ BBMASK += "meta-virtualization/recipes-networking/passt/"
 # ---------------------------------------------------------------------------
 # meta-oe / meta-networking: mask heavyweight categories
 # ---------------------------------------------------------------------------
-BBMASK += "meta-oe/recipes-(?!devtools|extended|support)"
-BBMASK += "meta-networking/recipes-(?!filter|support)"
+BBMASK += "meta-oe/recipes-(?!devtools|extended|support|connectivity/libwebsockets)"
+BBMASK += "meta-networking/recipes-(?!filter|support|connectivity/mosquitto)"
 BBMASK += "meta-openstack/recipes-dbs/postgresql/"
 BBMASK += "meta-oe/dynamic-layers/networking-layer/recipes-core/"
 BBMASK += "meta-openstack/recipes-extended/libvirt/"
Signature · 1 line
2.54.0
author
#33 lore raw

PATCH v2 8/13 recipes-containers/images: add app-container-valkey

In reply to Tim Orling

From: Tim Orling <[email protected]>

Add OCI container image recipe for the Valkey in-memory key-value datastore. The image uses multi-layer mode with separate base and valkey layers, exposes the standard Valkey port (6379), and launches valkey-server with its default config file as the entrypoint.

The stock valkey.conf shipped by meta-oe is tuned for a host install (daemonize yes, syslog-enabled yes, bind 127.0.0.1). In most use cases the user will want to provide their own valkey.conf. In that vein, provide container-entrypoint.sh which will use /path/to/valkey.conf passed in on the command line. If no .conf is passed in, we default to the same behavior as upstream containers (no config).

Add container-entrypoint.sh script based on upstream [1]

Add OCI_IMAGE_WORKINGDIR = "/data" to avoid issues writing .rdb on save since 'valkey' will default to '/' for the working dir.

Add NONROOT_OWNED_DIRS = "/data /var/lib/valkey /var/log/valkey /run/valkey" as the 'nonroot' user needs permissions on these directories.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by default. Can optionally set PACKAGECONFIG:pn-app-container-valkey = "dev" in local.conf or a distro/image config to run as 'root'.

Inherit container-volatile-fixup to fix 'log' and 'tmp' in /var/volatile.

Inherit container-dev-mode to optionally run as UID '0' root user. Since we use 'container-entrypoint.sh' which has #!/bin/sh we provide a shell for both '-dev' and 'production' container flavors, which matches upstream Docker Hardened Images behavior.

Relies on image-oci to auto-derive IMAGE_INSTALL from OCI_LAYERS :packages: layers.

[1] https://github.com/valkey-io/valkey-container/blob/mainline/9.1/debian/docker-entrypoint.sh

---
 .../images/app-container-valkey.bb            | 71 +++++++++++++++++++
 .../container-entrypoint.sh                   | 18 +++++
 2 files changed, 89 insertions(+)
 create mode 100644 recipes-containers/images/app-container-valkey.bb
 create mode 100644 recipes-containers/images/app-container-valkey/container-entrypoint.sh
recipes-containers/images/app-container-valkey.bb new file +71 -0
new file mode 100644
index 00000000..10c239e2
@@ -0,0 +1,71 @@
+SUMMARY = "Valkey key-value store container image"
+DESCRIPTION = "OCI container running the Valkey in-memory key-value \
+datastore, a flexible distributed datastore that supports both caching \
+and beyond caching workloads."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-valkey = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+inherit container-dev-mode
+
+# Unlike the other app-container-* recipes, valkey always ships a real shell,
+# in 'dev' and production alike. container-entrypoint.sh is a #!/bin/sh
+# script (upstream-compatible arg handling + umask hardening; see the file
+# for details) and requires an interpreter to boot at all — 'no shell in
+# production' isn't available here without dropping that script and its
+# behavior. Matches DHI's own 'hardened' valkey image (debian-13/9.1.yaml),
+# which ships bash for the exact same reason rather than go shell-less.
+CONTAINER_SHELL = "busybox"
+
+#  image.bbclass intentionally sets do_fetch, do_unpack and do_install to noexec.
+#  We do not want to abuse that isolation, so instead get the file from local
+#  tree and install in rootfs postprocess.
+ROOTFS_POSTPROCESS_COMMAND:append = " rootfs_install_entrypoint_sh ; "
+rootfs_install_entrypoint_sh () {
+    install -m 0755 ${THISDIR}/${BPN}/container-entrypoint.sh ${IMAGE_ROOTFS}/${bindir}/
+}
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    shell:packages:${CONTAINER_SHELL} \
+    valkey:packages:valkey+tini \
+    entrypoint:files:${bindir}/container-entrypoint.sh \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
+# The 'nonroot' user needs permissions on the following directories
+NONROOT_OWNED_DIRS = "/data /var/lib/valkey /var/log/valkey /run/valkey"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+inherit container-volatile-fixup
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# The stock valkey.conf shipped by meta-oe is tuned for a host install
+# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Most users will
+# want to create their own valkey.conf and pass it in to the
+# container-entrypoint.sh script
+OCI_IMAGE_ENTRYPOINT = "${bindir}/docker-init -- ${bindir}/container-entrypoint.sh"
+OCI_IMAGE_CMD = "${bindir}/valkey-server"
+OCI_IMAGE_PORTS = "6379/tcp"
+OCI_IMAGE_TAG = "latest"
+OCI_IMAGE_WORKINGDIR = "/data"
recipes-containers/images/app-container-valkey/container-entrypoint.sh new file +18 -0
new file mode 100644
index 00000000..36cdf93a
@@ -0,0 +1,18 @@
+#!/bin/sh
+# Based on https://github.com/valkey-io/valkey-container/blob/mainline/docker-entrypoint.sh
+# SPDX-License-Identifier: BSD-3-Clause
+set -e
+
+# first arg is `-f` or `--some-option`
+# or first arg is `something.conf`
+if [ "${1#-}" != "$1" ] || [ "${1%.conf}" != "$1" ]; then
+    set -- valkey-server "$@"
+fi
+
+# set an appropriate umask (if one isn't set already)
+um="$(umask)"
+if [ "$um" = '0022' ]; then
+    umask 0077
+fi
+
+exec "$@"
Signature · 1 line
2.54.0
author
#34 lore raw

PATCH v2 9/13 recipes-containers/images: add app-container-nginx

In reply to Tim Orling

From: Tim Orling <[email protected]>

Add OCI container image recipe for the NGINX web server. The image uses multi-layer mode with separate base, nginx packages, nginx runtime directories, and nginx log file layers. Exposes the standard HTTP port (80) and launches nginx with 'daemon off' so it stays in the foreground as PID 1 and logs to stderr.

Add ROOTFS_POSTPROCESS fixups to create the runtime directories nginx expects: /var/volatile/{tmp,log}, /var/log/nginx (resolved explicitly to guarantee inclusion in the container layer regardless of /var/log symlink ordering), and /run/nginx for nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) which are not created by any package. Also create empty /var/log/nginx/{access,error}.log to avoid do_image_oci warnings.

Since nginx uses user 'nginx' by name and not UID, we also need to change ownership of several files/directories for the container to be functional.

Inherit container-nonroot-user with NONROOT_USER = "nginx" to run with UID 65532 by default. Set PACKAGECONFIG:pn-app-container-nginx = "dev" in local.conf or distro/image config to run as 'root'.

Add NONROOT_OWNED_DIRS = "/run/nginx, /var/log/nginx, /var/cache/nginx, /usr/share/nginx/html" as the 'nonroot' user needs permissions on these directories.

Inherit container-volatile-fixup to fix 'log' and 'tmp' in /var/volatile.

Relies on image-oci to auto-derive IMAGE_INSTALL from OCI_LAYERS :packages: layers.

Inherit container-dev-mode to optionally run as UID '0' root user and provide a shell.

Add SKIP_RECIPE and comment to layer.conf since nginx requires meta-webserver.

---
 conf/layer.conf                               |  1 +
 .../images/app-container-nginx.bb             | 62 +++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100644 recipes-containers/images/app-container-nginx.bb
conf/layer.conf +1 -0
index 2a4a4c91..6ea8ccf8 100644
@@ -33,6 +33,7 @@ LAYERDEPENDS_virtualization-layer = " \
 # webserver:
 # - naigos requires apache2
 # - cockpit-machines requires cockpit
+# - app-container-nginx requires nginx
 LAYERRECOMMENDS_virtualization-layer = " \
     webserver \
     selinux \
recipes-containers/images/app-container-nginx.bb new file +62 -0

62 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#35 lore raw

PATCH v2 10/13 vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd/nginx

In reply to Tim Orling

Allow us to build nginx multiarch containers.
If we choose to support apache2 we can add it in the future.

While it seems that BBFILE_PATTERN_IGNORE_EMTPY_webserver = "1" in conf/distro/include/meta-virt-host.conf is no longer needed, we will see warnings from vruntime-* multiconfigs if we remove it.

---
 conf/distro/include/vcontainer-bbmask.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
conf/distro/include/vcontainer-bbmask.inc +1 -1
index f79ea9b9..0c9b8b02 100644
@@ -93,6 +93,7 @@ BBMASK += "meta-networking/recipes-(?!filter|support|connectivity/mosquitto)"
 BBMASK += "meta-openstack/recipes-dbs/postgresql/"
 BBMASK += "meta-oe/dynamic-layers/networking-layer/recipes-core/"
 BBMASK += "meta-openstack/recipes-extended/libvirt/"
+BBMASK += "meta-webserver/recipes-(?!httpd/nginx)"
 
 # ---------------------------------------------------------------------------
 # Entire layers with 0 recipes in the container image dependency graph.
@@ -101,7 +102,6 @@ BBMASK += "meta-openstack/recipes-extended/libvirt/"
 # ---------------------------------------------------------------------------
 BBMASK += "meta-filesystems/"
 BBMASK += "meta-python/"
-BBMASK += "meta-webserver/"
 # Warning suppression for these fully-masked layers is in meta-virt-host.conf
 # (BBFILE_PATTERN_IGNORE_EMPTY) because BitBake checks the base datastore,
 # not per-multiconfig datastores.
Signature · 1 line
2.54.0
author
#36 lore raw

PATCH v2 11/13 app-container-curl: use multilayer mode; container-nonroot-user

In reply to Tim Orling

From: Tim Orling <[email protected]>

* Move to recipes-containers/images to show maintenance intent
* Switch to multilayer mode to more like the other "library"/"official" container recipes.
* Change OCI_IMAGE_TAG to "latest" for similar reasons.
* Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream containers.
* Install ca-certificates to enable handling https:// sites
* Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by default.
* Inherit container-volatile-fixup to fix 'log' and 'tmp' in /var/volatile.
* Inherit container-dev-mode to optionally build a "-dev" image. Set PACKAGECONFIG:pn-app-container-curl = "dev" in local.conf or distro/image config to run as root and include a CONTAINER_SHELL.

---
 .../images/app-container-curl.bb              | 39 ++++++++++++++++
 recipes-demo/images/app-container-curl.bb     | 46 -------------------
 2 files changed, 39 insertions(+), 46 deletions(-)
 create mode 100644 recipes-containers/images/app-container-curl.bb
 delete mode 100644 recipes-demo/images/app-container-curl.bb
recipes-containers/images/app-container-curl.bb new file +39 -0
new file mode 100644
index 00000000..bfa3e708
@@ -0,0 +1,39 @@
+SUMMARY = "Curl Application container image"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds a shell to the container
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-curl = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+inherit container-dev-mode
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    curl:packages:curl+ca-certificates \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+inherit container-volatile-fixup
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+OCI_IMAGE_ENTRYPOINT = "curl"
+OCI_IMAGE_TAG = "latest"
+OCI_IMAGE_ENTRYPOINT_ARGS = "--help"
recipes-demo/images/app-container-curl.bb deleted +0 -46

46 lines not on this page · raw message

Signature · 1 line
2.54.0
author
#37 lore raw

PATCH v2 12/13 container-image-multiarch: add helper recipe

In reply to Tim Orling

Similar to core-image-ptest.bb, this recipe uses mcextend to add multiarch support for any recipe in CONTAINER_IMAGES variable.

---
 .../images/container-image-multiarch.bb       | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 recipes-containers/images/container-image-multiarch.bb
recipes-containers/images/container-image-multiarch.bb new file +31 -0
new file mode 100644
index 00000000..13c4ae24
@@ -0,0 +1,31 @@
+inherit features_check
+REQUIRED_DISTRO_FEATURES = "vcontainer"
+
+
+DESCRIPTION += "Enable building the ${MCNAME} multiarch image."
+SUMMARY ?= "${MCNAME} multiarch image."
+HOMEPAGE ?= "https://www.yoctoproject.org/"
+LICENSE ?= "MIT"
+LIC_FILES_CHKSUM ?= "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+
+CONTAINER_IMAGES ?= "\
+    container-base \
+    app-container-curl \
+    app-container-mosquitto \
+    app-container-nginx \
+    app-container-python \
+    app-container-valkey \
+"
+
+inherit oci-multiarch
+
+BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('CONTAINER_IMAGES').split()])}"
+
+OCI_MULTIARCH_RECIPE = "${MCNAME}"
+OCI_MULTIARCH_PLATFORMS = "aarch64 x86_64"
+
+python () {
+    if not d.getVar("MCNAME"):
+        raise bb.parse.SkipRecipe("No class extension set")
+}
Signature · 1 line
2.54.0
author
#38 lore raw

PATCH v2 13/13 vcontainer-tarball: fix buildbot-venv shadowing

In reply to Tim Orling

Prepend the SDK dirs (vdkr/vpdmn wrappers + native qemu) but keep \$PATH ahead of the host /usr/bin:/bin. The CI parser sources this into the push-containers step where bitbake also runs, and the worker's buildbot-venv is activated via \$PATH; putting /usr/bin first shadowed the venv python and made bitbake pick up the host websockets 9.1 (< 10) -> hashserv failure.

---
 recipes-containers/vcontainer/vcontainer-tarball.bb | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
recipes-containers/vcontainer/vcontainer-tarball.bb +2 -2
index f943cfd0..522cb0ba 100644
@@ -450,8 +450,8 @@ ENVEOF
 # SDK relocation rewrites these paths at install time.
 export VCONTAINER_DIR="${SDKPATH}"
 export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"
-export PATH="${SDKPATH}:${SDKPATHNATIVE}/usr/bin:/usr/bin:/bin:\$PATH"
-# Clean up - unset to avoid confusing other Yocto tools' >> $script
+export PATH="${SDKPATH}:${SDKPATHNATIVE}/usr/bin:\$PATH"
+# Clean up - unset to avoid confusing other Yocto tools'
 unset OECORE_NATIVE_SYSROOT
 CISCRIPT
     chmod 755 $ci_script
Signature · 1 line
2.54.0
#39 lore raw

Re: [meta-virtualization] [PATCH v2 00/13] Container Improvements

In reply to Tim Orling

Thanks Tim,

I'm out of the office for the next 1.5 weeks. so it will likely be after that before I look into this in any detail.

Bruce

Quoted 124 lines · Tim Orling via lists.yoctoproject.org

This series:
* Sets OCI_IMAGE_RUNTIME_UID with ??= weak assignment in image-oci to allow container-nonroot-user to set a different default
* Adds a class to create/run containers with a non-root user
* Adds a class to create non-volatile var log and tmp directories
* Adds a class to enable 'dev' mode where the container starts as the root user
* Adds new containers:
- app-container-python
- app-containter-mosquitto
- app-container-valkey
- app-container-nginx
* Modifies app-container-curl to be more like the upstream experience (and more like the above containers)
* Allows mosquitto and libwebsockets to be parsed for vcontainer distro so we can build multiarch containers for app-container-mosquitto.
* Allows meta-webserver/recipes-httpd/nginx to be parsed for vcontainer distro so we can build multiarch containers for app-container-nginx.
* Adds a fix for buildbot-venv shadowing for environment-setup-ci script

The resulting containers were tested with simple command line usage compared to Docker provided equivalents to ensure the same expected behavior.

Changes in v2:

There were a fair number of changes in v2 in reponse to review comments and observations during testing.

* image-oci.bbclass:
- set OCI_IMAGE_RUNTIME_UID with ??= to allow other classes to override
* container-nonroot-user.bbclass:
- set EXTRA_USERS_PARAMS with +=
- added comments to clarify the PYEOF Python heredoc behavior
- added logic to detect $image_name/index.json and fail if not found
* container-volatile-fixup.bbclass: avoid repeating the same rootfs postfunc by adding a helper class
* container-dev-mode.bbclass: avoid repeating the same logic to run container as root user in '-dev' mode
* app-container-python (and other container image recipes):
- drop now redundant IMAGE_INSTALL lines
- simplify by inheriting container-volatile-fixup
- inherit container-dev-mode
* app-container-mosquitto:
- verify '-dev' mode switches from root user to mosquitto user
- verify "production" mode runs as 'nonroot' user
- fix bbmask to allow mosquitto and required libwebsockets
* app-container-valkey:
- added tini and container-entrypoint.sh, like upstream containers
* app-container-nginx:
- fixup 'nonroot' user ownership of NONROOT_OWNED_DIRS
- fix bbmask to allow nginx (but not other recipes-httpd)
* fix for older distro hosts where the 'websockets' Python library is not new enough (>10.0) and on the AutoBuilder needs to use 'buildtools'. The PATH in environment-setup-ci now plays properly with such a set up by no longer added /usr/bin and /bin before existing $PATH.

Successful build https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/25

Tim Orling (13):
image-oci: set OCI_IMAGE_RUNTIME_UID with ??=
classes: add container-nonroot-user.bbclass
classes: add container-volatile-fixup.bbclass
classes: add container-dev-mode.bbclass recipes-containers/images: add app-container-python recipes-containers/images: add app-container-mosquitto vcontainer-bbmask.inc: allow 'mosquitto', 'libwebsockets' recipes-containers/images: add app-container-valkey recipes-containers/images: add app-container-nginx vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd/nginx
app-container-curl: use multilayer mode; container-nonroot-user
container-image-multiarch: add helper recipe
vcontainer-tarball: fix buildbot-venv shadowing

 classes/container-dev-mode.bbclass            |  34 ++++
 classes/container-nonroot-user.bbclass        | 155 ++++++++++++++++++
 classes/container-volatile-fixup.bbclass      |   6 +
 classes/image-oci.bbclass                     |   3 +-
 conf/distro/include/vcontainer-bbmask.inc     |   6 +-
 conf/layer.conf                               |   1 +
 .../images/app-container-curl.bb              |  39 +++++
 .../images/app-container-mosquitto.bb         |  43 +++++
 .../images/app-container-nginx.bb             |  62 +++++++
 .../images/app-container-python.bb            |  45 +++++
 .../images/app-container-valkey.bb            |  71 ++++++++
 .../container-entrypoint.sh                   |  18 ++
 .../images/container-image-multiarch.bb       |  31 ++++
 .../vcontainer/vcontainer-tarball.bb          |   4 +-
 recipes-demo/images/app-container-curl.bb     |  46 ------
 15 files changed, 512 insertions(+), 52 deletions(-)
 create mode 100644 classes/container-dev-mode.bbclass
 create mode 100644 classes/container-nonroot-user.bbclass
 create mode 100644 classes/container-volatile-fixup.bbclass
 create mode 100644 recipes-containers/images/app-container-curl.bb
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
 create mode 100644 recipes-containers/images/app-container-nginx.bb
 create mode 100644 recipes-containers/images/app-container-python.bb
 create mode 100644 recipes-containers/images/app-container-valkey.bb
 create mode 100644 recipes-containers/images/app-container-valkey/container-entrypoint.sh
 create mode 100644 recipes-containers/images/container-image-multiarch.bb
 delete mode 100644 recipes-demo/images/app-container-curl.bb

This series can be found at
https://github.com/moto-timo/meta-virtualization/tree/container-improvements-v2

--
2.54.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#9924): https://lists.yoctoproject.org/g/meta-virtualization/message/9924
Mute This Topic: https://lists.yoctoproject.org/mt/120143346/1050810
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 3 lines
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II
#40 lore raw

Re: [meta-virtualization] [PATCH v2 00/13] Container Improvements

In reply to Tim Orling

Hi Tim,

Thanks for the substantial rework. Reviewed the whole series against the v1 comments, and the two new pieces you factored out (container-volatile-fixup and container-dev-mode) look great. Series has been applied on master.

Here's my summary, just so we can have it archived and use it as a reference later if something breaks:

v1 items — status
-----------------

Addressed:

* EXTRA_USERS_PARAMS uses += now (was = in v1)
* PYEOF single-quoted heredoc has the "bitbake pre-expansion" comment
* NONROOT_USER bare-identifier requirement documented
* fix_oci_home_perms has the index.json guard + OCI_IMAGE_TAR_OUTPUT gate + bbfatal if the layout ever changes
* IMAGE_INSTALL is no longer duplicated against OCI_LAYERS packages across the image recipes (image-oci auto-derivation on master handles it)
* rootfs_fixup_var_volatile is factored into container-volatile-fixup.bbclass and app-container-python inherits it (was missing in v1)
* container-dev-mode.bbclass cleans up the '-dev' boilerplate that each recipe previously repeated
* app-container-valkey's persistence paths (/data, /var/lib/valkey, /var/log/valkey, /run/valkey) are in NONROOT_OWNED_DIRS
* The valkey security-caveats item is moot in v2 — the container-entrypoint.sh design supersedes the always-on protected-mode override I had asked to be flagged in DESCRIPTION

I did notice a few things on the way through, but it was more efficient if I just did them myself.

All three are behavior-neutral cleanups:

1. app-container-valkey: drop redundant OCI_IMAGE_RUNTIME_UID

The recipe carries a verbatim copy of the same

     OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG',
                                'dev', '0', '${NONROOT_UID}', d)}"

line that container-dev-mode.bbclass now supplies. Leftover from v1.

2. app-container-mosquitto: fix PACKAGECONFIG 'dev' comment

The block above PACKAGECONFIG was copied from app-container-python and mentions python3-pip / site-packages / pn-app-container-python — none of which apply. Rewrote it per-recipe. nginx and curl in the same series already have the correct wording.

3. container-nonroot-user.bbclass: document + assert inherit order

oci_nonroot_inject_user() is a do_image_oci prefunc that reads OCI_LAYER_*_ROOTFS. Those vars are populated by image-oci's OWN do_image_oci prefunc (oci_multilayer_install_packages), and prefunc execution follows inherit order. If a future recipe inherits container-nonroot-user before image-oci, injection runs against empty per-layer state, no layer ships /etc/passwd, and the only signal is a bb.warn that reviewers can miss. Added a header comment spelling out the requirement + an anonymous-python guard that raises bb.fatal at parse time. All five in-tree consumers already inherit in the right order, so this catches a future mistake rather than fixing a current one.

Also — the cover letter says "verify '-dev' mode switches from root user to mosquitto user", but the recipe doesn't set NONROOT_USER = "mosquitto" (unlike nginx which does set it to "nginx"). So either the cover letter is aspirational or something is happening implicitly that I'm missing. Any changes we can do on top of the series. I've put it on master-next for visibility.

Bruce

author
#41 lore raw

Re: [meta-virtualization] [PATCH v2 00/13] Container Improvements

Quoted 33 lines · Bruce Ashfield

Hi Tim,

Thanks for the substantial rework. Reviewed the whole series against the v1 comments, and the two new pieces you factored out (container-volatile-fixup and container-dev-mode) look great. Series has been applied on master.

Here's my summary, just so we can have it archived and use it as a reference later if something breaks:

v1 items — status
-----------------

Addressed:

* EXTRA_USERS_PARAMS uses += now (was = in v1)
* PYEOF single-quoted heredoc has the "bitbake pre-expansion" comment
* NONROOT_USER bare-identifier requirement documented
* fix_oci_home_perms has the index.json guard + OCI_IMAGE_TAR_OUTPUT gate + bbfatal if the layout ever changes
* IMAGE_INSTALL is no longer duplicated against OCI_LAYERS packages across the image recipes (image-oci auto-derivation on master handles it)
* rootfs_fixup_var_volatile is factored into container-volatile-fixup.bbclass and app-container-python inherits it (was missing in v1)
* container-dev-mode.bbclass cleans up the '-dev' boilerplate that each recipe previously repeated
* app-container-valkey's persistence paths (/data, /var/lib/valkey, /var/log/valkey, /run/valkey) are in NONROOT_OWNED_DIRS
* The valkey security-caveats item is moot in v2 — the container-entrypoint.sh design supersedes the always-on protected-mode override I had asked to be flagged in DESCRIPTION

Quoted 2 lines

I did notice a few things on the way through, but it was more efficient if I just did them myself.

Agreed.

Quoted 31 lines

All three are behavior-neutral cleanups:

1. app-container-valkey: drop redundant OCI_IMAGE_RUNTIME_UID

The recipe carries a verbatim copy of the same

     OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG',
                                'dev', '0', '${NONROOT_UID}', d)}"

line that container-dev-mode.bbclass now supplies. Leftover from v1.

2. app-container-mosquitto: fix PACKAGECONFIG 'dev' comment

The block above PACKAGECONFIG was copied from app-container-python and mentions python3-pip / site-packages / pn-app-container-python — none of which apply. Rewrote it per-recipe. nginx and curl in the same series already have the correct wording.

3. container-nonroot-user.bbclass: document + assert inherit order

oci_nonroot_inject_user() is a do_image_oci prefunc that reads OCI_LAYER_*_ROOTFS. Those vars are populated by image-oci's OWN do_image_oci prefunc (oci_multilayer_install_packages), and prefunc execution follows inherit order. If a future recipe inherits container-nonroot-user before image-oci, injection runs against empty per-layer state, no layer ships /etc/passwd, and the only signal is a bb.warn that reviewers can miss. Added a header comment spelling out the requirement + an anonymous-python guard that raises bb.fatal at parse time. All five in-tree consumers already inherit in the right order, so this catches a future mistake rather than fixing a current one.

Quoted 6 lines

Also — the cover letter says "verify '-dev' mode switches from root user to mosquitto user", but the recipe doesn't set NONROOT_USER = "mosquitto" (unlike nginx which does set it to "nginx"). So either the cover letter is aspirational or something is happening implicitly that I'm missing. Any changes we can do on top of the series. I've put it on master-next for visibility.

I addressed this in [1] but let me copy that content here for posterity.

"""
It turns out that in v1 we were not really running as the 'nonroot' user, because of image-oci and container-nonroot-user both setting OCI_IMAGE_RUNTIME_UID with ?= (which means the order of inheritance influenced the behavior). Part of v2 changes image-oci to OCI_IMAGE_RUNTIME_UID ??= so container-nonroot-user can 'override' it (while still being a 'soft' ?= assignment).

It runs fine in both 'production' (default) and '-dev' flavors.

$ docker run --rm -it registry.yocto.io/library/mosquitto:2

1783090865: Info: running mosquitto as user: nonroot.
1783090865: mosquitto version 2.1.2 starting
1783090865: Config loaded from /etc/mosquitto/mosquitto.conf.
1783090865: Bridge support available.
1783090865: Persistence support available.
1783090865: TLS support available.
1783090865: TLS-PSK support available.
1783090865: Websockets support available.
1783090865: Starting in local only mode. Connections will only be possible from clients running on this machine. 1783090865: Create a configuration file which defines a listener to allow remote access. 1783090865: For more details see https://mosquitto.org/documentation/authentication-methods/
1783090865: Opening ipv4 listen socket on port 1883.
1783090865: Opening ipv6 listen socket on port 1883.
1783090865: mosquitto version 2.1.2 running

$ docker run --rm -it registry.yocto.io/library/mosquitto:2-dev

1783090933: Info: running mosquitto as user: mosquitto.
1783090933: mosquitto version 2.1.2 starting
1783090933: Config loaded from /etc/mosquitto/mosquitto.conf.
1783090933: Bridge support available.
1783090933: Persistence support available.
1783090933: TLS support available.
1783090933: TLS-PSK support available.
1783090933: Websockets support available.
1783090933: Starting in local only mode. Connections will only be possible from clients running on this machine. 1783090933: Create a configuration file which defines a listener to allow remote access. 1783090933: For more details see https://mosquitto.org/documentation/authentication-methods/
1783090933: Opening ipv4 listen socket on port 1883.
1783090933: Opening ipv6 listen socket on port 1883.
1783090933: mosquitto version 2.1.2 running

For '-dev' mode, the container starts as 'root' and setuids to 'mosquitto' user.

One of the other things I noticed in testing is that the 'nonroot' UID 65532 was never making it into /etc/passwd, because the base-passwd package is what was providing /etc/passwd and the useradd/extrausers mechanism installs in IMAGE_ROOTFS, which was not being brought into the container layers. This has also been fixed in v2 of container-nonroot-user.bbclass.

The mosquitto.conf that is built by the recipe has all lines commented out. In practice, end users will almost always want to provide their own mosquitto.conf. For instance, they will probably want to require a password and access control (ACL).

Many real-world uses on target will probably want a multi-container approach, putting a reverse proxy like nginx in front of mosquitto or running with docker-compose. """

Quoted 3 lines

Bruce

[1] https://lore.kernel.org/yocto-meta-virtualization/178309247930.29524.9527654462555578401.b4-reply@b4/