Implement 'containers' jobs
v2 cover Implement 'containers' jobs
From: Tim Orling <[email protected]>
This series contains the 'yocto-autobuilder-helper' patches to enable:
* building, testing and publishing of 'vcontainer-tarball'
- the tarball is checked to see if a newer version exists and if so extracts the new version before testing
- vcontainer-tests job
* building and pushing containers from:
- meta-virtualization
* containers are pushed when the push_containers boolean is set in the AutoBuilder UI
- regstries are set with CONTAINER_REGISTRIES variable
* added getconfigdict() to scripts/utils to allow container names to mapped from recipe name to a more commonly used upstream container name (e.g. app-container-python -> python)
* authentication uses local user credentials (e.g. ~/.docker/config.json or .../auth.json for podman)
* containers are tagged with PV_MAJOR, PV_MAJOR.PV_MINOR, DISTRO_CODENAME and yocto-x.y release (yocto-x.y.z for release builds).
Additional features, such as container signing and attaching SLSA provenance attestations will come in a future series, once this initial functionality is agreed upon an stable.
Changes in v2:
* Simplify by merging vdkr-tests and vpdmn-tests into vcontainer-tests
* Simplify by building containers from meta-virtualization:
- Depends on resolution of: https://lists.yoctoproject.org/g/meta-virtualization/message/9826
* Workaround for recent (since Friday May 29, 2026) errors:
Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"
- This is probably related to either sstate changes or recent changes in vcontainer-common...
The following changes since commit b98a54a334d15179673207169cab00c1385bab2a:
config.json: Drop qemuarm testing from meta-virtualization (2026-05-31 18:18:44 +0100)
are available in the Git repository at:
https://git.yoctoproject.org/yocto-autobuilder-helper contrib/timo/containers-upstream-v2
https://git.yoctoproject.org/yocto-autobuilder-helper/log/?h=contrib/timo/containers-upstream-v2
for you to fetch changes up to aff1838ffcfb3e4c2aa1fe455dc0f74f87effdf6:
config.json: add 'containers-library' build job (2026-06-01 14:55:21 -0700)
----------------------------------------------------------------
Tim Orling (10):
scripts/utils: fix stale extraction dir when tarball is updated scripts/utils: warn and force re-download for HTTPS sources without
SHA256
config.json: add sha256sum for BUILDTOOLS URLs scripts/utils: add getconfigdict() for dict-type config values
scripts: add vcontainer-tarball setup, integration, and publishing config.json: add vcontainer-tarball build target
scripts: add run-vcontainer-tests for meta-virtualization
scripts: add container registry push, auth, tagging, runtime selection scripts/run-config: push_containers workaround config.json: add 'containers-library' build job
config.json | 129 ++++++++++++++++++++++++++- scripts/publish-artefacts | 5 ++ scripts/run-config | 150 ++++++++++++++++++++++++++++++++ scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++ scripts/shared-repo-unpack | 1 + scripts/utils.py | 109 +++++++++++++++++------ 6 files changed, 527 insertions(+), 31 deletions(-) create mode 100755 scripts/run-vcontainer-tests
Signature · 1 line
2.43.0
PATCH v2 1/10 scripts/utils: fix stale extraction dir when tarball is updated
From: Tim Orling <[email protected]>
Previously the entire download/cache-validation block in setup_tools_tarball() was guarded by if not os.path.exists(btdir). Once the extraction directory existed from a prior build, every subsequent call was a no-op: the cached tarball was never re-validated, so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently ignored and the stale btdir kept being used.
Fix by moving the lock/download block outside the btdir existence guard so cache validation always runs. Track tarball_updated to know when the cached download was actually replaced, then remove the stale btdir when True so the fresh tarball is re-extracted.
Also add an mtime-based staleness check for local-path (cp) sources so that a newer source file automatically invalidates the cached copy.
AI-Generated: Claude Cowork Sonnet 4.6
--- scripts/utils.py | 64 +++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 23 deletions(-)
scripts/utils.py +41 -23
@@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if ";" in bttarball: bttarball, sha256 = bttarball.split(";") btdir = os.path.abspath(btdir) + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) + btlock = btdlpath + ".lock" + if not os.path.exists(os.path.dirname(btdlpath)): + os.makedirs(os.path.dirname(btdlpath), exist_ok=True) + # Always run the cache-validation / download step so that a freshly + # published tarball is detected even when btdir already exists from a + # previous build. tarball_updated is set to True whenever the cached + # download is replaced, which triggers removal of the stale btdir. + tarball_updated = False + while True: + try: + with open(btlock, 'a+') as lf: + fileno = lf.fileno() + fcntl.flock(fileno, fcntl.LOCK_EX) + if sha256 and os.path.exists(btdlpath): + dl_sha256 = sha256_file(btdlpath) + if dl_sha256 != sha256: + os.unlink(btdlpath) + elif bttarball.startswith("/") and os.path.exists(btdlpath): + # For local-path sources (e.g. vcontainer-tarball-latest) + # invalidate the cached copy when the source is newer so + # that a freshly-published tarball is always picked up. + if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): + os.unlink(btdlpath) + os.unlink(btdlpath) + if not os.path.exists(btdlpath): + if bttarball.startswith("/"): + subprocess.check_call(["cp", bttarball, btdlpath]) + else: + subprocess.check_call(["wget", "-O", btdlpath, bttarball]) + os.chmod(btdlpath, 0o775) + tarball_updated = True + break + except OSError: + # We raced with someone else, try again + pass + # If the underlying tarball changed, remove any stale extraction + # directory so it is re-extracted below. + if tarball_updated and os.path.exists(btdir): + print("Removing stale %s extraction at %s" % (name, btdir)) + subprocess.check_call(["rm", "-rf", btdir]) if not os.path.exists(btdir): - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) print("Extracting %s %s" % (name, bttarball)) - btlock = btdlpath + ".lock" - if not os.path.exists(os.path.dirname(btdlpath)): - os.makedirs(os.path.dirname(btdlpath), exist_ok=True) - while True: - try: - with open(btlock, 'a+') as lf: - fileno = lf.fileno() - fcntl.flock(fileno, fcntl.LOCK_EX) - if sha256 and os.path.exists(btdlpath): - dl_sha256 = sha256_file(btdlpath) - if dl_sha256 != sha256: - os.unlink(btdlpath) - if not os.path.exists(btdlpath): - if bttarball.startswith("/"): - subprocess.check_call(["cp", bttarball, btdlpath]) - else: - subprocess.check_call(["wget", "-O", btdlpath, bttarball]) - os.chmod(btdlpath, 0o775) - break - except OSError: - # We raced with someone else, try again - pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_tools_tarball(btdir, name)
Signature · 1 line
2.43.0
PATCH v2 2/10 scripts/utils: warn and force re-download for HTTPS sources without SHA256
In reply to [email protected]From: Tim Orling <[email protected]>
Without a SHA256 checksum there is no way to verify that a cached HTTPS download is still current. Rather than silently reusing a potentially stale copy, delete the cached file and force a re-download each run, and emit a clear WARNING telling the operator how to avoid the overhead (by appending ;sha256=<hash> to the URL in their config).
AI-Generated: Claude Cowork Sonnet 4.6
--- scripts/utils.py | 8 ++++++++ 1 file changed, 8 insertions(+)
scripts/utils.py +8 -0
@@ -505,6 +505,14 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): # that a freshly-published tarball is always picked up. if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): os.unlink(btdlpath) + elif not bttarball.startswith("/") and os.path.exists(btdlpath): + # HTTPS/FTP source with no SHA256: there is no way to + # verify the cached copy is current, so force a + # re-download every run. Add a sha256=<hash> suffix to + # the URL in your config to avoid this. + print("WARNING: no SHA256 provided for %s source %s; " + "forcing re-download to avoid using a stale cached copy" + % (name, bttarball)) os.unlink(btdlpath) if not os.path.exists(btdlpath): if bttarball.startswith("/"):
Signature · 1 line
2.43.0
PATCH v2 3/10 config.json: add sha256sum for BUILDTOOLS URLs
In reply to [email protected]From: Tim Orling <[email protected]>
setup_tools_tarball() now checks the sha256sum of cached tools tarballs to determine if an update is available and needs to be re-downloaded.
To prevent unnecessary downloads, add ;sha256= to the BUILDTOOLS URLs.
extratools/EXTRATOOLS_URL uses the same utils.setup_tools_tarball(), so it also needs the ;sha256=.
--- config.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
config.json +4 -4
@@ -7,10 +7,10 @@ "BUILD_HISTORY_REPO" : "ssh://[email protected]/poky-buildhistory", "BUILD_HISTORY_FORKPUSH" : {"openembedded-core-contrib:ross/mut" : "openembedded-core:master", "openembedded-core-contrib:abelloni/master-next": "openembedded-core:master", "openembedded-core:master-next" : "openembedded-core:master"}, - "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh", - "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh", - "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh", + "BUILDTOOLS_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/x86_64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=5af9d92898af17fcc2fca4d07607a59f41b1c39a0a4ff058f64d834b98ec7fd5", + "BUILDTOOLS_ARM_URL" : "https://downloads.yoctoproject.org/releases/yocto/milestones/yocto-5.2_M2/buildtools/aarch64-buildtools-extended-nativesdk-standalone-5.1.sh;sha256=8074d582a60a5135fd5606b8326e749185d26f51aca27570447831310bcac187", + "BUILDTOOLS_MAKE_URL" : "https://downloads.yoctoproject.org/releases/yocto/yocto-5.0.4/buildtools/x86_64-buildtools-make-nativesdk-standalone-5.0.4.sh;sha256=2eb7a6c013113f4fdb87e800167606dc13af1bd8bbc1c9e2443b7be37fefd124", + "EXTRATOOLS_URL" : "https://downloads.yoctoproject.org/tools/buildtools/x86_64-buildtools-imagemagick-nativesdk-standalone-4.3+snapshot-5f2ba20f203114db9a3b11264467f8c23a05041d.sh;sha256=9cbff3a7cf524bdfa7779dce8afaf3453114d8017918d2927f723ea38a36ebdc", "REPO_STASH_DIR" : "${BASE_HOMEDIR}/git/mirror", "TRASH_DIR" : "${BASE_HOMEDIR}/git/trash",
Signature · 1 line
2.43.0
PATCH v2 4/10 scripts/utils: add getconfigdict() for dict-type config values
In reply to [email protected]From: Tim Orling <[email protected]>
Parallel to getconfiglist(), but for JSON object values. The merge priority is defaults < target-level < step-level so that more-specific keys win: a step can override individual entries in a target-level dict without replacing the whole thing, and both levels refine the defaults.
Used by the upcoming CONTAINER_IMAGES support, where each entry maps a Yocto recipe name (the on-disk OCI path stem) to an image name (the name pushed to the container registry).
AI-Generated: Claude Cowork Sonnet 4.6
--- scripts/utils.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
scripts/utils.py +15 -0
@@ -112,6 +112,21 @@ def getconfiglist(name, config, target, stepnum): ret.extend(config['defaults'][name]) return expandresult(ret, config) +# Get a build configuration dict, merging defaults < target < step so that +# more-specific entries win (step-level keys override target-level, which +# override defaults). +def getconfigdict(name, config, target, stepnum): + ret = {} + step = "step" + str(stepnum) + if name in config['defaults']: + ret.update(config['defaults'][name]) + if target in config['overrides']: + if name in config['overrides'][target]: + ret.update(config['overrides'][target][name]) + if step in config['overrides'][target] and name in config['overrides'][target][step]: + ret.update(config['overrides'][target][step][name]) + return expandresult(ret, config) + # Return only unique configuration values (identified with '=' in them) def getconfiglistfilter(name, config, target, stepnum): def merge(main, newvals):
Signature · 1 line
2.43.0
PATCH v2 5/10 scripts: add vcontainer-tarball setup, integration, and publishing
In reply to [email protected]From: Tim Orling <[email protected]>
Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step.
* scripts/utils.py: add setup_vcontainer_tarball(), and add an env_glob keyword argument to setup_tools_tarball() and enable_tools_tarball() so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob.
* scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS.
* scripts/shared-repo-unpack: invoke setup_vcontainer_tarball so workers extract the SDK during unpack.
* scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK.
AI-Generated: Claude Cowork Opus 4.7
--- config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++ scripts/shared-repo-unpack | 1 + scripts/utils.py | 22 ++++++++++++++++++---- 5 files changed, 44 insertions(+), 4 deletions(-)
config.json +1 -0
@@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true, "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"] } },
scripts/publish-artefacts +5 -0
@@ -146,5 +146,10 @@ case "$target" in sha256sums $TMPDIR/deploy/images/qemux86-64 cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;; + "vcontainer-tarball") + mkdir -p $DEST/vcontainer-tarball + sha256sums $TMPDIR/deploy/sdk + cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;; esac
scripts/run-config +19 -0
@@ -153,6 +153,25 @@ else: if args.phase == "init" and args.stepname == "buildtools": sys.exit(0) +if jcfg: + vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True) + if vcontainer: + addentry("vcontainer", "Setup vcontainer tarball", "init") +else: + # If we're executing a specific step, check whether vcontainer is disabled for it + vcontainer = True + if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"): + try: + vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase)) + except ValueError: + # Not an integer step phase + pass + + if vcontainer: + utils.setup_vcontainer_tarball(ourconfig, args.workername, args.builddir + "/../vcontainer-tarball") + if args.phase == "init" and args.stepname == "vcontainer": + sys.exit(0) + extratools = utils.getconfigvar("extratools", ourconfig, args.target) if jcfg: if extratools:
scripts/shared-repo-unpack +1 -0
@@ -77,6 +77,7 @@ for repo in sorted(repos.keys()): utils.flush() utils.setup_buildtools_tarball(ourconfig, args.workername, args.abworkdir + "/buildtools") +utils.setup_vcontainer_tarball(ourconfig, args.workername, args.abworkdir + "/vcontainer-tarball") if "bitbake" not in repos: sys.exit(0)
scripts/utils.py +18 -4
@@ -456,8 +456,8 @@ def sha256_file(filename): pass return method.hexdigest() -def enable_tools_tarball(btdir, name): - btenv = glob.glob(btdir + "/environment-setup*") +def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"): + btenv = glob.glob(btdir + env_glob) print("Using %s %s" % (name, btenv)) # We either parse or wrap all our execution calls, rock and a hard place :( with open(btenv[0], "r") as f:
@@ -474,6 +474,20 @@ def enable_tools_tarball(btdir, name): if line in os.environ: del os.environ[line] +def setup_vcontainer_tarball(ourconfig, workername, vcdir, checkonly=False): + vctarball = None + if "vcontainer" in ourconfig and workername: + vccfg = getconfig("vcontainer", ourconfig) + for entry in vccfg: + if fnmatch.fnmatch(workername, entry): + vctarball = vccfg[entry] + break + + if checkonly: + return vctarball + + setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci") + def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): bttarball = None if "buildtools" in ourconfig and workername:
@@ -488,7 +502,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False): setup_tools_tarball(ourconfig, btdir, bttarball) -def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): +def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"): btenv = None if bttarball:
@@ -548,7 +562,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if not os.path.exists(btdir): print("Extracting %s %s" % (name, bttarball)) subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) - enable_tools_tarball(btdir, name) + enable_tools_tarball(btdir, name, env_glob) def get_string_from_version(version, milestone=None, rc=None): """ Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
Signature · 1 line
2.43.0
PATCH v2 6/10 config.json: add vcontainer-tarball build target
In reply to [email protected]From: Tim Orling <[email protected]>
Add the vcontainer-tarball build target which produces the meta-virtualization SDK used by downstream container build/test jobs. Modelled after the buildtools-tarball target.
The target uses EXTRACMDS to reset BBMULTICONFIG for the worker shell, places the extravars in the per-step config, and includes a publish-artefacts step so the resulting SDK tarball is staged for reuse by container-tests and other consumers.
Since meta-virtualization is the first layer removed by 'remove-layers', our BBMULTICONFIGs become invalid immediately after layer removal and throw an error.
In scripts/run-config, EXTRACMDS runs after BBTARGETS, but before 'remove-layers'. Use 'sed' to reset BBMULTICONFIG to ''.
The error in the 'remove-layers' step can be summarized as:
bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf
--- config.json | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
config.json +26 -0
@@ -1864,6 +1864,32 @@ }, "toaster" : { "EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"] + }, + "vcontainer-tarball": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Build vcontainer-tarballs", + "BBTARGETS" : "vcontainer-tarball", + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'", + "BBMULTICONFIG = 'vruntime-aarch64 vruntime-x86-64'", + "INIT_MANAGER = 'systemd'" + ], + "EXTRACMDS" : ["sed -i '/vruntime-aarch64 vruntime-x86-64/d' ${HELPERBUILDDIR}/conf/auto.conf"] + }, + "step2" : { + "shortname" : "Publish vcontainer SDK for test reuse", + "EXTRACMDS" : [ + "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" + ] + } } }, "repo-defaults" : {
Signature · 1 line
2.43.0
PATCH v2 7/10 scripts: add run-vcontainer-tests for meta-virtualization
In reply to [email protected]From: Tim Orling <[email protected]>
Add scripts/run-vcontainer-tests, the test runner used by the vcontainer test jobs. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests:
- tests/test_container_cross_install.py
- tests/test_container_registry_script.py
- tests/test_vcontainer_auth_config.py
- tests/test_multiarch_oci.py
- tests/test_multilayer_oci.py
The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine):
- tests/test_vdkr.py
- tests/test_vdkr_registry.py
and
- tests/test_vpdmn.py
AI-Generated: Claude Cowork Opus 4.7
--- config.json | 34 ++++++++ scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+) create mode 100755 scripts/run-vcontainer-tests
config.json +34 -0
@@ -1890,6 +1890,40 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } } }, "repo-defaults" : {
scripts/run-vcontainer-tests new file +164 -0
164 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v2 8/10 scripts: add container registry push, auth, tagging, runtime selection
In reply to [email protected]From: Tim Orling <[email protected]>
Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags.
* config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG, CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE and CONTAINER_IMAGE_MAP configuration knobs.
* scripts/run-config: drive push-containers as a post-step action. Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to source PV from a different recipe than the image itself.
* Registry auth is staged via .../config.json or podman .../auth.json using CONTAINER_AUTH_CONFIG, replacing an interactive login that could hang. CONTAINER_RUNTIME picks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
* Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up.
AI-Generated: Claude Cowork Opus 4.7
--- config.json | 5 ++ scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+)
config.json +5 -0
@@ -43,6 +43,11 @@ "BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"], "WRITECONFIG" : true, "SENDERRORS" : true, + "CONTAINER_RUNTIME" : "vpdmn", + "CONTAINER_REGISTRIES" : [], + "CONTAINER_TAGS" : ["latest"], + "CONTAINER_TAG_CMDS" : [], + "CONTAINER_IMAGE_MAP" : {}, "extravars" : [ "SANITY_TESTED_DISTROS = ''", "BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'",
scripts/run-config +129 -0
141 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v2 9/10 scripts/run-config: push_containers workaround
In reply to [email protected]From: Tim Orling <[email protected]>
Workaround for:
Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"
The first time the blob (e.g. bin/bash.bash) is copied, it succeeds. All subsequent tries fail. Might possibly be an issue with sstate or a recent change to vcontainer-common to allow for multiarch containers.
--- scripts/run-config | 2 ++ 1 file changed, 2 insertions(+)
scripts/run-config +2 -0
@@ -381,6 +381,8 @@ def handle_stepnum(stepnum): # password from stdin (see login case in vcontainer-common.sh). "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime, "%s-$(arch) --config %s memres restart </dev/null" % (runtime, auth_config), + # workaround for 'Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"' + "%s-$(arch) image rm --all 2>/dev/null" % (runtime), ] tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, stepnum) version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, stepnum)
Signature · 1 line
2.43.0
PATCH v2 10/10 config.json: add 'containers-library' build job
In reply to [email protected]From: Tim Orling <[email protected]>
Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK:
* original container images from the intial "container-cross-install" branch of meta-virtualization
- container-base
- app-container-curl
* additional images modelled after docker.io/library/*
- app-container-python
- app-container-mosquitto
- app-container-valkey
- app-container-nginx
* Tag containers with versions based on the recipe to which they are
mapped, e.g. python:3, python:3.14, python:3.14.5.
--- config.json | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
config.json +55 -0
@@ -1929,6 +1929,61 @@ "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" ] } + }, + "containers-library": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-openembedded/meta-webserver", + "${BUILDDIR}/../meta-virtualization" + ], + "extravars" : [ + "DISTRO_FEATURES:append = ' virtualization vcontainer'" + ], + "CONTAINER_TAG_CMDS" : [ + "_PV_MAJOR=$(echo $_PV | cut -d. -f1)", + "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)", + "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\"" + ], + "step1" : { + "shortname" : "Build 'base' container", + "BBTARGETS" : "container-base", + "CONTAINER_IMAGE_MAP" : {"container-base": "base"}, + "CONTAINER_VERSION_RECIPE" : "base-files" + }, + "step2" : { + "shortname" : "Build 'curl' container", + "BBTARGETS" : "app-container-curl", + "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"}, + "CONTAINER_VERSION_RECIPE" : "curl" + }, + "step3" : { + "shortname" : "Build 'python' container", + "BBTARGETS" : "app-container-python", + "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"}, + "CONTAINER_VERSION_RECIPE" : "python3" + }, + "step4" : { + "shortname" : "Build 'mosquitto' container", + "BBTARGETS" : "app-container-mosquitto", + "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"}, + "CONTAINER_VERSION_RECIPE" : "mosquitto" + }, + "step5" : { + "shortname" : "Build 'valkey' container", + "BBTARGETS" : "app-container-valkey", + "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"}, + "CONTAINER_VERSION_RECIPE" : "valkey" + }, + "step6" : { + "shortname" : "Build 'nginx' container", + "BBTARGETS" : "app-container-nginx", + "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"}, + "CONTAINER_VERSION_RECIPE" : "nginx" + } } }, "repo-defaults" : {
Signature · 1 line
2.43.0
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated
In reply to [email protected]Quoted 57 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Previously the entire download/cache-validation block in
setup_tools_tarball()was guarded byif not os.path.exists(btdir). Once the extraction directory existed from a prior build, every subsequent call was a no-op: the cached tarball was never re-validated, so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently ignored and the stale btdir kept being used.Fix by moving the lock/download block outside the btdir existence guard so cache validation always runs. Track
tarball_updatedto know when the cached download was actually replaced, then remove the stale btdir when True so the fresh tarball is re-extracted.Also add an mtime-based staleness check for local-path (cp) sources so that a newer source file automatically invalidates the cached copy.
AI-Generated: Claude Cowork Sonnet 4.6
Signed-off-by Tim Orling <[email protected]>--- scripts/utils.py | 64 +++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 23 deletions(-)scripts/utils.py +26
-0@@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): if ";" in bttarball: bttarball, sha256 = bttarball.split(";") btdir = os.path.abspath(btdir) + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) + btlock = btdlpath + ".lock" + if not os.path.exists(os.path.dirname(btdlpath)): + os.makedirs(os.path.dirname(btdlpath), exist_ok=True) + # Always run the cache-validation / download step so that a freshly + # published tarball is detected even when btdir already exists from a + # previous build. tarball_updated is set to True whenever the cached + # download is replaced, which triggers removal of the stale btdir. + tarball_updated = False + while True: + try: + with open(btlock, 'a+') as lf: + fileno = lf.fileno() + fcntl.flock(fileno, fcntl.LOCK_EX) + if sha256 and os.path.exists(btdlpath): + dl_sha256 = sha256_file(btdlpath) + if dl_sha256 != sha256: + os.unlink(btdlpath) + elif bttarball.startswith("/") and os.path.exists(btdlpath): + # For local-path sources (e.g. vcontainer-tarball-latest) + # invalidate the cached copy when the source is newer so + # that a freshly-published tarball is always picked up. + if os.path.getmtime(bttarball) > os.path.getmtime(btdlpath): + os.unlink(btdlpath) + os.unlink(btdlpath) + if not os.path.exists(btdlpath):
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): · line 519
Quoted 10 lines
scripts/utils.py +10
-0+ if bttarball.startswith("/"): + subprocess.check_call(["cp", bttarball, btdlpath]) + else: + subprocess.check_call(["wget", "-O", btdlpath, bttarball]) + os.chmod(btdlpath, 0o775) + tarball_updated = True + break + except OSError: + # We raced with someone else, try again + pass
Hi Tim,
This loop allows us to handle random OSError conditions caused by a race, but it turns a deterministic OSError into an infinite loop. The logic already exists, you're just moving it around, but now I have seen it I cannot unsee it!
Should we add a timeout while we are here?
Quoted 22 lines
scripts/utils.py +5
-15+ # If the underlying tarball changed, remove any stale extraction + # directory so it is re-extracted below. + if tarball_updated and os.path.exists(btdir): + print("Removing stale %s extraction at %s" % (name, btdir)) + subprocess.check_call(["rm", "-rf", btdir]) if not os.path.exists(btdir): - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) + "/cluster-downloads-cache/" + os.path.basename(bttarball) print("Extracting %s %s" % (name, bttarball)) - btlock = btdlpath + ".lock" - if not os.path.exists(os.path.dirname(btdlpath)): - os.makedirs(os.path.dirname(btdlpath), exist_ok=True) - while True: - try: - with open(btlock, 'a+') as lf: - fileno = lf.fileno() - fcntl.flock(fileno, fcntl.LOCK_EX) - if sha256 and os.path.exists(btdlpath): - dl_sha256 = sha256_file(btdlpath) - if dl_sha256 != sha256: - os.unlink(btdlpath) - if not os.path.exists(btdlpath): - if bttarball.startswith("/"):
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): · line 528
Quoted 10 lines
scripts/utils.py +0
-8- subprocess.check_call(["cp", bttarball, btdlpath]) - else: - subprocess.check_call(["wget", "-O", btdlpath, bttarball]) - os.chmod(btdlpath, 0o775) - break - except OSError: - # We raced with someone else, try again - pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_tools_tarball(btdir, name)
Best regards,
Signature · 1 line
Paul Barker
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization
In reply to [email protected]Quoted 95 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add
scripts/run-vcontainer-tests, the test runner used by the vcontainer test jobs. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests:-
tests/test_container_cross_install.py
-tests/test_container_registry_script.py
-tests/test_vcontainer_auth_config.py
-tests/test_multiarch_oci.py
-tests/test_multilayer_oci.pyThe 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine):
-
tests/test_vdkr.py
-tests/test_vdkr_registry.pyand
-
tests/test_vpdmn.pyAI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>--- config.json | 34 ++++++++ scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+) create mode 100755 scripts/run-vcontainer-testsconfig.json +34
-0@@ -1890,6 +1890,40 @@ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh" ] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"], + "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ + "VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization" + ] + } } }, "repo-defaults" : {scripts/run-vcontainer-tests new file +10
-0@@ -0,0 +1,164 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainer +# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake +# step. +# +# Arguments: +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn"
scripts/run-vcontainer-tests line 20
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+# $2 - bitbake build directory (${BUILDDIR}) +# $3 - path to the meta-virtualization layer +# +# Optional environment variables: +# RESULTS_DIR - directory to copy pytest artefacts (junit xml / log) to +# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball +# (default: ${builddir}/vcontainer-test-extracted) +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# VDKR_ARCH - target architecture for vdkr/vpdmn tests (default: x86_64)
Taking some arguments on the command line and others via environment variables is confusing. I wonder if should use getopts to parse arguments.
Quoted 55 lines
scripts/run-vcontainer-tests new file +55
-0+# +# The script is intentionally conservative: any pytest tests that cannot run +# in the CI environment (those marked "slow", "network", "boot") are skipped +# so that the autobuilder step completes without needing network access. Those +# can be re-enabled by exporting META_VIRT_PYTEST_MARKERS before invocation. +# +# It is assumed that /dev/kvm is writable by the CI user running the tests, +# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +if [ $# -lt 3 ]; then + echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2 + echo " suite: vcontainer | vdkr | vpdmn" >&2 + exit 2 +fi + +suite="$1" +builddir=$(realpath "$2") +metavirtdir=$(realpath "$3") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2 + exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer an externally-built +# SDK passed via VCONTAINER_SDK (the autobuilder -tests jobs share the SDK +# produced by the separate vcontainer-tarball builder), and fall back to +# looking in the local build's deploy/sdk directory when running stand-alone. +sdk_tarball="" +if [ -n "${VCONTAINER_SDK:-}" ]; then + if [ -f "$VCONTAINER_SDK" ]; then + sdk_tarball="$VCONTAINER_SDK" + else + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not a file" >&2 + exit 1 + fi +fi +if [ -z "$sdk_tarball" ]; then + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g. versioned) + alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true) + if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Set VCONTAINER_SDK to an existing SDK installer, or" >&2 + echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2 + exit 1
scripts/run-vcontainer-tests line 85
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+ fi + fi +fi + +extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}" +rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -y
Patch 5 in this series added support for setting up the vcontainer tarball for a step. I think the commit message in this patch needs to explain why we can't use that in the steps that call run-vcontainer-tests.
scripts/run-vcontainer-tests line 90
Quoted 5 lines
scripts/run-vcontainer-tests new file +5
-0+ +# Prepare a Python venv so we don't pollute the worker's system packages. +python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate"
Note that this will shadow the venv we use when running test jobs on the autobuilder workers. That may be ok, but we need to be careful.
scripts/run-vcontainer-tests line 93
Quoted 3 lines
scripts/run-vcontainer-tests new file +3
-0+# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip. +python3 -m pip install --quiet --upgrade pip setuptools wheel +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpect
Do we want to automatically install the latest versions of these tools, via the network, in each build? It's better to pin versions to ensure that upgrades are planned rather than automagical.
Also, should we store this list of dependencies in a requirements.txt file somewhere? Perhaps tests/requirements.txt in meta-virtualization.
scripts/run-vcontainer-tests line 96
Quoted 3 lines
scripts/run-vcontainer-tests new file +3
-0+ +# Default marker filter excludes long running / infrastructure dependent tests. +marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and not boot and not incus and not k3s}"
Would it be better to have a positive marker, e.g. 'autobuilder', to mark tests that we do want to run? That would allow us to maintain the filtering solely in meta-virtualization instead of needing updates here as well when things change.
Quoted 68 lines
scripts/run-vcontainer-tests new file +68
-0+ +# Per-suite test file selection. Uses -k/-m for fine-grained filtering and +# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py" + "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn't require the + # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2 + exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter" + --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then + pytest_args+=(--oci-image "$TEST_OCI_IMAGE") +fi + +# Pass architecture through when set in the environment (default is x86_64). +if [ -n "${VDKR_ARCH:-}" ]; then + pytest_args+=(--arch "$VDKR_ARCH") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect the junit +# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "${RESULTS_DIR:-}" ]; then + mkdir -p "$RESULTS_DIR" + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/" 2>/dev/null || true + if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log "$RESULTS_DIR/pytest-$suite.log" || true + fi +fi + +exit $rc
Signature · 1 line
Paul Barker
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job
In reply to [email protected]Quoted 15 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK:
* original container images from the intial "container-cross-install" branch of meta-virtualization
- container-base
- app-container-curl
* additional images modelled after docker.io/library/*
- app-container-python
- app-container-mosquitto
- app-container-valkey
- app-container-nginx
* Tag containers with versions based on the recipe to which they are
Quoted 1 line
mapped, e.g. python:3, python:3.14, python:3.14.5.
These version tags may exist in docker library/hardened images, but are they actually appropriate for the images we create?
I would expect python:master, python:blacksail, etc tags which track our branches.
Best regards,
Signature · 1 line
Paul Barker
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 10/10] config.json: add 'containers-library' build job
Quoted 19 lines · Paul Barker via lists.yoctoproject.org
Quoted 15 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add 'containers-library' build job that build container images on top of the vcontainer-tarball SDK:
* original container images from the intial "container-cross-install" branch of meta-virtualization
- container-base
- app-container-curl
* additional images modelled after docker.io/library/*
- app-container-python
- app-container-mosquitto
- app-container-valkey
- app-container-nginx
* Tag containers with versions based on the recipe to which they areQuoted 1 line
mapped, e.g. python:3, python:3.14, python:3.14.5.
Quoted 3 lines
These version tags may exist in docker library/hardened images, but are they actually appropriate for the images we create?
They are tagged with 'wrynose' and 'yocto-6.0' (and release builds would be tagged with full 'yocto-6.0.n').
We need to think about container consumers, not just Yocto users. Container consumers want to know
what version of the tool is in the container they are consuming, so we should follow the same tagging
pattern. Tags are cheap (they are just an alias for the same hash).
Quoted 24 lines
I would expect python:master, python:blacksail, etc tags which track our branches.
Best regards,
--
Paul Barker-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#4112):
https://lists.yoctoproject.org/g/yocto-patches/message/4112
Mute This Topic: https://lists.yoctoproject.org/mt/119603250/924729
Group Owner: [email protected] Unsubscribe:
https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
In reply to [email protected]Quoted 70 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step.
*
scripts/utils.py: addsetup_vcontainer_tarball(), and add anenv_globkeyword argument tosetup_tools_tarball()andenable_tools_tarball()so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob.
*scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS.
*scripts/shared-repo-unpack: invokesetup_vcontainer_tarballso workers extract the SDK during unpack.
*scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK.AI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>---
config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++scripts/shared-repo-unpack| 1 + scripts/utils.py | 22 ++++++++++++++++++----5 files changed, 44 insertions(+), 4 deletions(-)config.json +1
-0@@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true,"EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"] }
},scripts/publish-artefacts +0
-0@@ -146,5 +146,10 @@ case "$target" insha256sums $TMPDIR/deploy/images/qemux86-64
cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest ;;
+ "vcontainer-tarball")
+ mkdir -p $DEST/vcontainer-tarball
+ sha256sums $TMPDIR/deploy/sdk
+ cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball + ;;
esacscripts/run-config +0
-0@@ -153,6 +153,25 @@ else:if args.phase == "init" and args.stepname == "buildtools":
sys.exit(0)scripts/run-config +3
-0+if jcfg: + vcontainer = utils.setup_vcontainer_tarball(ourconfig, args.workername, None, checkonly=True) + if vcontainer:
scripts/run-config else: · line 168
Quoted 10 lines
scripts/run-config +10
-0+ addentry("vcontainer", "Setup vcontainer tarball", "init") +else: + # If we're executing a specific step, check whether vcontainer is disabled for it + vcontainer = True + if args.stepname in ("build-targets", "cmds", "test-targets", "plain-cmds"): + try: + vcontainer = not utils.getconfigvar("NOVCONTAINER", ourconfig, args.target, int(args.phase)) + except ValueError: + # Not an integer step phase + pass
I suspect this logic also needs to be in the if jcfg block above, else it will add the task to all jobs on the autobuilder, then just do nothing in the task. If there isn't anything to do, we may as well just not add it all?
Cheers,
Richard
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection
In reply to [email protected]Quoted 24 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags.
* config.json: add
CONTAINER_REGISTRIES,CONTAINER_AUTH_CONFIG,CONTAINER_RUNTIME,CONTAINER_TAG_CMDS,CONTAINER_VERSION_RECIPEandCONTAINER_IMAGE_MAPconfiguration knobs.
*scripts/run-config: drive push-containers as a post-step action. Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) withCONTAINER_VERSION_RECIPEallowing a step to source PV from a different recipe than the image itself.
* Registry auth is staged via.../config.jsonor podman.../auth.jsonusingCONTAINER_AUTH_CONFIG, replacing an interactive login that could hang.CONTAINER_RUNTIMEpicks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
* Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up.
Quoted 6 lines
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>---
config.json | 5 ++
scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++2 files changed, 134 insertions(+)
This is adding way too much code for a specific use case to run-config. This needs to be in a separate script (in the same way docs builds or other things are).
Cheers,
Richard
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
In reply to [email protected]Quoted 41 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step.
*
scripts/utils.py: addsetup_vcontainer_tarball(), and add anenv_globkeyword argument tosetup_tools_tarball()andenable_tools_tarball()so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob.
*scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS.
*scripts/shared-repo-unpack: invokesetup_vcontainer_tarballso workers extract the SDK during unpack.
*scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK.AI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>---
config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++scripts/shared-repo-unpack| 1 + scripts/utils.py | 22 ++++++++++++++++++----5 files changed, 44 insertions(+), 4 deletions(-)config.json +1
-0@@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true,
Quoted 3 lines
"EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index ${HELPERBUILDDIR}/../"] }
},
Why is this step needing to opt out of a vcontainer? vcontainers are only enabled for specific steps and off by default?
I looked further and realised you're basically using a vcontainer everywhere as far as I can tell? Why do we want to have the overhead of doing that outside of the vcontainer builds?
For buildtools, it makes sense we need it everywhere except where we configure otherwise but I think for vcontainer, you want to opposite, you only use it in jobs where we actually need/use it?
Cheers,
Richard
PATCH 1/1 scripts/utils.py: add timeout check for tarball extraction
In reply to Paul BarkerThe while True, try, except OSError: pass loop has a risk of getting caught in an infinite loop. Add an overall timeout to ensure we raise the underlying OSError after we have tried for 300 seconds/5 minutes.
--- scripts/utils.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/utils.py +5 -1
@@ -506,6 +506,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): # previous build. tarball_updated is set to True whenever the cached # download is replaced, which triggers removal of the stale btdir. tarball_updated = False + tarball_timeout = time.monotonic() + 300 # 5 min, tune as needed while True: try: with open(btlock, 'a+') as lf:
@@ -548,7 +549,10 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): break except OSError: # We raced with someone else, try again - pass + if time.monotonic() > tarball_timeout: + raise + time.sleep(1) + # If the underlying tarball changed, remove any stale extraction # directory so it is re-extracted below. if tarball_updated and os.path.exists(btdir):
Signature · 1 line
2.54.0
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 07/10] scripts: add run-vcontainer-tests for meta-virtualization
In reply to Paul BarkerQuoted 127 lines · Paul Barker via lists.yoctoproject.org
Quoted 39 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add
scripts/run-vcontainer-tests, the test runner used by the vcontainer test jobs. It sources the vcontainer-tarball SDK, discovers the meta-virtualization pytest suite, and runs a configurable set of suites (vdkr, vpdmn, memres) against the checked-out layers. Suites can be selected per-step so the top-level 'vcontainer-tests' job runs the container engine agnostic tests:-
tests/test_container_cross_install.py
-tests/test_container_registry_script.py
-tests/test_vcontainer_auth_config.py
-tests/test_multiarch_oci.py
-tests/test_multilayer_oci.pyThe 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective suites (including memres for each container engine):
-
tests/test_vdkr.py
-tests/test_vdkr_registry.pyand
-
tests/test_vpdmn.pyAI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>--- config.json | 34 ++++++++ scripts/run-vcontainer-tests | 164 +++++++++++++++++++++++++++++++++++ 2 files changed, 198 insertions(+) create mode 100755 scripts/run-vcontainer-testsconfig.json +0
-0config.json line 1890
Quoted 1 line
config.json +0
-0@@ -1890,6 +1890,40 @@ "install -d${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755
${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/ vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/ vcontainer-standalone.sh.new${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"config.json
Quoted 5 lines
config.json +3
-0] } + }, + "vcontainer-tests": { + "NEEDREPOS" : ["bitbake", "meta-openembedded","meta-virtualization"],
config.json
Quoted 13 lines
config.json +13
-0+ "ADDLAYER" : [ + "${BUILDDIR}/../meta-openembedded/meta-oe", + "${BUILDDIR}/../meta-openembedded/meta-python", + "${BUILDDIR}/../meta-openembedded/meta-networking", + "${BUILDDIR}/../meta-openembedded/meta-filesystems", + "${BUILDDIR}/../meta-virtualization" + ], + "step1" : { + "shortname" : "Run vcontainer pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ +"VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vcontainer ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
config.json
Quoted 8 lines
config.json +8
-0+ ] + }, + "step2" : { + "shortname" : "Run vdkr pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ +"VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vdkr ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
config.json
Quoted 8 lines
config.json +8
-0+ ] + }, + "step3" : { + "shortname" : "Run vpdmn pytest suite", + "NOBUILDTOOLS" : 1, + "NOVCONTAINER" : 1, + "EXTRACMDS" : [ +"VCONTAINER_SDK=${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh RESULTS_DIR=${HELPERRESULTSDIR} ${SCRIPTSDIR}/run-vcontainer-tests vpdmn ${BUILDDIR} ${BUILDDIR}/../meta-virtualization"
Quoted 16 lines
config.json +2
-0+ ] + } } }, "repo-defaults" : {scripts/run-vcontainer-tests new file +5
-0@@ -0,0 +1,164 @@ +#!/bin/bash +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Run meta-virtualization pytest test suites against the vcontainerscripts/run-vcontainer-tests line 15
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake +# step. +# +# Arguments: +# $1 - suite name: one of "vcontainer", "vdkr", "vpdmn" +# $2 - bitbake build directory (${BUILDDIR}) +# $3 - path to the meta-virtualization layer +# +# Optional environment variables: +# RESULTS_DIR - directory to copy pytest artefacts (junit xml /log) to
scripts/run-vcontainer-tests
Quoted 5 lines
scripts/run-vcontainer-tests new file +5
-0+# VCONTAINER_EXTRACT_DIR - where to extract the standalone SDK tarball +# (default: ${builddir}/vcontainer-test-extracted) +# TEST_OCI_IMAGE - path to an OCI image directory (enables vdkr/vpdmn +# import tests) +# VDKR_ARCH - target architecture for vdkr/vpdmn tests(default:
x86_64)
Quoted 3 lines
Taking some arguments on the command line and others via environment variables is confusing. I wonder if should use getopts to parse arguments.
This is addressed in v3.
Quoted 92 lines
scripts/run-vcontainer-tests line 22
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+# +# The script is intentionally conservative: any pytest tests thatcannot run
Quoted 1 line
scripts/run-vcontainer-tests new file +1
-0+# in the CI environment (those marked "slow", "network", "boot") areskipped
Quoted 1 line
+# so that the autobuilder step completes without needing network
access. Those
Quoted 1 line
+# can be re-enabled by exporting
META_VIRT_PYTEST_MARKERSbeforeinvocation.
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+# +# It is assumed that /dev/kvm is writable by the CI user running thetests,
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+# since the performance is significantly faster with 'memres'. +# + +set -e +set -u +set -o pipefail +set -x + +if [ $# -lt 3 ]; then + echo "Usage: $0 <suite> <builddir> <meta-virtualization-dir>" >&2scripts/run-vcontainer-tests
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+ echo " suite: vcontainer | vdkr | vpdmn" >&2 + exit 2 +fi + +suite="$1" +builddir=$(realpath "$2") +metavirtdir=$(realpath "$3") + +if [ ! -d "$metavirtdir/tests" ]; then + echo "ERROR: meta-virtualization tests directory not found at$metavirtdir/tests" >&2
scripts/run-vcontainer-tests
Quoted 4 lines
scripts/run-vcontainer-tests new file +4
-0+ exit 1 +fi + +# Locate the vcontainer standalone SDK tarball. Prefer anexternally-built
Quoted 1 line
+# SDK passed via
VCONTAINER_SDK(the autobuilder -tests jobs share theSDK
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+# produced by the separate vcontainer-tarball builder), and fall back to +# looking in the local build's deploy/sdk directory when runningstand-alone.
scripts/run-vcontainer-tests
Quoted 6 lines
scripts/run-vcontainer-tests new file +6
-0+sdk_tarball="" +if [ -n "${VCONTAINER_SDK:-}" ]; then + if [ -f "$VCONTAINER_SDK" ]; then + sdk_tarball="$VCONTAINER_SDK" + else + echo "ERROR: VCONTAINER_SDK=$VCONTAINER_SDK is set but not afile" >&2
scripts/run-vcontainer-tests
Quoted 7 lines
scripts/run-vcontainer-tests new file +7
-0+ exit 1 + fi +fi +if [ -z "$sdk_tarball" ]; then + sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh" + if [ ! -f "$sdk_tarball" ]; then + # Try to find any matching tarball in case naming changed (e.g.versioned)
Quoted 1 line
+ alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh
2>/dev/null | head -n1 || true)
scripts/run-vcontainer-tests
Quoted 5 lines
scripts/run-vcontainer-tests new file +5
-0+ if [ -n "$alt" ]; then + sdk_tarball="$alt" + else + echo "ERROR: vcontainer standalone SDK not found." >&2 + echo " Set VCONTAINER_SDK to an existing SDKinstaller, or" >&2
Quoted 1 line
+ echo " build vcontainer-tarball so
$builddir/tmp/deploy/sdk/vcontainer-standalone.shexists." >&2scripts/run-vcontainer-tests line 79
Quoted 6 lines
scripts/run-vcontainer-tests new file +5
-0+ exit 1 + fi + fi +fi ++extract_dir="${VCONTAINER_EXTRACT_DIR:-$builddir/vcontainer-test-extracted}"
scripts/run-vcontainer-tests line 85
Quoted 5 lines
scripts/run-vcontainer-tests new file +5
-0+rm -rf "$extract_dir" +mkdir -p "$(dirname "$extract_dir")" + +# Self-extracting installer (silent, -y agrees to license, -d picks dir) +"$sdk_tarball" -d "$extract_dir" -yPatch 5 in this series added support for setting up the vcontainer tarball for a step. I think the commit message in this patch needs to explain why we can't use that in the steps that call run-vcontainer-tests.
Added information related to this context to v3.
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+ +# Prepare a Python venv so we don't pollute the worker's systempackages.
Quoted 7 lines
scripts/run-vcontainer-tests line 90
Quoted 3 lines
scripts/run-vcontainer-tests new file +3
-0+python3 -m venv "$builddir/meta-virt-test-venv" +# shellcheck disable=SC1091 +source "$builddir/meta-virt-test-venv/bin/activate"Note that this will shadow the venv we use when running test jobs on the autobuilder workers. That may be ok, but we need to be careful.
This venv is very very specifically for only the meta-virtualization tests. Other venvs in
autobuilder workers as far as I can tell do not have 'pip' and do not install 'pytest' nor
the other requirements.
Quoted 10 lines
Quoted 1 line
+# Avoid warnings by upgrading pip; install pytest/pexpect into the venv
via pip.
scripts/run-vcontainer-tests line 93
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+python3 -m pip install --quiet --upgrade pip setuptools wheel +python3 -m pip install --quiet --upgrade pytest pytest-timeout pexpectDo we want to automatically install the latest versions of these tools, via the network, in each build? It's better to pin versions to ensure that upgrades are planned rather than automagical.
In v3, I altered the pip, setuptools, wheel upgrade to exporting PIP_DISABLE_PIP_VERSION_CHECK=1
to silence the pip upgrade warning.
Quoted 3 lines
Also, should we store this list of dependencies in a requirements.txt file somewhere? Perhaps
tests/requirements.txtin meta-virtualization.
In v3, I changed the installation of pytest, pytest-timeout and pexpect to be instead from ameta-virtualization/tests/requirements.txt file (pending).
Quoted 12 lines
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+ +# Default marker filter excludes long running / infrastructuredependent tests.
Quoted 1 line
+marker_filter="${META_VIRT_PYTEST_MARKERS:-not slow and not network and
not boot and not incus and not k3s}"
Would it be better to have a positive marker, e.g. 'autobuilder', to mark tests that we do want to run? That would allow us to maintain the filtering solely in meta-virtualization instead of needing updates here as well when things change.
Making CI related changes in meta-virtualization is STRONGLY rejected by the maintainer of meta-virtualization and you will have to convince the upstream layer maintainer.
This pattern was lifted directly from the meta-virtualization README and existing examples.
Quoted 94 lines
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+ +# Per-suite test file selection. Uses -k/-m for fine-grained filteringand
Quoted 5 lines
scripts/run-vcontainer-tests new file +5
-0+# keeps the CLI small for logging clarity. +case "$suite" in + vdkr) + test_files=( + "tests/test_vdkr.py"scripts/run-vcontainer-tests
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+ "tests/test_vdkr_registry.py" + ) + ;; + vpdmn) + test_files=( + "tests/test_vpdmn.py" + ) + ;; + vcontainer) + # Broad vcontainer/bbclass/tooling coverage that doesn'trequire the
scripts/run-vcontainer-tests
Quoted 11 lines
scripts/run-vcontainer-tests new file +11
-0+ # vdkr/vpdmn CLI harness to be running. + test_files=( + "tests/test_container_cross_install.py" + "tests/test_container_registry_script.py" + "tests/test_vcontainer_auth_config.py" + "tests/test_multiarch_oci.py" + "tests/test_multilayer_oci.py" + ) + ;; + *) + echo "ERROR: unknown suite '$suite' (expectedvcontainer|vdkr|vpdmn)" >&2
Quoted 8 lines
scripts/run-vcontainer-tests new file +8
-0+ exit 2 + ;; +esac + +pytest_args=( + -v + --tb=short + -m "$marker_filter"scripts/run-vcontainer-tests
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+ --vdkr-dir "$extract_dir" + --junitxml="$builddir/pytest-$suite-results.xml" +) + +# Allow tests that consume an OCI image (import/save/load) to find one. +if [ -n "${TEST_OCI_IMAGE:-}" ] && [ -d "${TEST_OCI_IMAGE}" ]; then + pytest_args+=(--oci-image "$TEST_OCI_IMAGE") +fi + +# Pass architecture through when set in the environment (default is
x86_64).scripts/run-vcontainer-tests
Quoted 6 lines
scripts/run-vcontainer-tests new file +6
-0+if [ -n "${VDKR_ARCH:-}" ]; then + pytest_args+=(--arch "$VDKR_ARCH") +fi + +cd "$metavirtdir" +# Don't let a single failing test kill the whole step - collect thejunit
scripts/run-vcontainer-tests
Quoted 10 lines
scripts/run-vcontainer-tests new file +10
-0+# report, then surface the exit code via the junit file + exit status. +set +e +python3 -m pytest "${pytest_args[@]}" "${test_files[@]}" +rc=$? +set -e + +# Copy artefacts to the results dir if one was provided. +if [ -n "${RESULTS_DIR:-}" ]; then + mkdir -p "$RESULTS_DIR" + cp -f "$builddir/pytest-$suite-results.xml" "$RESULTS_DIR/"2>/dev/null || true
scripts/run-vcontainer-tests
Quoted 2 lines
scripts/run-vcontainer-tests new file +2
-0+ if [ -f /tmp/pytest-vcontainer.log ]; then + cp -f /tmp/pytest-vcontainer.log"$RESULTS_DIR/pytest-$suite.log" || true
scripts/run-vcontainer-tests line 164
Quoted 4 lines
scripts/run-vcontainer-tests new file +4
-0+ fi +fi + +exit $rc--
Paul Barker-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#4111):
https://lists.yoctoproject.org/g/yocto-patches/message/4111
Mute This Topic: https://lists.yoctoproject.org/mt/119603246/924729
Group Owner: [email protected] Unsubscribe:
https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 05/10] scripts: add vcontainer-tarball setup, integration, and publishing
In reply to Richard PurdieQuoted 51 lines · Richard Purdie via lists.yoctoproject.org
Quoted 37 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Introduce the vcontainer-tarball SDK plumbing used by container build jobs. The vcontainer-tarball is a meta-virtualization-derived SDK (modelled after buildtools-tarball) that ships the container build toolchain so worker jobs do not need to rebuild it for every step.
*
scripts/utils.py: addsetup_vcontainer_tarball(), and add anenv_globkeyword argument tosetup_tools_tarball()andenable_tools_tarball()so the vcontainer-tarball can source its specific environment-setup-ci file rather than the universal glob.
*scripts/run-config: source the vcontainer-tarball environment for build-targets / cmds / test-targets / plain-cmds steps, gated by a new NOVCONTAINER step variable so individual steps (such as the dashboard indexing step) can opt out independently of NOBUILDTOOLS.
*scripts/shared-repo-unpack: invokesetup_vcontainer_tarballso workers extract the SDK during unpack.
*scripts/publish-artefacts: publish the vcontainer-tarball artefact so downstream test jobs can fetch a stable SDK.AI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>--- config.json | 1 + scripts/publish-artefacts | 5 +++++ scripts/run-config | 19 +++++++++++++++++++ scripts/shared-repo-unpack | 1 + scripts/utils.py | 22 ++++++++++++++++++---- 5 files changed, 44 insertions(+), 4 deletions(-)config.json +0
-0config.json line 1437
Quoted 5 lines
config.json +1
-0@@ -1433,6 +1433,7 @@ "step3" : { "shortname" : "Populate/update dashboard site", "NOBUILDTOOLS" : true, + "NOVCONTAINER" : true, "EXTRACMDS" : ["${SCRIPTSDIR}/run-dashboard-index${HELPERBUILDDIR}/../"]
config.json line 1439
Quoted 2 lines
config.json +0
-0} },Why is this step needing to opt out of a vcontainer? vcontainers are only enabled for specific steps and off by default?
Quoted 9 lines
I looked further and realised you're basically using a vcontainer everywhere as far as I can tell? Why do we want to have the overhead of doing that outside of the vcontainer builds?
For buildtools, it makes sense we need it everywhere except where we configure otherwise but I think for vcontainer, you want to opposite, you only use it in jobs where we actually need/use it?
The simple answer is that the 'buildtools' pattern was followed and an attempt
was made to reuse the 'buildtools' code as much as possible. Upon further review, the 'extratools' pattern is a better fit. This was addressed in v3.
Cheers,
Quoted 17 lines
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#4153):
https://lists.yoctoproject.org/g/yocto-patches/message/4153
Mute This Topic: https://lists.yoctoproject.org/mt/119603244/924729
Group Owner: [email protected] Unsubscribe:
https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 08/10] scripts: add container registry push, auth, tagging, runtime selection
In reply to Richard PurdieQuoted 33 lines · Richard Purdie via lists.yoctoproject.org
Quoted 24 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Add the push-containers infrastructure that drives the post-build steps for the 'containers-library' job. After each build step the runtime container store is harvested and pushed to one or more registries with derived per-step tags.
* config.json: add
CONTAINER_REGISTRIES,CONTAINER_AUTH_CONFIG,CONTAINER_RUNTIME,CONTAINER_TAG_CMDS,CONTAINER_VERSION_RECIPEandCONTAINER_IMAGE_MAPconfiguration knobs.
*scripts/run-config: drive push-containers as a post-step action. Tags are generated from recipe and distro metadata (yocto- tag uses major.minor on snapshots and full PV on releases) withCONTAINER_VERSION_RECIPEallowing a step to source PV from a different recipe than the image itself.
* Registry auth is staged via.../config.jsonor podman.../auth.jsonusingCONTAINER_AUTH_CONFIG, replacing an interactive login that could hang.CONTAINER_RUNTIMEpicks between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
* Robustness: skip gracefully when no registries are configured, fix the OCI directory path, handle memres already running, and avoid hanging when memres has not yet come up.Quoted 6 lines
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by Tim Orling <[email protected]>--- config.json | 5 ++ scripts/run-config | 129 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+)
Quoted 5 lines
This is adding way too much code for a specific use case to run-config. This needs to be in a separate script (in the same way docs builds or other things are).
Agreed, and I already had the sense that this "wall of text" was too much. Broken out into a dedicated 'run-pull-containers' script in v3.
Cheers,
Quoted 17 lines
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#4152):
https://lists.yoctoproject.org/g/yocto-patches/message/4152
Mute This Topic: https://lists.yoctoproject.org/mt/119603248/924729
Group Owner: [email protected] Unsubscribe:
https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [yocto-patches] [yocto-autobuilder-helper][PATCH v2 01/10] scripts/utils: fix stale extraction dir when tarball is updated
In reply to Paul BarkerQuoted 83 lines · Paul Barker via lists.yoctoproject.org
scripts/utils.py def setup_tools_tarball(ourconfig, btdir,
Quoted 28 lines · Tim Orling via lists.yoctoproject.org
From: Tim Orling <[email protected]>
Previously the entire download/cache-validation block in
setup_tools_tarball()was guarded byif not os.path.exists(btdir). Once the extraction directory existed from a prior build, every subsequent call was a no-op: the cached tarball was never re-validated, so a freshly-published SDK (e.g. vcontainer-tarball-latest) was silently ignored and the stale btdir kept being used.Fix by moving the lock/download block outside the btdir existence guard so cache validation always runs. Track
tarball_updatedto know when the cached download was actually replaced, then remove the stale btdir when True so the fresh tarball is re-extracted.Also add an mtime-based staleness check for local-path (cp) sources so that a newer source file automatically invalidates the cached copy.
AI-Generated: Claude Cowork Sonnet 4.6
Signed-off-by Tim Orling <[email protected]>--- scripts/utils.py | 64 +++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 23 deletions(-)scripts/utils.py +0
-0@@ -481,31 +481,49 @@ def setup_tools_tarball(ourconfig, btdir,bttarball, name="buildtools"):
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 4 lines
scripts/utils.py +1
-0if ";" in bttarball: bttarball, sha256 = bttarball.split(";") btdir = os.path.abspath(btdir) + btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) +"/cluster-downloads-cache/" +
os.path.basename(bttarball)scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 4 lines
scripts/utils.py +4
-0+ btlock = btdlpath + ".lock" + if not os.path.exists(os.path.dirname(btdlpath)): + os.makedirs(os.path.dirname(btdlpath), exist_ok=True) + # Always run the cache-validation / download step so that afreshly
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 489
Quoted 1 line
scripts/utils.py +1
-0+ # published tarball is detected even when btdir already existsfrom a
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 490
Quoted 1 line
scripts/utils.py +1
-0+ # previous build. tarball_updated is set to True whenever thecached
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 491
Quoted 1 line
scripts/utils.py +1
-0+ # download is replaced, which triggers removal of the stalebtdir.
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 11 lines
scripts/utils.py +11
-0+ tarball_updated = False + while True: + try: + with open(btlock, 'a+') as lf: + fileno = lf.fileno() + fcntl.flock(fileno, fcntl.LOCK_EX) + if sha256 and os.path.exists(btdlpath): + dl_sha256 = sha256_file(btdlpath) + if dl_sha256 != sha256: + os.unlink(btdlpath) + elif bttarball.startswith("/") and
os.path.exists(btdlpath):scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 503
Quoted 1 line
scripts/utils.py +1
-0+ # For local-path sources (e.g.vcontainer-tarball-latest)
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 504
Quoted 1 line
scripts/utils.py +1
-0+ # invalidate the cached copy when the source isnewer so
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 505
Quoted 1 line
scripts/utils.py +1
-0+ # that a freshly-published tarball is alwayspicked up.
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 506
Quoted 1 line
scripts/utils.py +1
-0+ if os.path.getmtime(bttarball) >
os.path.getmtime(btdlpath):scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 5 lines
scripts/utils.py +5
-0+ os.unlink(btdlpath) + os.unlink(btdlpath) + if not os.path.exists(btdlpath): + if bttarball.startswith("/"): + subprocess.check_call(["cp", bttarball,btdlpath])
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 513
Quoted 2 lines
scripts/utils.py +2
-0+ else: + subprocess.check_call(["wget", "-O",btdlpath, bttarball])
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): · line 519
Quoted 6 lines
scripts/utils.py +6
-0+ os.chmod(btdlpath, 0o775) + tarball_updated = True + break + except OSError: + # We raced with someone else, try again + pass
Quoted 9 lines
Hi Tim,
This loop allows us to handle random OSError conditions caused by a race, but it turns a deterministic OSError into an infinite loop. The logic already exists, you're just moving it around, but now I have seen it I cannot unsee it!
Should we add a timeout while we are here?
This patch was already merged, but the timeout concern was submitted separately:
https://lore.kernel.org/yocto-patches/[email protected]/
Quoted 57 lines
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 8 lines
scripts/utils.py +5
-1+ # If the underlying tarball changed, remove any stale extraction + # directory so it is re-extracted below. + if tarball_updated and os.path.exists(btdir): + print("Removing stale %s extraction at %s" % (name, btdir)) + subprocess.check_call(["rm", "-rf", btdir]) if not os.path.exists(btdir): - btdlpath = getconfig("BASE_SHAREDDIR", ourconfig) +"/cluster-downloads-cache/" +
os.path.basename(bttarball)Quoted 6 lines
scripts/utils.py +0
-5print("Extracting %s %s" % (name, bttarball)) - btlock = btdlpath + ".lock" - if not os.path.exists(os.path.dirname(btdlpath)): - os.makedirs(os.path.dirname(btdlpath), exist_ok=True) - while True: - try:scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
Quoted 10 lines
scripts/utils.py +0
-10- with open(btlock, 'a+') as lf: - fileno = lf.fileno() - fcntl.flock(fileno, fcntl.LOCK_EX) - if sha256 and os.path.exists(btdlpath): - dl_sha256 = sha256_file(btdlpath) - if dl_sha256 != sha256: - os.unlink(btdlpath) - if not os.path.exists(btdlpath): - if bttarball.startswith("/"): - subprocess.check_call(["cp", bttarball,btdlpath])
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, · line 503
Quoted 2 lines
scripts/utils.py +0
-2- else: - subprocess.check_call(["wget", "-O",btdlpath, bttarball])
scripts/utils.py def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"): · line 528
Quoted 7 lines
scripts/utils.py +0
-5- os.chmod(btdlpath, 0o775) - break - except OSError: - # We raced with someone else, try again - pass subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"]) enable_tools_tarball(btdir, name)Best regards,
--
Paul Barker-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#4110):
https://lists.yoctoproject.org/g/yocto-patches/message/4110
Mute This Topic: https://lists.yoctoproject.org/mt/119603240/924729
Group Owner: [email protected] Unsubscribe:
https://lists.yoctoproject.org/g/yocto-patches/leave/13169857/924729/1023951714/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-