tests: standalone test-suite framework plus the first unit test
v4 cover tests: standalone test-suite framework plus the first unit test
wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.
- the suite skeleton (pyproject test extra, pytest config, layout,
a justfile command runner, and a README Testing section);
- a conftest that imports wic once up front (stopping the session with
a clear message if that fails) and prints a session banner;
- optional coverage and ruff linting, each exposed as just recipes;
- a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError
handler works, as its own commit; - unit tests for mkdirhier() that pin its behaviour and catch that bug.
The suite is green at every commit and the test tree is ruff-clean.
Changes in v4 (in response to review feedback, thanks to Paul Barker):
v4 restructures the v3 10-patch series into 6 patches. How the old patches map to the new ones:
- v3 1/10 (skeleton) -> v4 1/6, which also gains a justfile command runner in place of the run-tests.sh wrapper and no longer creates a tests/docs/ tree;
- v3 2/10 (conftest banner) -> v4 2/6, which also absorbs the
wic-import check (see next line);
- v3 3/10 (run-tests.sh wrapper) -> dropped; the justfile replaces it
and its wic-import check moves into conftest, so the check runs on
every invocation rather than only through the wrapper;
- v3 4/10 (coverage) -> v4 3/6, exposed as just recipes;
- v3 5/10 (ruff linting) -> v4 4/6, exposed as just recipes, with
ruff now pinned exactly so the clean-bar gate is reproducible;
- v3 8/10 (tests/** E402 ignore) -> folded into v4 4/6;
- v3 6/10, 7/10, 10/10 (docs README, authoring guide, review rubric)
-> dropped; the suite ships a short README Testing section instead
of a tests/docs/ tree;- v3 9/10 (test plus folded-in errno fix) -> split into v4 5/6 (the errno fix, on its own, first) and v4 6/6 (the tests), and the tests are reframed to target mkdirhier's own behaviour rather than re-testing os.makedirs.
Dependency versions were also refreshed to current releases.
Changes in v3:
- test_bb_utils used the tmp_path fixture instead of tempfile.mkdtemp(), so the tests no longer leaked scratch directories under /tmp;
- documented the standard pytest scratch-directory controls (TMPDIR
and --basetemp) rather than a custom variable;
- added a review-rubric doc.
Changes in v2:
- v1 was a single ~6000-line commit that used xfail markers to record the bugs it found; v2 broke the work into a reviewable series where each test lands green next to the fix that makes it pass, with no xfails.
Trevor Woerner (6):
tests: add the standalone unit-test suite skeleton
tests: add conftest with a wic-import preflight and session banner
tests: add optional coverage reporting
add ruff linting
bb/utils: import errno so mkdirhier's OSError handler works tests/unit/test_bb_utils: cover mkdirhier()
.gitignore | 4 ++ README.md | 15 +++++++ justfile | 27 +++++++++++++ pyproject.toml | 24 +++++++++++ src/wic/bb/utils.py | 1 + tests/conftest.py | 28 +++++++++++++ tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++ 7 files changed, 180 insertions(+) create mode 100644 justfile create mode 100644 tests/conftest.py create mode 100644 tests/unit/test_bb_utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 1/6 tests: add the standalone unit-test suite skeleton
wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This starts a small standalone suite that runs from a plain checkout, with no BitBake and no OpenEmbedded build.
This is the skeleton only and adds no tests. It gives pyproject.toml a "tests" extra (pytest and the just command runner) and a pytest config section, adds a justfile with default and tests recipes, creates tests/unit/ with a placeholder .gitkeep, ignores .pytest_cache/, and adds a README Testing section covering install and how to run the suite.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- drop the tests/docs/ documentation tree; the suite now ships with a short README Testing section only.
- add a justfile as the command runner and drop the run-tests.sh wrapper; just joins the tests extra.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 1 + README.md | 15 +++++++++++++++ justfile | 10 ++++++++++ pyproject.toml | 14 ++++++++++++++ tests/unit/.gitkeep | 0 5 files changed, 40 insertions(+) create mode 100644 justfile create mode 100644 tests/unit/.gitkeep
.gitignore +1 -0
@@ -1 +1,2 @@ **/__pycache__ +.pytest_cache/
README.md +15 -0
@@ -47,6 +47,21 @@ environment file or folder (generated via `bitbake -c rootfs_wicenv - `src/bb/*`: various bitbake helpers that were brought along and used in other parts of wic. - `src/oe/*`: various oe-core helpers that were brought along and used in other parts of wic. +## Testing + +wic's test suite lives under `tests/` and runs from a plain checkout +with nothing but `pytest`; it needs no BitBake and no OpenEmbedded +build. Install the test extras and run it: + +``` +pip install -e ".[tests]" +just tests +``` + +The test extras include [just](https://just.systems/), the command +runner used to drive the suite; run `just` on its own to list the +available recipes. + ## Contributing Please send all patches, comments, or questions to the yocto-patches
justfile new file +10 -0
@@ -0,0 +1,10 @@ +# wic test-suite command runner. Run `just` to list recipes. + +# List the available recipes. +default: + @just --list + +# Run the test suite. Extra args pass through to pytest +# (e.g. `just tests -k <expr>` or `just tests tests/unit`). +tests *args: + pytest {{args}}
pyproject.toml +14 -0
@@ -21,6 +21,12 @@ classifiers = [ Homepage = "https://git.yoctoproject.org/wic" Repository = "https://git.yoctoproject.org/wic" +[project.optional-dependencies] +tests = [ + "pytest >= 9.1.1", + "rust-just >= 1.55.1", +] + [project.scripts] wic = "wic.cli:main"
@@ -36,3 +42,11 @@ build-backend = "hatchling.build" [tool.hatch.version] path = "src/wic/cli.py" + +[tool.pytest.ini_options] +testpaths = ["tests"] +# Keep a test's scratch directory only when that test fails, and keep +# just the most recent failing session, so repeated runs do not pile up +# leftover directories under the pytest base temp directory. +tmp_path_retention_policy = "failed" +tmp_path_retention_count = 1
tests/unit/.gitkeep new file +0 -0
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 2/6 tests: add conftest with a wic-import preflight and session banner
In reply to Trevor Woernertests/conftest.py prepares the suite and describes the run. It adds the in-tree src/ to sys.path so the tests import the wic in the checkout without an install, then imports wic once up front: if that fails, the session stops immediately with pytest.exit() and the reason, rather than letting every test error out one by one against a missing or wrong wic.
A pytest_report_header hook prints a short banner naming the host, the Python and pytest versions, and the wic under test with its version and imported path, so a run can be attributed when more than one checkout or virtualenv is in play.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- move the wic-import check here from the run-tests.sh wrapper, so it runs on every invocation regardless of how pytest is launched; on failure the session stops with pytest.exit().
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- tests/conftest.py | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 tests/conftest.py
tests/conftest.py new file +28 -0
@@ -0,0 +1,28 @@ +# Session setup and banner for the wic test suite. + +import platform +import sys +from pathlib import Path + +import pytest + +_SRC = Path(__file__).resolve().parent.parent / "src" +if str(_SRC) not in sys.path: + sys.path.insert(0, str(_SRC)) + +try: + import wic.cli as _wic_cli +except Exception as exc: + pytest.exit("cannot import wic from %s: %s" % (_SRC, exc), returncode=1) + + +def pytest_report_header(config): + return "\n".join([ + "wic test suite", + " host: %s %s %s" % ( + platform.node(), platform.system(), platform.machine()), + " python: %s pytest: %s" % ( + platform.python_version(), pytest.__version__), + " wic: %s (%s)" % ( + getattr(_wic_cli, "__version__", "(unknown)"), _wic_cli.__file__), + ])
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 3/6 tests: add optional coverage reporting
In reply to Trevor WoernerAdd branch coverage of the wic source as opt-in just recipes, so a plain run stays fast and quiet. coverage and pytest-cov join the tests extra; the coverage recipe prints a terminal report of the lines that were missed, and coverage-html additionally writes a browsable report to htmlcov/. .gitignore ignores the .coverage data file and htmlcov/.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- expose coverage through just recipes (coverage, coverage-html) rather than run-tests.sh flags.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 2 ++ justfile | 8 ++++++++ pyproject.toml | 2 ++ 3 files changed, 12 insertions(+)
.gitignore +2 -0
@@ -1,2 +1,4 @@ **/__pycache__ .pytest_cache/ +.coverage +htmlcov/
justfile +8 -0
@@ -8,3 +8,11 @@ default: # (e.g. `just tests -k <expr>` or `just tests tests/unit`). tests *args: pytest {{args}} + +# Run the suite with a terminal branch-coverage report for wic. +coverage *args: + pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing + +# Run the suite with an HTML coverage report (written to htmlcov/). +coverage-html *args: + pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing --cov-report=html
pyproject.toml +2 -0
@@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic" [project.optional-dependencies] tests = [ "pytest >= 9.1.1", + "coverage >= 7.15.0", + "pytest-cov >= 7.1.0", "rust-just >= 1.55.1", ]
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 4/6 add ruff linting
In reply to Trevor WoernerLint with ruff via just recipes. lint-tests runs ruff over tests/ and is held to a clean bar; lint also runs it over src/, whose findings are a preview since the wic source is not yet ruff-clean. ruff joins the tests extra and .gitignore ignores its .ruff_cache/.
ruff is pinned exactly so the gate is reproducible: a ruff minor release can add rules that turn a previously clean tree red, so every machine must lint with the same version.
Test modules relax E402 via per-file-ignores: they prepend src/ to sys.path before importing wic, which trips the rule, and that ordering is deliberate.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- expose linting through just recipes (lint-tests, lint) rather than run-tests.sh flags.
- pin ruff exactly (was a stale >= floor) so the clean-bar gate is reproducible across machines.
- fold in the tests/** E402 ignore (its own patch in v2 and v3) and drop the linting.md doc; the pyproject.toml comment explains it.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 1 + justfile | 9 +++++++++ pyproject.toml | 8 ++++++++ 3 files changed, 18 insertions(+)
.gitignore +1 -0
@@ -2,3 +2,4 @@ .pytest_cache/ .coverage htmlcov/ +.ruff_cache/
justfile +9 -0
@@ -16,3 +16,12 @@ coverage *args: # Run the suite with an HTML coverage report (written to htmlcov/). coverage-html *args: pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing --cov-report=html + +# Lint the test suite; it is held to a clean bar, so this must be clean. +lint-tests: + ruff check tests + +# Lint the test suite and then the wic source. src/ is not yet +# ruff-clean, so its findings are a preview, not a gate. +lint: lint-tests + ruff check src
pyproject.toml +8 -0
@@ -26,6 +26,7 @@ tests = [ "pytest >= 9.1.1", "coverage >= 7.15.0", "pytest-cov >= 7.1.0", + "ruff == 0.15.20", "rust-just >= 1.55.1", ]
@@ -52,3 +53,10 @@ testpaths = ["tests"] # leftover directories under the pytest base temp directory. tmp_path_retention_policy = "failed" tmp_path_retention_count = 1 + +[tool.ruff.lint.per-file-ignores] +# Test modules prepend the in-tree src/ directory to sys.path before +# importing wic, so the suite runs against a checkout that has not been +# installed. That bootstrap runs before the wic imports, which trips +# E402; the ordering is deliberate, so ignore E402 for the test tree. +"tests/**" = ["E402"]
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 5/6 bb/utils: import errno so mkdirhier's OSError handler works
In reply to Trevor Woernermkdirhier() calls os.makedirs(directory, exist_ok=True) and, if that still raises OSError, re-checks the error before deciding what to do:
except OSError as e:
if e.errno != errno.EEXIST or not os.path.isdir(directory):
raise e
but the module never imports errno. The reference to errno.EEXIST is only evaluated when os.makedirs() actually raises, so the defect stays hidden on every ordinary call. The moment a real OSError occurs -- a parent component that is a file, a permission error, anything -- the handler itself raises NameError: name 'errno' is not defined, masking the original error with a misleading one.
Import errno so the handler behaves as intended: it re-raises every real error, while tolerating only an EEXIST whose path is in fact already a directory.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- new commit: the errno import fix now lands on its own, before the test that exercises it. In v2 and v3 it was folded into the test_bb_utils commit.
--- src/wic/bb/utils.py | 1 + 1 file changed, 1 insertion(+)
src/wic/bb/utils.py +1 -0
@@ -1,6 +1,7 @@ """ Minimal subset of BitBake's bb.utils used by standalone wic. """ +import errno import os # from bitbake/lib/bb/utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v4 6/6 tests/unit/test_bb_utils: cover mkdirhier()
In reply to Trevor WoernerAdd unit tests for mkdirhier(), the one function in wic.bb.utils. They pin the behaviour wic relies on so it cannot regress: it creates missing directories, accepts a directory that already exists, rejects a path containing an unexpanded bitbake variable (${...}), and allows a name with a lone brace or a lone dollar, neither of which is the ${ marker.
Three tests cover the OSError handler that the errno import repairs: a path whose parent component is a regular file, a target that already exists as a file rather than a directory, and a directory that appears concurrently mid-call (a create race), which must be treated as success. Run against a wic.bb.utils that does not import errno, all three fail with NameError instead, so they catch that bug directly.
The tests replace the tests/unit/.gitkeep placeholder.
AI-Generated: codex/claude-opus 4.8 (xhigh)
---
changes in v4:
- reframe the tests to target mkdirhier's own behaviour: drop the cases that only re-exercised os.makedirs and the assertions that pinned implementation details.
- the errno fix moved to its own preceding commit; this is tests only.
changes in v3:
- switch the tests from tempfile.mkdtemp() to pytest's tmp_path fixture so each test's scratch directory is cleaned up instead of leaking under /tmp; no change to what is tested.
changes in v2:
- v1 recorded this bug with an xfail marker in one large commit; v2 drops the xfail, asserts the correct behaviour directly, and lands the one-line errno-import fix in this same commit so the test passes green.
--- tests/unit/.gitkeep | 0 tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+) delete mode 100644 tests/unit/.gitkeep create mode 100644 tests/unit/test_bb_utils.py
tests/unit/.gitkeep deleted +0 -0
tests/unit/test_bb_utils.py new file +81 -0
81 lines not on this page · raw message
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
Quoted 12 lines · Trevor Woerner via lists.yoctoproject.org
wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.
- the suite skeleton (pyproject test extra, pytest config, layout,
a justfile command runner, and a README Testing section); - a conftest that imports wic once up front (stopping the session with a clear message if that fails) and prints a session banner; - optional coverage and ruff linting, each exposed as just recipes; - a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError handler works, as its own commit;
Quoted 4 lines
- unit tests for
mkdirhier()that pin its behaviour and catch that bug.The suite is green at every commit and the test tree is ruff-clean.
All LGTM now!
Best regards,
Signature · 1 line
Paul Barker
Patch series applied to wic, master branch.
Thanks Paul!
Quoted 82 lines · Trevor Woerner
wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.
- the suite skeleton (pyproject test extra, pytest config, layout,
a justfile command runner, and a README Testing section); - a conftest that imports wic once up front (stopping the session with a clear message if that fails) and prints a session banner; - optional coverage and ruff linting, each exposed as just recipes; - a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError handler works, as its own commit;- unit tests for
mkdirhier()that pin its behaviour and catch that bug.The suite is green at every commit and the test tree is ruff-clean.
Changes in v4 (in response to review feedback, thanks to Paul Barker):
v4 restructures the v3 10-patch series into 6 patches. How the old patches map to the new ones:
- v3 1/10 (skeleton) -> v4 1/6, which also gains a justfile command runner in place of the
run-tests.shwrapper and no longer creates a tests/docs/ tree;
- v3 2/10 (conftest banner) -> v4 2/6, which also absorbs thewic-import check (see next line); - v3 3/10 (run-tests.sh wrapper) -> dropped; the justfile replaces it and its wic-import check moves into conftest, so the check runs on every invocation rather than only through the wrapper; - v3 4/10 (coverage) -> v4 3/6, exposed as just recipes; - v3 5/10 (ruff linting) -> v4 4/6, exposed as just recipes, with ruff now pinned exactly so the clean-bar gate is reproducible; - v3 8/10 (tests/** E402 ignore) -> folded into v4 4/6; - v3 6/10, 7/10, 10/10 (docs README, authoring guide, review rubric) -> dropped; the suite ships a short README Testing section instead of a tests/docs/ tree;- v3 9/10 (test plus folded-in errno fix) -> split into v4 5/6 (the errno fix, on its own, first) and v4 6/6 (the tests), and the tests are reframed to target mkdirhier's own behaviour rather than re-testing os.makedirs.
Dependency versions were also refreshed to current releases.
Changes in v3:
-
test_bb_utilsused thetmp_pathfixture instead oftempfile.mkdtemp(), so the tests no longer leaked scratch directories under /tmp;
- documented the standard pytest scratch-directory controls (TMPDIRand --basetemp) rather than a custom variable;- added a review-rubric doc.
Changes in v2:
- v1 was a single ~6000-line commit that used xfail markers to record the bugs it found; v2 broke the work into a reviewable series where each test lands green next to the fix that makes it pass, with no xfails.
Trevor Woerner (6):
tests: add the standalone unit-test suite skeleton
tests: add conftest with a wic-import preflight and session banner
tests: add optional coverage reporting
add ruff linting
bb/utils: import errno so mkdirhier's OSError handler works tests/unit/test_bb_utils: covermkdirhier().gitignore | 4 ++ README.md | 15 +++++++ justfile | 27 +++++++++++++ pyproject.toml | 24 +++++++++++ src/wic/bb/utils.py | 1 + tests/conftest.py | 28 +++++++++++++ tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++ 7 files changed, 180 insertions(+) create mode 100644 justfile create mode 100644 tests/conftest.py create mode 100644 tests/unit/test_bb_utils.pySignature · 1 line
2.50.0.173.g8b6f19ccfc3a