Skip to contentSkip to conversation

tests: standalone test-suite framework plus the first unit test

PATCH In review Trevor Woerner 2 participants 9 messages Jul 6 – Jul 9

lore
Share & tools
lore
#0 lore raw

v4 cover tests: standalone test-suite framework plus the first unit test

wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.

- the suite skeleton (pyproject test extra, pytest config, layout,

    a justfile command runner, and a README Testing section);
  - a conftest that imports wic once up front (stopping the session with
    a clear message if that fails) and prints a session banner;
  - optional coverage and ruff linting, each exposed as just recipes;
  - a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError
    handler works, as its own commit;

- unit tests for mkdirhier() that pin its behaviour and catch that bug.

The suite is green at every commit and the test tree is ruff-clean.

Changes in v4 (in response to review feedback, thanks to Paul Barker):

v4 restructures the v3 10-patch series into 6 patches. How the old patches map to the new ones:

- v3 1/10 (skeleton) -> v4 1/6, which also gains a justfile command runner in place of the run-tests.sh wrapper and no longer creates a tests/docs/ tree;
- v3 2/10 (conftest banner) -> v4 2/6, which also absorbs the

    wic-import check (see next line);
  - v3 3/10 (run-tests.sh wrapper) -> dropped; the justfile replaces it
    and its wic-import check moves into conftest, so the check runs on
    every invocation rather than only through the wrapper;
  - v3 4/10 (coverage) -> v4 3/6, exposed as just recipes;
  - v3 5/10 (ruff linting) -> v4 4/6, exposed as just recipes, with
    ruff now pinned exactly so the clean-bar gate is reproducible;
  - v3 8/10 (tests/** E402 ignore) -> folded into v4 4/6;
  - v3 6/10, 7/10, 10/10 (docs README, authoring guide, review rubric)
    -> dropped; the suite ships a short README Testing section instead
    of a tests/docs/ tree;

- v3 9/10 (test plus folded-in errno fix) -> split into v4 5/6 (the errno fix, on its own, first) and v4 6/6 (the tests), and the tests are reframed to target mkdirhier's own behaviour rather than re-testing os.makedirs.

Dependency versions were also refreshed to current releases.

Changes in v3:

- test_bb_utils used the tmp_path fixture instead of tempfile.mkdtemp(), so the tests no longer leaked scratch directories under /tmp;
- documented the standard pytest scratch-directory controls (TMPDIR

    and --basetemp) rather than a custom variable;

- added a review-rubric doc.

Changes in v2:

- v1 was a single ~6000-line commit that used xfail markers to record the bugs it found; v2 broke the work into a reviewable series where each test lands green next to the fix that makes it pass, with no xfails.

Trevor Woerner (6):
tests: add the standalone unit-test suite skeleton
tests: add conftest with a wic-import preflight and session banner
tests: add optional coverage reporting
add ruff linting
bb/utils: import errno so mkdirhier's OSError handler works tests/unit/test_bb_utils: cover mkdirhier()

 .gitignore                  |  4 ++
 README.md                   | 15 +++++++
 justfile                    | 27 +++++++++++++
 pyproject.toml              | 24 +++++++++++
 src/wic/bb/utils.py         |  1 +
 tests/conftest.py           | 28 +++++++++++++
 tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++
 7 files changed, 180 insertions(+)
 create mode 100644 justfile
 create mode 100644 tests/conftest.py
 create mode 100644 tests/unit/test_bb_utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#1 lore raw

PATCH v4 1/6 tests: add the standalone unit-test suite skeleton

wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This starts a small standalone suite that runs from a plain checkout, with no BitBake and no OpenEmbedded build.

This is the skeleton only and adds no tests. It gives pyproject.toml a "tests" extra (pytest and the just command runner) and a pytest config section, adds a justfile with default and tests recipes, creates tests/unit/ with a placeholder .gitkeep, ignores .pytest_cache/, and adds a README Testing section covering install and how to run the suite.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- drop the tests/docs/ documentation tree; the suite now ships with a short README Testing section only.
- add a justfile as the command runner and drop the run-tests.sh wrapper; just joins the tests extra.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.

---
 .gitignore          |  1 +
 README.md           | 15 +++++++++++++++
 justfile            | 10 ++++++++++
 pyproject.toml      | 14 ++++++++++++++
 tests/unit/.gitkeep |  0
 5 files changed, 40 insertions(+)
 create mode 100644 justfile
 create mode 100644 tests/unit/.gitkeep
.gitignore +1 -0
index eeb8a6ec4087..a0f016bc5823 100644
@@ -1 +1,2 @@
 **/__pycache__
+.pytest_cache/
README.md +15 -0
index 75229421763c..c9c38eb5e98a 100644
@@ -47,6 +47,21 @@ environment file or folder (generated via `bitbake -c rootfs_wicenv
 - `src/bb/*`: various bitbake helpers that were brought along and used in other parts of wic.
 - `src/oe/*`: various oe-core helpers that were brought along and used in other parts of wic.
 
+## Testing
+
+wic's test suite lives under `tests/` and runs from a plain checkout
+with nothing but `pytest`; it needs no BitBake and no OpenEmbedded
+build. Install the test extras and run it:
+
+```
+pip install -e ".[tests]"
+just tests
+```
+
+The test extras include [just](https://just.systems/), the command
+runner used to drive the suite; run `just` on its own to list the
+available recipes.
+
 ## Contributing
 
 Please send all patches, comments, or questions to the yocto-patches
justfile new file +10 -0
new file mode 100644
index 000000000000..41a9d3d236f4
@@ -0,0 +1,10 @@
+# wic test-suite command runner. Run `just` to list recipes.
+
+# List the available recipes.
+default:
+    @just --list
+
+# Run the test suite. Extra args pass through to pytest
+# (e.g. `just tests -k <expr>` or `just tests tests/unit`).
+tests *args:
+    pytest {{args}}
pyproject.toml +14 -0
index fdc1ce0f5ece..e60c4fcc1595 100644
@@ -21,6 +21,12 @@ classifiers = [
 Homepage = "https://git.yoctoproject.org/wic"
 Repository = "https://git.yoctoproject.org/wic"
 
+[project.optional-dependencies]
+tests = [
+    "pytest >= 9.1.1",
+    "rust-just >= 1.55.1",
+]
+
 [project.scripts]
 wic = "wic.cli:main"
 
@@ -36,3 +42,11 @@ build-backend = "hatchling.build"
 
 [tool.hatch.version]
 path = "src/wic/cli.py"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+# Keep a test's scratch directory only when that test fails, and keep
+# just the most recent failing session, so repeated runs do not pile up
+# leftover directories under the pytest base temp directory.
+tmp_path_retention_policy = "failed"
+tmp_path_retention_count  = 1
tests/unit/.gitkeep new file +0 -0
new file mode 100644
index 000000000000..e69de29bb2d1
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#2 lore raw

PATCH v4 2/6 tests: add conftest with a wic-import preflight and session banner

In reply to Trevor Woerner

tests/conftest.py prepares the suite and describes the run. It adds the in-tree src/ to sys.path so the tests import the wic in the checkout without an install, then imports wic once up front: if that fails, the session stops immediately with pytest.exit() and the reason, rather than letting every test error out one by one against a missing or wrong wic.

A pytest_report_header hook prints a short banner naming the host, the Python and pytest versions, and the wic under test with its version and imported path, so a run can be attributed when more than one checkout or virtualenv is in play.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- move the wic-import check here from the run-tests.sh wrapper, so it runs on every invocation regardless of how pytest is launched; on failure the session stops with pytest.exit().
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.

---
 tests/conftest.py | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 tests/conftest.py
tests/conftest.py new file +28 -0
new file mode 100644
index 000000000000..13b8953cec40
@@ -0,0 +1,28 @@
+# Session setup and banner for the wic test suite.
+
+import platform
+import sys
+from pathlib import Path
+
+import pytest
+
+_SRC = Path(__file__).resolve().parent.parent / "src"
+if str(_SRC) not in sys.path:
+    sys.path.insert(0, str(_SRC))
+
+try:
+    import wic.cli as _wic_cli
+except Exception as exc:
+    pytest.exit("cannot import wic from %s: %s" % (_SRC, exc), returncode=1)
+
+
+def pytest_report_header(config):
+    return "\n".join([
+        "wic test suite",
+        "  host:   %s  %s %s" % (
+            platform.node(), platform.system(), platform.machine()),
+        "  python: %s   pytest: %s" % (
+            platform.python_version(), pytest.__version__),
+        "  wic:    %s  (%s)" % (
+            getattr(_wic_cli, "__version__", "(unknown)"), _wic_cli.__file__),
+    ])
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#3 lore raw

PATCH v4 3/6 tests: add optional coverage reporting

In reply to Trevor Woerner

Add branch coverage of the wic source as opt-in just recipes, so a plain run stays fast and quiet. coverage and pytest-cov join the tests extra; the coverage recipe prints a terminal report of the lines that were missed, and coverage-html additionally writes a browsable report to htmlcov/. .gitignore ignores the .coverage data file and htmlcov/.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- expose coverage through just recipes (coverage, coverage-html) rather than run-tests.sh flags.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.

---
 .gitignore     | 2 ++
 justfile       | 8 ++++++++
 pyproject.toml | 2 ++
 3 files changed, 12 insertions(+)
.gitignore +2 -0
index a0f016bc5823..e30cafa40f7f 100644
@@ -1,2 +1,4 @@
 **/__pycache__
 .pytest_cache/
+.coverage
+htmlcov/
justfile +8 -0
index 41a9d3d236f4..1be3e01e7c7d 100644
@@ -8,3 +8,11 @@ default:
 # (e.g. `just tests -k <expr>` or `just tests tests/unit`).
 tests *args:
     pytest {{args}}
+
+# Run the suite with a terminal branch-coverage report for wic.
+coverage *args:
+    pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing
+
+# Run the suite with an HTML coverage report (written to htmlcov/).
+coverage-html *args:
+    pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing --cov-report=html
pyproject.toml +2 -0
index e60c4fcc1595..2ad694b13358 100644
@@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic"
 [project.optional-dependencies]
 tests = [
     "pytest >= 9.1.1",
+    "coverage >= 7.15.0",
+    "pytest-cov >= 7.1.0",
     "rust-just >= 1.55.1",
 ]
 
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#4 lore raw

PATCH v4 4/6 add ruff linting

In reply to Trevor Woerner

Lint with ruff via just recipes. lint-tests runs ruff over tests/ and is held to a clean bar; lint also runs it over src/, whose findings are a preview since the wic source is not yet ruff-clean. ruff joins the tests extra and .gitignore ignores its .ruff_cache/.

ruff is pinned exactly so the gate is reproducible: a ruff minor release can add rules that turn a previously clean tree red, so every machine must lint with the same version.

Test modules relax E402 via per-file-ignores: they prepend src/ to sys.path before importing wic, which trips the rule, and that ordering is deliberate.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- expose linting through just recipes (lint-tests, lint) rather than run-tests.sh flags.
- pin ruff exactly (was a stale >= floor) so the clean-bar gate is reproducible across machines.
- fold in the tests/** E402 ignore (its own patch in v2 and v3) and drop the linting.md doc; the pyproject.toml comment explains it.
changes in v3:
- no change in this revision.
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.

---
 .gitignore     | 1 +
 justfile       | 9 +++++++++
 pyproject.toml | 8 ++++++++
 3 files changed, 18 insertions(+)
.gitignore +1 -0
index e30cafa40f7f..f2fd2f56b6d7 100644
@@ -2,3 +2,4 @@
 .pytest_cache/
 .coverage
 htmlcov/
+.ruff_cache/
justfile +9 -0
index 1be3e01e7c7d..5ccd8144181d 100644
@@ -16,3 +16,12 @@ coverage *args:
 # Run the suite with an HTML coverage report (written to htmlcov/).
 coverage-html *args:
     pytest {{args}} --cov=wic --cov-branch --cov-report=term-missing --cov-report=html
+
+# Lint the test suite; it is held to a clean bar, so this must be clean.
+lint-tests:
+    ruff check tests
+
+# Lint the test suite and then the wic source. src/ is not yet
+# ruff-clean, so its findings are a preview, not a gate.
+lint: lint-tests
+    ruff check src
pyproject.toml +8 -0
index 2ad694b13358..569944ff468c 100644
@@ -26,6 +26,7 @@ tests = [
     "pytest >= 9.1.1",
     "coverage >= 7.15.0",
     "pytest-cov >= 7.1.0",
+    "ruff == 0.15.20",
     "rust-just >= 1.55.1",
 ]
 
@@ -52,3 +53,10 @@ testpaths = ["tests"]
 # leftover directories under the pytest base temp directory.
 tmp_path_retention_policy = "failed"
 tmp_path_retention_count  = 1
+
+[tool.ruff.lint.per-file-ignores]
+# Test modules prepend the in-tree src/ directory to sys.path before
+# importing wic, so the suite runs against a checkout that has not been
+# installed. That bootstrap runs before the wic imports, which trips
+# E402; the ordering is deliberate, so ignore E402 for the test tree.
+"tests/**" = ["E402"]
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#5 lore raw

PATCH v4 5/6 bb/utils: import errno so mkdirhier's OSError handler works

In reply to Trevor Woerner

mkdirhier() calls os.makedirs(directory, exist_ok=True) and, if that still raises OSError, re-checks the error before deciding what to do:

except OSError as e:
if e.errno != errno.EEXIST or not os.path.isdir(directory):
raise e

but the module never imports errno. The reference to errno.EEXIST is only evaluated when os.makedirs() actually raises, so the defect stays hidden on every ordinary call. The moment a real OSError occurs -- a parent component that is a file, a permission error, anything -- the handler itself raises NameError: name 'errno' is not defined, masking the original error with a misleading one.

Import errno so the handler behaves as intended: it re-raises every real error, while tolerating only an EEXIST whose path is in fact already a directory.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- new commit: the errno import fix now lands on its own, before the test that exercises it. In v2 and v3 it was folded into the test_bb_utils commit.

---
 src/wic/bb/utils.py | 1 +
 1 file changed, 1 insertion(+)
src/wic/bb/utils.py +1 -0
index 3750056ba563..0b40dd0c1d59 100644
@@ -1,6 +1,7 @@
 """
 Minimal subset of BitBake's bb.utils used by standalone wic.
 """
+import errno
 import os
 
 # from bitbake/lib/bb/utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#6 lore raw

PATCH v4 6/6 tests/unit/test_bb_utils: cover mkdirhier()

In reply to Trevor Woerner

Add unit tests for mkdirhier(), the one function in wic.bb.utils. They pin the behaviour wic relies on so it cannot regress: it creates missing directories, accepts a directory that already exists, rejects a path containing an unexpanded bitbake variable (${...}), and allows a name with a lone brace or a lone dollar, neither of which is the ${ marker.

Three tests cover the OSError handler that the errno import repairs: a path whose parent component is a regular file, a target that already exists as a file rather than a directory, and a directory that appears concurrently mid-call (a create race), which must be treated as success. Run against a wic.bb.utils that does not import errno, all three fail with NameError instead, so they catch that bug directly.

The tests replace the tests/unit/.gitkeep placeholder.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v4:
- reframe the tests to target mkdirhier's own behaviour: drop the cases that only re-exercised os.makedirs and the assertions that pinned implementation details.
- the errno fix moved to its own preceding commit; this is tests only.
changes in v3:
- switch the tests from tempfile.mkdtemp() to pytest's tmp_path fixture so each test's scratch directory is cleaned up instead of leaking under /tmp; no change to what is tested.
changes in v2:
- v1 recorded this bug with an xfail marker in one large commit; v2 drops the xfail, asserts the correct behaviour directly, and lands the one-line errno-import fix in this same commit so the test passes green.

---
 tests/unit/.gitkeep         |  0
 tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+)
 delete mode 100644 tests/unit/.gitkeep
 create mode 100644 tests/unit/test_bb_utils.py
tests/unit/.gitkeep deleted +0 -0
deleted file mode 100644
index e69de29bb2d1..000000000000
tests/unit/test_bb_utils.py new file +81 -0

81 lines not on this page · raw message

Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
Reviewed-by on the v4 series
#7 lore raw
In reply to Trevor Woerner
Quoted 12 lines · Trevor Woerner via lists.yoctoproject.org

wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.

- the suite skeleton (pyproject test extra, pytest config, layout,

    a justfile command runner, and a README Testing section);
  - a conftest that imports wic once up front (stopping the session with
    a clear message if that fails) and prints a session banner;
  - optional coverage and ruff linting, each exposed as just recipes;
  - a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError
    handler works, as its own commit;
Quoted 4 lines

- unit tests for mkdirhier() that pin its behaviour and catch that bug.

The suite is green at every commit and the test tree is ruff-clean.

All LGTM now!

Best regards,

Signature · 1 line
Paul Barker
#8 lore raw
In reply to Trevor Woerner

Patch series applied to wic, master branch.
Thanks Paul!

Quoted 82 lines · Trevor Woerner

wic has no test mechanism of its own; it relies on the oe-selftest from oe-core, which needs a full bitbake build to exercise even pure-Python logic. This series starts a small standalone suite that runs from a plain checkout, framework first, then the first fix, and its test.

- the suite skeleton (pyproject test extra, pytest config, layout,

    a justfile command runner, and a README Testing section);
  - a conftest that imports wic once up front (stopping the session with
    a clear message if that fails) and prints a session banner;
  - optional coverage and ruff linting, each exposed as just recipes;
  - a one-line fix to wic.bb.utils (import errno) so mkdirhier's OSError
    handler works, as its own commit;

- unit tests for mkdirhier() that pin its behaviour and catch that bug.

The suite is green at every commit and the test tree is ruff-clean.

Changes in v4 (in response to review feedback, thanks to Paul Barker):

v4 restructures the v3 10-patch series into 6 patches. How the old patches map to the new ones:

- v3 1/10 (skeleton) -> v4 1/6, which also gains a justfile command runner in place of the run-tests.sh wrapper and no longer creates a tests/docs/ tree;
- v3 2/10 (conftest banner) -> v4 2/6, which also absorbs the

    wic-import check (see next line);
  - v3 3/10 (run-tests.sh wrapper) -> dropped; the justfile replaces it
    and its wic-import check moves into conftest, so the check runs on
    every invocation rather than only through the wrapper;
  - v3 4/10 (coverage) -> v4 3/6, exposed as just recipes;
  - v3 5/10 (ruff linting) -> v4 4/6, exposed as just recipes, with
    ruff now pinned exactly so the clean-bar gate is reproducible;
  - v3 8/10 (tests/** E402 ignore) -> folded into v4 4/6;
  - v3 6/10, 7/10, 10/10 (docs README, authoring guide, review rubric)
    -> dropped; the suite ships a short README Testing section instead
    of a tests/docs/ tree;

- v3 9/10 (test plus folded-in errno fix) -> split into v4 5/6 (the errno fix, on its own, first) and v4 6/6 (the tests), and the tests are reframed to target mkdirhier's own behaviour rather than re-testing os.makedirs.

Dependency versions were also refreshed to current releases.

Changes in v3:

- test_bb_utils used the tmp_path fixture instead of tempfile.mkdtemp(), so the tests no longer leaked scratch directories under /tmp;
- documented the standard pytest scratch-directory controls (TMPDIR

    and --basetemp) rather than a custom variable;

- added a review-rubric doc.

Changes in v2:

- v1 was a single ~6000-line commit that used xfail markers to record the bugs it found; v2 broke the work into a reviewable series where each test lands green next to the fix that makes it pass, with no xfails.

Trevor Woerner (6):
tests: add the standalone unit-test suite skeleton
tests: add conftest with a wic-import preflight and session banner
tests: add optional coverage reporting
add ruff linting
bb/utils: import errno so mkdirhier's OSError handler works tests/unit/test_bb_utils: cover mkdirhier()

 .gitignore                  |  4 ++
 README.md                   | 15 +++++++
 justfile                    | 27 +++++++++++++
 pyproject.toml              | 24 +++++++++++
 src/wic/bb/utils.py         |  1 +
 tests/conftest.py           | 28 +++++++++++++
 tests/unit/test_bb_utils.py | 81 +++++++++++++++++++++++++++++++++++++
 7 files changed, 180 insertions(+)
 create mode 100644 justfile
 create mode 100644 tests/conftest.py
 create mode 100644 tests/unit/test_bb_utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a