tests: standalone test-suite framework plus the first unit test
v2 cover tests: standalone test-suite framework plus the first unit test
v1 sent the whole test suite as a single 6000-line commit that leaned on xfail markers to record the bugs it found. Review feedback was that this is not reviewable: the request was for a series of small commits where each test lands green next to the source fix that makes it pass, with no xfails and no failing tests at any commit boundary.
This v2 is that rework. Rather than resend the entire suite at once, it establishes the test framework and a single representative test so the structure and conventions can be reviewed first. Later functions will follow as their own per-function commits once this base is agreed.
The series contains:
- the suite skeleton (pyproject test config, layout, .gitignore,
README Testing section);
- the conftest session banner;
- the run-tests.sh wrapper, with optional coverage and ruff linting
added in their own commits;
- the suite docs (overview README and authoring guide) and the one
ruff per-file-ignore the sys.path bootstrap needs; - the first unit test, tests/unit/test_bb_utils, which tests mkdirhier() and carries the one-line errno-import fix the test exposes, so the suite is green.
The whole suite is green at every commit (zero failures, zero xfails) and tests/run-tests.sh --lint-tests is clean throughout.
Trevor Woerner (9):
tests: add the standalone test-suite skeleton
tests: add a session banner via conftest.py
tests: add the run-tests.sh wrapper
tests: add optional coverage reporting to run-tests.sh
tests: add ruff linting to run-tests.sh
tests/docs: add the suite overview README
tests/docs: add the test-authoring guide
tests: ignore E402 in the test tree for the sys.path bootstrap tests/unit/test_bb_utils: test mkdirhier() and fix its missing errno
import
.gitignore | 10 +++ README.md | 15 ++++ pyproject.toml | 30 +++++++ src/wic/bb/utils.py | 1 + tests/conftest.py | 38 +++++++++ tests/docs/README.md | 61 ++++++++++++++ tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++ tests/docs/linting.md | 59 +++++++++++++ tests/run-tests.sh | 159 ++++++++++++++++++++++++++++++++++++ tests/unit/.gitkeep | 0 tests/unit/test_bb_utils.py | 126 ++++++++++++++++++++++++++++ 11 files changed, 623 insertions(+) create mode 100644 tests/conftest.py create mode 100644 tests/docs/README.md create mode 100644 tests/docs/authoring.md create mode 100644 tests/docs/linting.md create mode 100755 tests/run-tests.sh create mode 100644 tests/unit/.gitkeep create mode 100644 tests/unit/test_bb_utils.py
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 1/9 tests: add the standalone test-suite skeleton
wic currently has no test mechanism of its own; it relies on the oe-selftest from oe-core for all its testing, which means a full bitbake build is needed to exercise even pure-Python logic. This commit lays the groundwork for a small standalone suite that runs from a plain checkout with nothing but pytest, so that logic can be pinned down and kept stable as the code evolves.
This commit adds the skeleton only; it does not add any tests.
What this adds:
- pyproject.toml: a "tests" optional-dependency group (pytest only) so that "pip install -e .[tests]" pulls in what the suite needs, plus a [tool.pytest.ini_options] section. The pytest options keep a test's scratch directory only when that test fails, so repeated runs do not accumulate leftover files.
- tests/unit/ and tests/docs/: the directories that hold the in-process unit suites and the suite documentation. Git cannot track an empty directory, so a placeholder .gitkeep is committed in each to create it. The .gitkeep files have no content and are removed once real files land in those directories.
- .gitignore: ignore the .pytest_cache/ directory that pytest writes at the top of the checkout.
- README.md: a Testing section pointing at the suite and at tests/docs/, and a tests/* entry in the project layout list.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 3 +++ README.md | 14 ++++++++++++++ pyproject.toml | 12 ++++++++++++ tests/docs/.gitkeep | 0 tests/unit/.gitkeep | 0 5 files changed, 29 insertions(+) create mode 100644 tests/docs/.gitkeep create mode 100644 tests/unit/.gitkeep
.gitignore +3 -0
@@ -1 +1,4 @@ **/__pycache__ + +# pytest cache +/.pytest_cache/
README.md +14 -0
@@ -46,6 +46,20 @@ environment file or folder (generated via `bitbake -c rootfs_wicenv - `src/wic/*`: core engine, plugins, and helpers. - `src/bb/*`: various bitbake helpers that were brought along and used in other parts of wic. - `src/oe/*`: various oe-core helpers that were brought along and used in other parts of wic. +- `tests/*`: the standalone test suite and its documentation (see Testing below). + +## Testing + +wic ships a standalone test suite under `tests/` that runs from a plain +checkout, with no bitbake and no OpenEmbedded build required. The test +extras pull in everything the suite needs: + +``` +pip install -e ".[tests]" +``` + +See [tests/docs/](tests/docs/) for how to run the suite and the +conventions it follows. ## Contributing
pyproject.toml +12 -0
@@ -21,6 +21,11 @@ classifiers = [ Homepage = "https://git.yoctoproject.org/wic" Repository = "https://git.yoctoproject.org/wic" +[project.optional-dependencies] +tests = [ + "pytest >= 7.0", +] + [project.scripts] wic = "wic.cli:main"
@@ -36,3 +41,10 @@ build-backend = "hatchling.build" [tool.hatch.version] path = "src/wic/cli.py" + +[tool.pytest.ini_options] +# Keep a test's scratch directory only when it fails; a passing test's +# tmp_path is removed automatically so repeated runs do not accumulate +# leftover files under the pytest base temp directory. +tmp_path_retention_policy = "failed" +tmp_path_retention_count = 1
tests/docs/.gitkeep new file +0 -0
tests/unit/.gitkeep new file +0 -0
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 2/9 tests: add a session banner via conftest.py
In reply to Trevor WoernerWhen a test run starts it is useful to see, at a glance, exactly what is being tested: which host the run is on, which Python and pytest are in use, and -- most importantly -- which wic the suite imported and what version it reports. Without that, a passing or failing run is hard to attribute, especially when more than one wic checkout or virtualenv is in play.
pytest looks for a file named conftest.py and, among other things, calls its pytest_report_header() hook to print extra lines in the session header before collection begins. This commit adds that file with a single hook that prints a short banner:
- the host node name, operating system, and machine type;
- the running Python version and the pytest version;
- the version wic reports and the filesystem path the wic module was imported from.
The wic lookup is defensive: if wic cannot be imported (for example the test extras were not installed, or the suite is being run outside the checkout) the banner says so rather than aborting the run, so the failure is visible in the header instead of as an opaque collection error.
The file name conftest.py is mandatory; pytest discovers it by name, so it cannot be called anything else.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- tests/conftest.py | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 tests/conftest.py
tests/conftest.py new file +38 -0
@@ -0,0 +1,38 @@ +# Session banner for the wic test suite. +# +# Before collection, print the environment the tests exercise: +# - the host +# - the Python and pytest versions +# - the wic under test (its version and the module path of its import) + +import platform + +import pytest + + +def _wic_under_test(): + """Return (version, module_path) for the wic being tested.""" + try: + import wic.cli as wic_cli + except Exception as exc: # pragma: no cover - reported in the banner + return ("(import failed: %s)" % exc, "(unimported)") + version = getattr(wic_cli, "__version__", "(unknown)") + module_path = getattr(wic_cli, "__file__", "(unknown)") + return (version, module_path) + + +def _format_banner(): + wic_version, wic_path = _wic_under_test() + lines = [ + "wic test suite", + " host: %s %s %s" % ( + platform.node(), platform.system(), platform.machine()), + " python: %s pytest: %s" % ( + platform.python_version(), pytest.__version__), + " wic: %s (%s)" % (wic_version, wic_path), + ] + return "\n".join(lines) + + +def pytest_report_header(config): + return _format_banner()
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 3/9 tests: add the run-tests.sh wrapper
In reply to Trevor WoernerThe suite is run with pytest, but invoking pytest directly has two sharp edges: it must be run from the right directory for the pyproject configuration to apply, and it happily runs against whatever Python happens to be first on PATH, even one that cannot import wic. A run against the wrong interpreter produces a misleading banner and, worse, can silently exercise a different wic than the one in the checkout.
This commit adds tests/run-tests.sh, a thin wrapper that removes both hazards:
- it locates the repository root from its own path and changes there
before running, so it works regardless of the caller's directory;
- it checks that the interpreter can import wic before handing off to
pytest, and fails loudly with the install command if it cannot;
- with no arguments it runs the whole suite under tests/; any argument it does not recognise (a path, -k EXPR, -v, ...) is passed straight through to pytest, and an explicit -- forwards everything after it.
The interpreter can be overridden with the PYTHON environment variable (default python3). -h/--help prints usage and exits.
The README Testing section gains the run-tests.sh invocation alongside the install step.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- README.md | 1 + tests/run-tests.sh | 75 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 76 insertions(+) create mode 100755 tests/run-tests.sh
README.md +1 -0
@@ -56,6 +56,7 @@ extras pull in everything the suite needs: ``` pip install -e ".[tests]" +tests/run-tests.sh ``` See [tests/docs/](tests/docs/) for how to run the suite and the
tests/run-tests.sh new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 4/9 tests: add optional coverage reporting to run-tests.sh
In reply to Trevor WoernerKnowing which lines a test run actually exercised is the difference between "the suite is green" and "the suite is green and we know what it touched". This commit wires branch coverage of the wic source into the runner, kept entirely opt-in so a plain run stays fast and quiet.
pyproject.toml gains coverage and pytest-cov in the tests extra, so "pip install -e .[tests]" pulls in what the new flags need.
run-tests.sh gains two options:
- --coverage measures branch coverage of src/wic during the run and
prints a terminal report listing the lines that were missed;
- --html [DIR] additionally writes a browsable HTML report (default htmlcov/, or DIR if given) and implies --coverage.
If --coverage is requested but pytest-cov is not installed the runner fails loudly with the install command rather than running without the measurement it was asked for.
The coverage data file and the HTML report directory are build artifacts, so .gitignore learns to ignore .coverage and htmlcov/.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 4 ++++ pyproject.toml | 2 ++ tests/run-tests.sh | 39 ++++++++++++++++++++++++++++++++++++++- 3 files changed, 44 insertions(+), 1 deletion(-)
.gitignore +4 -0
@@ -2,3 +2,7 @@ # pytest cache /.pytest_cache/ + +# coverage data and reports +/.coverage +/htmlcov/
pyproject.toml +2 -0
@@ -24,6 +24,8 @@ Repository = "https://git.yoctoproject.org/wic" [project.optional-dependencies] tests = [ "pytest >= 7.0", + "coverage >= 7.0", + "pytest-cov >= 4.0", ] [project.scripts]
tests/run-tests.sh +38 -1
@@ -13,9 +13,13 @@ set -euo pipefail usage() { cat <<'USAGE' Usage: - tests/run-tests.sh [pytest args] + tests/run-tests.sh [--coverage] [--html [DIR]] [pytest args] Options: + --coverage also measure branch coverage of src/wic and print a + terminal report listing the lines that were missed + --html [DIR] also write an HTML coverage report (default dir: + htmlcov/); implies --coverage -h, --help show this help and exit Anything else is passed straight through to pytest (for example a
@@ -24,6 +28,9 @@ tests/ is run. Examples: tests/run-tests.sh # whole suite + tests/run-tests.sh --coverage # + terminal coverage report + tests/run-tests.sh --html # + HTML report in htmlcov/ + tests/run-tests.sh --html /tmp/cov # + HTML report in /tmp/cov tests/run-tests.sh -k filemap -v # pass args through to pytest tests/run-tests.sh tests/unit # a single tier or file
@@ -34,9 +41,25 @@ USAGE REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" PY="${PYTHON:-python3}" +coverage=0 +html=0 +html_dir="htmlcov" pytest_args=() while [ $# -gt 0 ]; do case "$1" in + --coverage) + coverage=1 + ;; + --html) + coverage=1 + html=1 + # Optional directory argument: consume the next token only if + # it is not another option or the pytest pass-through marker. + case "${2:-}" in + ""|-*|--) ;; + *) html_dir="$2"; shift ;; + esac + ;; -h|--help) usage exit 0
@@ -72,4 +95,18 @@ if [ ${#pytest_args[@]} -eq 0 ]; then pytest_args=("tests") fi +if [ "$coverage" -eq 1 ]; then + if ! "$PY" -c "import pytest_cov" >/dev/null 2>&1; then + echo "error: coverage requested but pytest-cov is not installed." >&2 + echo " run: $PY -m pip install -e \".[tests]\"" >&2 + exit 1 + fi + cov_args=(--cov=wic --cov-branch --cov-report=term-missing) + if [ "$html" -eq 1 ]; then + cov_args+=(--cov-report="html:${html_dir}") + echo "HTML coverage report: ${html_dir}/index.html" >&2 + fi + exec "$PY" -m pytest "${pytest_args[@]}" "${cov_args[@]}" +fi + exec "$PY" -m pytest "${pytest_args[@]}"
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 5/9 tests: add ruff linting to run-tests.sh
In reply to Trevor WoernerA test suite is only trustworthy if its own code is clean, so this commit brings ruff into the runner and holds the test tree to a clean bar. It also makes it easy to preview what ruff thinks of the wic source, without yet enforcing it.
pyproject.toml gains ruff in the tests extra and a [tool.ruff] section. The configuration is deliberately minimal for now: the test suite is the only tree under an enforced clean bar; the wic source under src/ is not yet ruff-clean and is reported, not gated.
run-tests.sh gains two lint modes, each used on its own:
- --lint-tests runs ruff over tests/ and exits. The test suite must report nothing; a finding here is a bug in our own test code and is expected to be fixed.
- --lint-src runs ruff over src/ and exits. The source is not yet ruff-clean, so this is a preview: the runner prints ruff's findings and exits with its status, but nothing in the suite asserts on them. Keeping the two trees on separate flags means cleaning up the source later does not disturb the test-tree gate.
A lint mode cannot be combined with coverage, with the other lint mode, or with pytest arguments; the runner rejects such combinations loudly rather than silently dropping the extras. If ruff is not installed it fails with the install command.
tests/docs/linting.md documents the two modes and why src/ is held back for now. That file replaces the tests/docs/.gitkeep placeholder, which is no longer needed now that the directory has real content.
.gitignore learns to ignore ruff's .ruff_cache/ directory.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- .gitignore | 3 +++ pyproject.toml | 8 ++++++++ tests/docs/.gitkeep | 0 tests/docs/linting.md | 39 +++++++++++++++++++++++++++++++++++ tests/run-tests.sh | 47 +++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 97 insertions(+) delete mode 100644 tests/docs/.gitkeep create mode 100644 tests/docs/linting.md
.gitignore +3 -0
@@ -6,3 +6,6 @@ # coverage data and reports /.coverage /htmlcov/ + +# ruff cache +/.ruff_cache/
pyproject.toml +8 -0
@@ -26,6 +26,7 @@ tests = [ "pytest >= 7.0", "coverage >= 7.0", "pytest-cov >= 4.0", + "ruff >= 0.5", ] [project.scripts]
@@ -50,3 +51,10 @@ path = "src/wic/cli.py" # leftover files under the pytest base temp directory. tmp_path_retention_policy = "failed" tmp_path_retention_count = 1 + +[tool.ruff] +# For now only the test suite is actually linted (run-tests.sh +# --lint-tests passes the tests/ path); the wic source under src/ is +# not yet ruff-clean and is left out until its findings are fixed (see +# tests/docs/linting.md). --lint-src can still be run to preview the +# source findings, but it is reported, not enforced.
tests/docs/.gitkeep deleted +0 -0
tests/docs/linting.md new file +39 -0
@@ -0,0 +1,39 @@ +# Linting + +## Contents + +- [Running the linter](#running-the-linter) +- [tests/ must be clean](#tests-must-be-clean) +- [src/ is not linted yet](#src-is-not-linted-yet) + +The test suite is linted with [ruff](https://docs.astral.sh/ruff/). It +is configured in `pyproject.toml` (`[tool.ruff]`). + +## Running the linter + +The runner exposes ruff through two separate modes, each used on its +own: + +```bash +tests/run-tests.sh --lint-tests # ruff over tests/ +tests/run-tests.sh --lint-src # ruff over src/ (preview only) +``` + +A lint mode cannot be combined with coverage, with the other lint +mode, or with pytest arguments; the runner rejects such combinations. + +## tests/ must be clean + +Our own test code is held to a clean bar: `tests/run-tests.sh +--lint-tests` reports nothing. If you add a test that trips a rule, fix +the test before the change lands. + +## src/ is not linted yet + +`--lint-src` runs ruff over the wic source, but the source is **not** +yet ruff-clean, so its findings are a preview report rather than a +gate: the runner prints them and exits with ruff's status, but nothing +in the suite asserts on them. Treating `src/` findings as a hard +failure now would block every run on fixes that have not landed. Once +the source is cleaned up, `src/` can be promoted to the same clean bar +as `tests/`.
tests/run-tests.sh +47 -0
88 lines not on this page · raw message
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 6/9 tests/docs: add the suite overview README
In reply to Trevor WoernerThe test suite needs a front door: a short document that says what the suite is, how it is laid out, and how to run it. This commit adds tests/docs/README.md to be that overview.
The README covers:
- the layout of tests/, file by file;
- that the suite is unit-only and needs no host tools or build
environment, so it runs from a plain checkout;
- how to install the test extras and invoke run-tests.sh, pointing at
run-tests.sh --help as the authoritative, always-current list of
options rather than duplicating them here where they would drift;
- that anything run-tests.sh does not recognise is passed through to
pytest; - a pointer to linting.md for how ruff is applied;
- a small table of the other documents under tests/docs/.
It deliberately does not enumerate every run-tests.sh option, since the runner is expected to grow more of them; the table and the --help output stay the source of truth.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- tests/docs/README.md | 61 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 tests/docs/README.md
tests/docs/README.md new file +61 -0
@@ -0,0 +1,61 @@ +# wic test suite + +A standalone test suite for the wic source tree. It runs from a plain +checkout with nothing but `pytest` -- no bitbake, no OpenEmbedded +build, and no target image -- so wic's logic can be exercised and kept +stable as the code changes. + +## Contents + +- [Layout](#layout) +- [Running](#running) +- [Linting](#linting) +- [Documentation](#documentation) + +## Layout + +``` +tests/ + conftest.py session banner describing the run + run-tests.sh wrapper for running the suite + unit/ unit tests that import wic modules directly + docs/ this documentation +``` + +The suite is unit-only: every test under `tests/unit/` imports a wic +module in-process and asserts on its behaviour, so none of it needs +host tools or a build environment. + +## Running + +Install wic with its test extras, then run the suite: + +```bash +pip install -e ".[tests]" +tests/run-tests.sh +``` + +`run-tests.sh` works from anywhere in the checkout. It can also report +branch coverage of the wic source; run `tests/run-tests.sh --help` for +the current list of options. + +Anything `run-tests.sh` does not recognise is handed straight to +pytest, so the whole pytest command line is available: + +```bash +tests/run-tests.sh -k filemap -v # one area, verbose +tests/run-tests.sh tests/unit # a single tier or file +``` + +## Linting + +The test suite is linted with ruff and is held to a clean bar. See +[linting.md](linting.md) for the lint modes and how the source tree is +treated. + +## Documentation + +| File | Content | +|------|---------| +| [authoring.md](authoring.md) | How to add a unit test to the suite | +| [linting.md](linting.md) | How ruff is used on the suite |
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 7/9 tests/docs: add the test-authoring guide
In reply to Trevor WoernerA suite is only as good as the tests added to it over time, so it needs a guide that says where a new test goes, what one looks like, and how to choose its inputs and assertions. This commit adds tests/docs/authoring.md.
The guide covers:
- where a test goes: one test_<area>.py per wic module under
tests/unit/, extending an existing file or starting a new one;
- the shape of a test: a small worked example, a preference for parametrize over in-test loops so each input/output pair is its own named case, and use of the tmp_path fixture for scratch files;
- asserting the correct behaviour rather than whatever wic currently does, so a test never bakes in a bug; when a test exposes a defect, the source fix lands in the same change so the test passes;
- probing the boundaries: a checklist of input classes worth covering for numbers, strings, paths, types/shape, and state, each asserting a specific value or exception rather than merely "it did not crash";
- keeping the assertion strong: never weaken an assertion to get
green;
- how to run just the test you wrote, with and without coverage.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- tests/docs/authoring.md | 124 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 tests/docs/authoring.md
tests/docs/authoring.md new file +124 -0
@@ -0,0 +1,124 @@ +# Authoring a unit test + +Unit tests live under `tests/unit/`, one module per area of wic, named +`test_<area>.py`. They import the wic module under test directly and +run in-process, so they need no host tools and no build environment. + +## Contents + +- [Where a test goes](#where-a-test-goes) +- [Shape of a test](#shape-of-a-test) +- [Assert the correct behaviour](#assert-the-correct-behaviour) +- [Probe the boundaries](#probe-the-boundaries) +- [Keep the assertion strong](#keep-the-assertion-strong) +- [Running what you wrote](#running-what-you-wrote) + +## Where a test goes + +Group tests by the wic module they exercise, one `test_<area>.py` per +module. Start a new file when you begin covering a module that has no +file yet; otherwise extend the existing one. + +## Shape of a test + +```python +import pytest + +from wic.ksparser import sizetype + + +class TestSizetype: + # sizetype(default, size_in_bytes=False) returns a parser f(arg); + # with default "M", a bare number is read as mebibytes-in-KiB. + def test_plain_value_uses_default_unit(self): + parse = sizetype("M") + assert parse("100") == 100 * 1024 + + @pytest.mark.parametrize("arg,expected", [ + ("1K", 1), + ("1G", 1 * 1024 * 1024), + ("0", 0), + ]) + def test_suffixes(self, arg, expected): + assert sizetype("M")(arg) == expected +``` + +Prefer `@pytest.mark.parametrize` to express each input/output pair as +its own case rather than looping inside one test, so a single bad +value shows up as one named failure. + +Use `pytest`'s `tmp_path` fixture for any scratch files; a passing +test's scratch directory is removed automatically (see the retention +policy in `pyproject.toml`), and a failing one is kept for inspection. + +## Assert the correct behaviour + +Every assertion states the behaviour wic is *expected* to provide, +never the behaviour it currently happens to have. A test that locks in +a wrong result is worse than no test: it gives false confidence and it +turns red exactly when someone repairs the code. + +When a test exposes a defect, the fix to the wic source lands in the +same change as the test, so the test asserts the correct behaviour and +passes. Do not commit a test that asserts a known-wrong result. + +Wrong -- bakes in the bug: + +```python +# value lost its comma; asserting the broken output +assert result["file"] == "s3://bucket/a" +``` + +Right -- asserts the correct output (and the fix lands with it): + +```python +assert result["file"] == "s3://bucket/a,b.img" +``` + +## Probe the boundaries + +For a parameter, probe the boundary and just past it rather than only +the comfortable middle of its range. The classes worth covering: + +- numbers: empty, zero, negative, the maximum and one beyond it, + non-power-of-two and non-multiple-of-1024/1000 values, primes, + fractional values where an integer is assumed, `0` versus `0.0` + versus `"0"`, off-by-one at a block or sector boundary, and very + large magnitudes near `2**31` and `2**63` +- strings: the empty string, whitespace only, embedded spaces, tabs, + newlines and CRLF endings, non-ASCII characters, shell + metacharacters in a value that becomes part of a command line, + leading-dash values that look like options, over-delimited or + malformed forms (stray commas, doubled or missing separators), and + very long values +- paths: traversal (`../`), doubled slashes, a trailing slash or its + absence, `.` and `..` components, broken or looping symlinks, and a + non-existent path +- types and shape: each wrong type the parameter might receive (a + string where an integer is expected, a list where a string is + expected, `None`, and so on), the wrong argument count, and + duplicate keys or entries +- state: a second call that must not see stale cached data, a + `cache=False` path that must evict, calling an operation twice, and + finalising a half-constructed object + +Each case asserts a specific outcome -- an exact value, or a specific +exception -- rather than merely "it did not crash." A test that only +checks for the absence of an exception will pass against badly wrong +output. + +## Keep the assertion strong + +If a test is hard to make pass, the answer is in the code or in +understanding the correct behaviour, never in weakening the assertion +to something vague enough to pass. Loosening an assertion to get green +is how a suite quietly stops catching regressions. + +## Running what you wrote + +```bash +tests/run-tests.sh tests/unit/test_<area>.py -v +tests/run-tests.sh --coverage tests/unit/test_<area>.py +``` + +A new test should leave the suite green, with zero failures.
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 8/9 tests: ignore E402 in the test tree for the sys.path bootstrap
In reply to Trevor WoernerThe unit tests that follow run against a plain checkout that has not been installed. To do that, each test module prepends the in-tree src/ directory to sys.path and only then imports wic:
_SRC = Path(__file__).resolve().parent.parent.parent / "src" if str(_SRC) not in sys.path:
sys.path.insert(0, str(_SRC))
from wic.bb.utils import mkdirhier
That bootstrap necessarily runs before the wic imports, which trips ruff's E402 (module-level import not at top of file). The ordering is required, not accidental: the import would fail without the sys.path adjustment ahead of it.
Rather than scatter per-line noqa comments through every test module, this commit relaxes E402 for the test tree once, via a [tool.ruff.lint.per-file-ignores] entry scoped to tests/**. It is the only rule relaxed for tests/; the suite is otherwise held to a clean bar by --lint-tests.
tests/docs/linting.md gains a section describing this single exception and why it exists.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 submitted the entire test suite as a single commit; v2 breaks the work into a reviewable series, and this patch is one step of it.
--- pyproject.toml | 8 ++++++++ tests/docs/linting.md | 20 ++++++++++++++++++++ 2 files changed, 28 insertions(+)
pyproject.toml +8 -0
@@ -58,3 +58,11 @@ tmp_path_retention_count = 1 # not yet ruff-clean and is left out until its findings are fixed (see # tests/docs/linting.md). --lint-src can still be run to preview the # source findings, but it is reported, not enforced. + +[tool.ruff.lint.per-file-ignores] +# Test modules prepend the in-tree src/ directory to sys.path before +# importing wic, so the suite runs against a checkout that has not been +# installed. That bootstrap necessarily runs before the wic imports, +# which trips E402 (module-level import not at top of file). The +# ordering is deliberate; ignore E402 for the test tree only. +"tests/**" = ["E402"]
tests/docs/linting.md +20 -0
@@ -4,6 +4,7 @@ - [Running the linter](#running-the-linter) - [tests/ must be clean](#tests-must-be-clean) +- [The one intentional exception in tests/](#the-one-intentional-exception-in-tests) - [src/ is not linted yet](#src-is-not-linted-yet) The test suite is linted with [ruff](https://docs.astral.sh/ruff/). It
@@ -28,6 +29,25 @@ Our own test code is held to a clean bar: `tests/run-tests.sh --lint-tests` reports nothing. If you add a test that trips a rule, fix the test before the change lands. +## The one intentional exception in tests/ + +Test modules prepend the in-tree `src/` directory to `sys.path` before +importing `wic`, so the suite runs against a plain checkout that has not +been installed: + +```python +_SRC = Path(__file__).resolve().parent.parent.parent / "src" +if str(_SRC) not in sys.path: + sys.path.insert(0, str(_SRC)) + +from wic.bb.utils import mkdirhier +``` + +That bootstrap necessarily runs before the `wic` imports, which trips +`E402` (module-level import not at top of file). The ordering is +required, so `E402` is ignored for the test tree via `per-file-ignores` +in `pyproject.toml`. This is the only rule relaxed for `tests/`. + ## src/ is not linted yet `--lint-src` runs ruff over the wic source, but the source is **not**
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
PATCH v2 9/9 tests/unit/test_bb_utils: test mkdirhier() and fix its missing errno import
In reply to Trevor Woernermkdirhier() wraps os.makedirs() and, on OSError, re-checks the error to decide whether to swallow a benign "already exists" race or re-raise:
except OSError as e:
if e.errno != errno.EEXIST or not os.path.isdir(directory):
raise e
but the module never imports errno. The reference to errno.EEXIST is only evaluated when os.makedirs() actually raises, so the defect is invisible on the happy path and on the unexpanded-${} guard. The moment a real OSError occurs -- a parent component that is a file, a permission error, anything -- the handler itself raises NameError: name 'errno' is not defined, masking the original error with a misleading one.
The fix is a one-line import. With errno available the handler behaves as intended: an EEXIST on an existing directory is swallowed, while any other OSError (and an EEXIST whose path is not a directory) propagates unchanged.
This commit adds tests/unit/test_bb_utils.py, covering mkdirhier() end to end:
- the happy path (nested, single-level, idempotent, existing dir);
- the unexpanded-${} guard (rejected, nothing created, and that a
bare brace without a dollar is allowed);
- the OSError handler: a real mkdir under a file component surfaces an OSError, an arbitrary OSError propagates with its errno intact, an EEXIST on an existing directory is swallowed, and an EEXIST on a regular file propagates.
The error-path tests fail with NameError without the import and pass with it, so the test and the fix belong together in this one change.
AI-Generated: codex/claude-opus 4.7 (xhigh)
---
changes in v2:
- v1 recorded this bug with an xfail marker in one large commit; v2 drops the xfail, asserts the correct behaviour directly, and lands the one-line errno-import fix in this same commit so the test passes green.
--- src/wic/bb/utils.py | 1 + tests/unit/test_bb_utils.py | 126 ++++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+) create mode 100644 tests/unit/test_bb_utils.py
src/wic/bb/utils.py +1 -0
@@ -1,6 +1,7 @@ """ Minimal subset of BitBake's bb.utils used by standalone wic. """ +import errno import os # from bitbake/lib/bb/utils.py
tests/unit/test_bb_utils.py new file +126 -0
126 lines not on this page · raw message
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a