Kleidos QA — pytest system#

One industry-standard pytest system is the single entry-point for all Kleidos QA: it owns the hardware-in-the-loop (HIL) tests AND thinly wraps the native Unity C++ suite (pio test -e native), so one command runs everything. From there you either run it all or point at one scenario, with configurable DUT/host selection, live logging, standard pytest console output, and an Allure dashboard with per-test evidence.

qa/ (this tree, committed) is not test/ (the native C++/Unity host unit tests, which stay and are wrapped here).

Install#

cd qa
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt -c constraints.txt

The Allure dashboard additionally needs the Allure CLI (a Java tool):

brew install allure          # needs a JRE

Results generation (writing allure-results/) works without it; only rendering the HTML dashboard needs the CLI.

Run#

# Everything runnable given whatever hardware is connected (auto-skips the rest).
pytest

# No-hardware mode (mock transport + native wrapper) — CI/dev safe, no devices.
pytest -m "not needs_hardware"

# Fast gate: skip the slow native compile too.
pytest -m "not needs_hardware and not slow"

# A single scenario against a specific DUT + host.
pytest tests/ble/test_ble_pairing.py::test_happy_path --dut sticks3 --host mastodon --run-hardware

# Smoke across every connected DUT (capability mismatches auto-skip).
pytest -m smoke --dut all --run-hardware

Hardware tests are skipped by default; pass --run-hardware to enable them. DUTs are selected by name from inventory/devices.yaml (identity is the MAC, never the transient /dev/cu.usbmodem* node); hosts from inventory/hosts.yaml.

Flash a device#

Two workflows, depending on how many boards are connected:

# Single board — PlatformIO auto-detects the port (no setup needed):
pio run -e sticks3_debug -t upload

# Multi-board bench — resolve port by MAC before flashing:
python qa/scripts/flash.py sticks3

# Override env or skip the post-flash boot-marker check:
python qa/scripts/flash.py sticks3 --env sticks3 --no-verify-marker

python qa/scripts/flash.py --help

flash.py is a wrapper around qa_lib.dut.flash.flash_debug: it reads the device name from inventory/devices.yaml, resolves the current serial port by MAC (via identify.resolve_verified_port), confirms the MAC, and calls pio run -e <env>_debug -t upload --upload-port <port>. _secure envs are refused. Device MACs live in the gitignored inventory/devices.local.yaml.

Configure#

Copy the template and fill in your lab coordinates (gitignored):

cp inventory/.env.example inventory/.env && $EDITOR inventory/.env

Committed files carry variable names and placeholders only — never IPs, hosts, ports, users, MACs, interfaces, or secrets.

Dashboard#

# Live (no history/trends):
pytest -m "not needs_hardware" --alluredir .artifacts/allure-results
allure serve .artifacts/allure-results

# Static, shareable HTML with accumulating history/trends:
scripts/allure_report.sh

scripts/allure_report.sh carries the previous report's history/ dir into the new results before allure generate --clean, so trends accumulate across runs (Allure 2 requires this; Allure 3 removes the manual step — a future upgrade).

Layout#

qa_lib/     the reusable library (transport, DUT/host control, redaction, matchers)
tests/      thin scenarios: arrange (fixtures) -> act (qa_lib) -> assert (matchers)
inventory/  devices.yaml (by MAC), hosts.yaml, .env.example
fixtures/   non-secret test credential / vault-state data
scripts/    allure_report.sh and other QA-local helpers

Non-negotiables (enforced in every test/helper)#

• MAC is the only DUT identity — verified before flashing/driving; abort on mismatch.
• No secrets in output — one central redaction filter (qa_lib/redaction.py), guarded by tests/qa_lib/test_redaction.py; covers logs, console, Allure attachments.
• Port-stuck ≠ disconnected — on open failure, kill stale holders + USB-reset, verify by MAC.
• Debug-build gating — serial debug commands only on *_debug builds; skip otherwise.
• Never touch _secure builds.
• .env privacy — no lab coordinates in committed files.