kleidos/qa/tests/hardware/ at main · jmrp.io/kleidos · Tangled

jmrp.io / kleidos

0

[READ-ONLY] Mirror of https://github.com/jmrplens/kleidos. Kleidos — Hardware BLE password manager for ESP32-S3 (M5StickC Plus2, StickS3, M5Stack Gray, T-Deck, Cardputer) jmrplens.github.io/kleidos/

0

kleidos / qa / tests / hardware /

at main 17 files

José M. Requena Plens Formalize trackball QA: TRKRW parser + hardware suite 5w ago

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_deep_sleep.py

Add LOCKOUT?/LOCKOUTSET/LOCKOUTFAIL + SLEEPTEST debug commands + QA tests Enable two previously-impossible QA areas: brute-force-lockout testing and deep-sleep/wake testing. Read + modify commands, DEBUG_SERIAL_BUTTONS-gated. Firmware: - brute_force_guard: read-only shortLockThreshold()/wipeThreshold() getters + debug-only debugSetAttempts() (shared owner) — and fixed a latent WIPEVAULT zeroize using sizeof(pointer) instead of the 32-byte key array. - LOCKOUT? (fail_count/locked/lockout_threshold=4/wipe_threshold=10/remaining_ms), LOCKOUTSET <n> (clamped to wipe-1, never wipes on its own), LOCKOUTFAIL (one real registerFailure — increments + threshold eval + wipes only at 10). - SLEEPTEST <ms> (500..30000): a real timer-wake deep sleep even on debug builds (via a new sleep::enableTimerWakeup facade + AppContext::debugTimerSleep that reuses the security teardown but calls deepSleepStart directly so the timer fires on every variant). Builds sticks3_debug + gray_debug + release + native (1168/1168) clean. QA: DebugConsole lockout/lockout_set/lockout_fail/sleeptest; tests/hardware/ test_lockout.py (thresholds + LOCKOUTSET reflection + LOCKOUTFAIL counter, all capped far below wipe; timed-lockout engagement gated to `rtc`), test_deep_sleep.py (SLEEPTEST ack + recovery — wake=TIMER can't be auto-asserted since opening the CDC port issues USB_UART_CHIP_RESET which clobbers the wake cause), test_vault_ops.py (WIPEVAULT->provision->ADDCRED CRUD, destructive/opt-in). Findings: the debug PIN command intentionally bypasses the guard (so QA can send wrong PINs safely) — LOCKOUTFAIL is the real-failure seam. RTC-less boards (Stick) have no timed lockout (only the count-based wipe at 10); the timed lockout engages on RTC boards. Validated: Stick S3 3 passed/1 rtc-skip + deep-sleep + vault CRUD; cores3_se (RTC) timed-lockout engagement passed. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_diagnostics.py

Add 11 system-monitor debug commands (NVS/heap/clock/chip/rng/temp/secboot/audio/part/wdt/font) Maximize observability — each command monitors a subsystem, so a QA test failing here flags a firmware regression to review. All read-only, DEBUG_SERIAL_BUTTONS- gated, KEY=value; every raw esp_*/heap_caps_*/nvs_*/partition/chip/secure-boot/ temp call is wrapped in a small src/platform/ facade (abstraction-first). Firmware: 11 commands (NVS?/HEAP?/CLOCK?/CHIP?/RNG?/TEMP?/SECBOOT?/AUDIO?/PART?/ WDT?/FONT?) + platform facades (system heap/clock/partitions, security_fuses, nvs stats, temperature [new, SOC-gated], watchdog, speaker codec name, text renderer atlas stats). Both sticks3_debug (S3) + m5stack_gray_debug (classic ESP32) -Werror clean; native 1168/1168. QA: console methods + tests/hardware/test_diagnostics.py (one monitor per command, incl. a secure-boot-off safety assertion + running-partition check); diagnostics marker. identify.confirm_mac now tolerates an unreadable MAC on a bridge board (retry then proceed on the unique USB-bridge serial; abort only on a real mismatch) — fixes Gray/CP210x --flash. Validated: Stick S3 11 passed, Gray classic ESP32 11 passed. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_display.py

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test(qa): formalize the IMU comms + read/write self-test into hardware QA Add the IMU self-test to the pytest hardware suite so every IMU board is regression-gated on explicit comms + WHO_AM_I READ + register WRITE-readback + gravity data-path, not ad-hoc serial checks. qa_lib gains an ImuSelfTest parser + DebugConsole.imu_selftest() (drives the new IMURW command); test_imu.py gains test_imu_selftest (imu-gated, asserts the write value was read back + chip/family oracle + gravity window) and test_imu_absent_graceful (no-IMU boards report absent with no false detection); a no-hardware parser unit test pins the parsing. devices.yaml flips cardputer + core2_v13 to imu:true (both carry a BMI270), so the imu-gated tests fan out to them under --dut all. HIL: all 6 IMU boards PASS (cardputer@0x69, core2_v13, plus1/plus2/gray, sticks3 pending a PWR tap); the 3 no-IMU boards skip cleanly + pass the absent check.

1 month ago

test_input_introspection.py

Add KBD?/TOUCH?/TRACKBALL?/PMIC? input+PMIC introspection commands + QA tests Close the input-controller serial-exposure gap: the console could INJECT keyboard/touch/trackball events (KEY/DRAG/BALL) but not READ the controller state. All read-only, DEBUG_SERIAL_BUTTONS-gated, KEY=value, per-variant guarded. Firmware getters promoted to base interfaces (non-destructive; never consume input): KBD? (available/buffered/model), TOUCH? (available/model/w/h/touched), TRACKBALL? (available/click via isCenterPressed, not the resetting poll()), PMIC? (model/batt_pct/batt_mv/charging; AXP2101 boards report model, ADC/M5PM1 none). Builds sticks3_debug + tdeck_debug -Werror clean; native still 1168/1168. QA: DebugConsole kbd_info/touch_info/trackball_info/pmic; test_input_introspection.py cross-checks firmware KBD?/TOUCH?/TRACKBALL? detection vs inventory on every board; controller-detected tests in keyboard/touch/trackball; PMIC in test_power. Validated: Stick S3 4 passed (input available=0 = inventory, pmic=none), T-Deck 14 passed (keyboard/touch/trackball detected + cross-check). Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_keyboard.py

test(qa): formalize the keyboard comms + self-test into hardware QA Extend the pytest QA the same way the IMU self-test was formalized: - A dut.kbd_selftest() helper plus a KbdSelfTest dataclass and parse_kbd_selftest parser in qa_lib/dut/debug_console.py (mirrors imu_selftest + parse_imu_selftest). - A no-hardware parser unit test (test_kbd_selftest_parse.py) pinning the TCA8418 and LilyGo detail decoding over a MockDut. - A keyboard-capability-gated test_kbd_selftest + test_kbd_snapshot_alive plus a test_kbd_absent_graceful for the no-keyboard fleet (test_keyboard.py). Claude-Session: https://claude.ai/code/session_01SV58JXhfxhhc9DC6vdjBo4

1 month ago

test_lockout.py

Add LOCKOUT?/LOCKOUTSET/LOCKOUTFAIL + SLEEPTEST debug commands + QA tests Enable two previously-impossible QA areas: brute-force-lockout testing and deep-sleep/wake testing. Read + modify commands, DEBUG_SERIAL_BUTTONS-gated. Firmware: - brute_force_guard: read-only shortLockThreshold()/wipeThreshold() getters + debug-only debugSetAttempts() (shared owner) — and fixed a latent WIPEVAULT zeroize using sizeof(pointer) instead of the 32-byte key array. - LOCKOUT? (fail_count/locked/lockout_threshold=4/wipe_threshold=10/remaining_ms), LOCKOUTSET <n> (clamped to wipe-1, never wipes on its own), LOCKOUTFAIL (one real registerFailure — increments + threshold eval + wipes only at 10). - SLEEPTEST <ms> (500..30000): a real timer-wake deep sleep even on debug builds (via a new sleep::enableTimerWakeup facade + AppContext::debugTimerSleep that reuses the security teardown but calls deepSleepStart directly so the timer fires on every variant). Builds sticks3_debug + gray_debug + release + native (1168/1168) clean. QA: DebugConsole lockout/lockout_set/lockout_fail/sleeptest; tests/hardware/ test_lockout.py (thresholds + LOCKOUTSET reflection + LOCKOUTFAIL counter, all capped far below wipe; timed-lockout engagement gated to `rtc`), test_deep_sleep.py (SLEEPTEST ack + recovery — wake=TIMER can't be auto-asserted since opening the CDC port issues USB_UART_CHIP_RESET which clobbers the wake cause), test_vault_ops.py (WIPEVAULT->provision->ADDCRED CRUD, destructive/opt-in). Findings: the debug PIN command intentionally bypasses the guard (so QA can send wrong PINs safely) — LOCKOUTFAIL is the real-failure seam. RTC-less boards (Stick) have no timed lockout (only the count-based wipe at 10); the timed lockout engages on RTC boards. Validated: Stick S3 3 passed/1 rtc-skip + deep-sleep + vault CRUD; cores3_se (RTC) timed-lockout engagement passed. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_no_serial_in_release.py

Parametrize the release serial-console guard over the whole fleet (P0-06) test_no_serial_in_release was sticks3-only; every other <dut>_release build was unverified. Parametrize over the fleet (sticks3, plus1, plus2, gray, core2_v13, cores3_se, tdeck, cardputer, m5core_ink), each case resolving its DUT by MAC and skipping cleanly when absent, so a partial bench still covers the boards present. Validated end-to-end on sticks3_release (flash + no-banner + no-command-response). Also record the Tier-1 audit re-verification: the residual WiFi findings (LIFEC-008, OTA-004, OPT-003) are already remediated in main (dossier was stale), and the _secure eFuse-burn end-to-end HIL is DONE — the bench devkit boots with Secure Boot v2 + Flash Encryption burned (espefuse-verified). Claude-Session: https://claude.ai/code/session_01SV58JXhfxhhc9DC6vdjBo4

1 month ago

Fix the two failures the fleet power-QA sweep exposed - Gray (IP5306): test_battery_voltage_plausible asserted a LiPo voltage range on every DUT, but the IP5306 has no battery-voltage ADC — the backend structurally reports 0 mV. Key the assertion on the PMIC model and assert the structural 0 for IP5306 (HIL: model=IP5306 batt_pct=100 charging=1, batt_mv=0). - AXP backend: the boot-time PMIC probe can transiently NACK right after a serial-open DTR/RTS reset, leaving the whole boot with no PMIC (model=none, 0 readings) — observed intermittently on core2_v13 and plus1. Add a bounded lazy re-probe (one retry on first telemetry read) so a single missed probe no longer costs the boot's telemetry. Verified on core2_v13: 3 consecutive full power-QA runs, 5/5 each. Claude-Session: https://claude.ai/code/session_01SV58JXhfxhhc9DC6vdjBo4

1 month ago

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

Add hardware coverage: buttons(all), keyboard, touch, trackball, IMU, RTC, audio, SD, I2C, display, power, crypto Broaden HIL coverage from serial/BLE/WiFi/video to every hardware peripheral, capability-gated so each test spawns only on the boards that have it. No firmware change needed — the debug console already exposes every command. Inventory: devices.yaml capabilities now carry imu/rtc/audio/sd/trackball per the per-variant hardware matrix; config.py adds the matching capability predicates. DebugConsole: key, ball, drag, beep, imu, rtc_set, spi, disp, brightness, sd, sdtest, i2c_scan, cryptokat, pbkdf2bench. New tests: - Input: test_buttons (1/2/3-button PRESS/RELEASE + tap, beyond two_button), test_keyboard (KEY), test_touch (DRAG), test_trackball (BALL). - tests/hardware/: display (SPI?/DISP?/BRIGHT), rtc (RTCSET+RTC?), imu (~1 g), audio (BEEP), sd (SD?/SDTEST), i2c (scan + expected addrs), power (batt %/mV), crypto (CRYPTOKAT 6/6 + PBKDF2BENCH). Markers: capability (trackball/imu/rtc/audio/sd/buttons) + hardware-domain (display/power/crypto/i2c/onboarding/editor) so you can run/skip a single domain (-m imu), file, test, or -k substring. Documented in qa-system.md. Validated: Stick S3 13 passed / 10 capability-skipped; T-Deck 19 passed / 4 skipped. ruff + black clean; no-hardware suite green. Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_sd_backup.py

fix(vault): close secure-wipe probe FD before unlink (prevents LittleFS corruption on vault wipe) cryptoEraseFile() declared the read probe as a function-scoped `const File`, so the read handle stayed OPEN through lfs::remove() — LittleFS refuses to unlink a file with an open FD ("Failed to unlink … Has open FD"), leaving the directory pair to corrupt on the next mount and forcing a reformat: silent vault loss on every wipe (10-failure wipe / RESETOB / WIPEVAULT). Scope the probe in its own block so RAII closes it before any remove. Surfaced by the new on-device HIL SD-backup test (native fake-FS does not enforce the open-FD-unlink constraint, so native missed it — the HIL test is the guard). Also fix test_sd_backup.py no-oracle assertion to bind the SDIMPORT_ reply line rather than the interleaved (generic, debug-only) unwrap log. Claude-Session: https://claude.ai/code/session_01SLSU3WoYEyX1J7qWXxmyKo

1 month ago

Add SYS/VER/FLASH/FS/SHAKE debug introspection commands + QA tests Close HAL serial-exposure gaps found in the QA gap analysis: build/system state that had no debug command. All read-only, DEBUG_SERIAL_BUTTONS-gated, KEY=value. Firmware: - platform/system.cpp: new resetReason() facade over esp_reset_reason() (native returns "UNKNOWN") — the console goes through the facade, not the ESP-IDF call. - ui/debug: VER? (version/device/id/release/dbg_level/pbkdf2_iters/cpu_mhz/mac), SYS? (reset reason + wake cause + from_deep_sleep — named SYS? not RESET? so it never collides with the base RESET restart), FLASH? (size/running/next-OTA/state), FS? (LittleFS mounted/total/used/free), SHAKE? (enabled/threshold, available=0 without an IMU). One snapshot helper each in debug_diagnostics.cpp. - Documented in serial-debug-commands.md. Builds sticks3_debug + tdeck_debug -Werror clean; native suite still 1168/1168. QA: - DebugConsole: ver, sys_info, flash_info, fs, shake. - tests/hardware/test_system.py (VER? cross-checks DEVICE_ID vs inventory + asserts an open debug build; SYS?/FLASH? plausible; FS? tolerates the on-demand mount — LittleFS is mounted per vault-op and unmounted for deep sleep, so at rest mounted=0 is normal). Shake config check added to test_imu.py. - `system` domain marker. Validated: Stick S3 5 passed, T-Deck 4 passed (DEVICE_ID cross-check green on both). Claude-Session: https://claude.ai/code/session_01Y7Cy1HetRp6TZAUAeekN8X

1 month ago

test_trackball.py

Formalize trackball QA: TRKRW parser + hardware suite - debug_console.py: TrkSelfTest dataclass, parse_trk_selftest, and the dut.trk_selftest() helper (device token, pin mapping, click level, ISR-armed state, PASS/FAIL) mirroring kbd_selftest/imu_selftest. - test_trk_selftest_parse.py: no-hardware parser unit tests over a MockDut (quadrature PASS, click-held informational, absent, not-armed FAIL, boot noise, missing line). - test_trackball.py: trackball-capability-gated self-test (PASS, device match, ISRs armed — the load-bearing input-path evidence for a device whose motion cannot be synthesized) plus an absent-graceful check for the rest of the fleet. Motion is intentionally not driven — it needs a physical finger. Claude-Session: https://claude.ai/code/session_01SV58JXhfxhhc9DC6vdjBo4

1 month ago