Commits
Adds a "Liveness detection (model-free PAD)" section to CONTRIBUTING.md
covering: the evidence/veto composition (noisy-OR capped by veto product,
why not a weighted average, veto-only and fail-closed semantics, the
NotApplicable additivity that makes strobing structurally more secure);
the NIR reflectance gate; the removed motion-magnitude dead end and its
intended planar-homography replacement; the mandatory observe-only-until-
tuned posture and the `calibrate-liveness` workflow; and honest scope
(casual vs determined attacks, ISO/IEC 30107-3 per-attack evaluation).
Updates the "future milestone" PAD note to reflect what now exists, and
extends the defense matrix with a tuned-liveness column.
The guided live-vs-spoof profiling workflow that turns the liveness
checks' guessed thresholds into data-derived ones. Incremental and
GUI-shaped — each invocation does one thing, accumulating into a
persisted profile:
- `capture <class> [--label .. --append --conditions ..]` — capture a
labeled batch (live / printed-photo / screen / other). Opens the IR
camera, runs the same detect → liveness-policy pipeline auth uses,
harvests each check's sub-score `detail` into a feature vector, and
folds the batch into the profile (replacing the class, or appending).
- `status` — per-class sample counts.
- `report` — per-feature live-vs-spoof separation (means ± std, d-prime).
- `suggest-threshold` — per-feature threshold separating live from each
spoof, with the false-reject / false-accept it would incur.
- `reset [<class>]` — clear all, or one class.
Each verb is one core operation, so a future GUI can drive the same
flow — and because core ingests caller-supplied frames/features, a GUI
that owns the camera for preview can feed its own frames rather than
contend for the device.
Features are namespaced by check (e.g. `nir_reflectance.brightness`) so
multiple checks' sub-scores don't collide. A no-face capture (e.g. a
screen that's dark in IR) is skipped and counted — and an all-skipped
batch errors with a note that no-face is itself the dual-modality
defense. Gated on v4l2+inference with an arg-compatible fallback;
verified the --no-default-features build still compiles.
`LivenessProfileStore`: a single-file CBOR store for the accumulating
guided-calibration profile. Like the camera calibration store and unlike
the enrollment store, no HMAC — it holds tuning data, not secrets, and
liveness degrades gracefully on an absent/unreadable profile. Atomic
writes (temp + fsync + rename) so a crash never leaves a torn file;
versioned (refuses unknown future versions). `load` returns None when
absent (uncalibrated, normal), `load_or_default` for read-modify-write
verbs, plus `save`/`remove`.
Tested: missing-is-None, save/load round-trip, idempotent remove, no
temp-file leakage, an arbitrary-profile round-trip proptest, and a
16-thread concurrent-writer stress test asserting the file is never torn
(every post-race load decodes; last-writer-wins on content).
Property tests over the profile analysis:
- FeatureStats: min <= mean <= max, std >= 0, exact count, for any
finite value set.
- suggest_threshold: reported error rates are always valid fractions in
[0,1]; and — the headline invariant — the chosen threshold *minimises*
total error (live-reject + spoof-accept) over the sample set: no other
observed cut point does better. That's the guarantee that the
suggester actually picks the best separating threshold.
- separation: its sign follows the class-mean ordering; analysis never
panics on arbitrary data.
The data side of guided liveness calibration: an accumulating profile of
labeled samples plus pure analysis over it. No hardware — a sample is a
named feature vector (the CheckEvaluation `detail` a check produces) with
a class label, so the caller supplies samples (CLI from V4L2, a future
GUI from frames it owns). This keeps the camera-ownership boundary clean:
core never opens a device.
- `SampleClass`: Live / PrintedPhoto / Screen / Other(label) — flexible
per-attack-type labeling for ISO-30107-3-style reporting.
- `LivenessSample`: { class, features: Vec<(name, value)>, timestamp }.
Stores computed features, not raw frames — privacy (no face images in
a non-HMAC file), size, and check-agnostic extensibility. Format
versioned so raw-frame retention could be added later.
- `LivenessProfile`: accumulate via `replace_class` (default re-capture)
or `append_batch` (--append), plus `clear_class`, `class_counts`,
`feature_names`. Carries a free-form `conditions` note since a profile
is only valid for similar capture conditions.
- Analysis: `FeatureStats`, `separation` (d-prime-like class separation
per feature), and `suggest_threshold` (sweeps cut points to minimise
live-reject + spoof-accept, orienting correctly whether live reads
higher or lower than the spoof). Suggestions only — never auto-applied.
Unit-tested: accumulation semantics, stats, separation magnitude, and
threshold suggestion on cleanly-separable / overlapping / inverted-sense
data.
A policy with only veto checks and no evidence check — exactly our
current NIR-gate-only config — always scored combined = 0, because
combined = noisy_or(evidence) × ∏(veto) and noisy_or([]) = 0. So a live
face the NIR veto scored 0.94 came out as 0.000. (Surfaced in testing: a
clearly-live face logged nir_reflectance 0.94 but Liveness combined
0.000.)
`assess` now composes by *what applied*:
- nothing applicable (no evidence, no veto) → 0.0, so a positive
threshold fails closed (operator enabled enforcement but nothing ran);
- veto(s) but no evidence → the veto product alone (a veto-only policy
reduces to the vetoes' verdict — the fix);
- evidence present → noisy-OR of evidence, capped by vetoes.
Because present evidence caps *further* than vetoes alone, the veto-only
neutral baseline never weakens security; it only stops an empty
noisy-OR from zeroing a working veto. `evidence` is still reported as
the raw noisy-OR for diagnostics.
Tests: veto-only reduces to the veto score (and still zeroes a spoof),
and "nothing applicable" stays fail-closed-at-positive-threshold,
distinct from veto-only.
A check's single combined score hides *why* it landed there — e.g. a
genuine face scoring 0.69 on the NIR gate could be limited by any one of
its brightness/structure/smoothness sub-scores, and you can't tune or
trust a threshold without knowing which.
`LivenessCheck::evaluate` now returns a `CheckEvaluation { outcome,
detail }`, where `detail` is a list of named `(sub-score, value)` pairs —
purely diagnostic, never affecting composition. `CheckScore` carries it
through to the assessment, and `test`'s liveness output prints the
breakdown indented under each check.
The NIR gate reports its three sub-scores (brightness/structure/
smoothness) plus the raw measurements (mean/std/roughness), so a
borderline live result is interpretable and guided calibration has the
per-feature data it needs per attack type.
Chose the explicit return-struct refactor over a recompute-on-demand
`detail()` method: single evaluation, detail is first-class. Checks with
no breakdown use `CheckEvaluation::bare`.
The strongest, cheapest model-free liveness signal on a NIR sensor, and
the research-recommended first check. A single-frame veto on the face
region exploiting physics RGB can't:
- Screens emit ~no near-IR → a displayed face is dark and flat under the
sensor → vetoed.
- Skin has a characteristic NIR reflectance and sub-surface scattering →
a real face is bright with rich but diffusely-smooth structure; paper
reflects differently (flatter, or halftone-textured).
`NirReflectanceGate` (role: Veto) combines three face-region features
into a confidence-live score, taking their *minimum* so an attack need
fail only one to be vetoed and a genuine face must satisfy all:
1. brightness ≥ `min_mean` (screens too dark);
2. structural contrast (std) ≥ `min_std` (flat surfaces fail);
3. diffuse smoothness — normalised mean |Laplacian| ≤ `max_roughness`
(print halftone / screen grid add high-frequency texture).
It measures the detected face region (via the plumbed bbox), not the
whole frame, so it doesn't confuse a dim room with a screen; it abstains
(`NotApplicable`) when there's no face box rather than veto on a weak
signal. Registered in `test`'s policy.
`face_region` is extended to handle `Rgb8` as well as `Gray8`: the IR
capture path delivers RGB8 with the grey value triplicated into R/G/B,
so a single channel is the intensity. Without this the gate saw RGB8
frames and always returned NotApplicable (the `n/a` it reported in
practice) — so this is what makes the gate actually run.
Tested: dark/flat screen-like region vetoes (~0), bright smoothly-shaded
region passes, high-frequency-textured region is penalised, score in
[0,1], and RGB8 intensity extraction. Thresholds are first-cut estimates
to be tuned by guided calibration; liveness stays observe-only
(threshold 0.0) until then.
Face-region liveness checks (the NIR reflectance gate next) need to
measure *where the face is*, not the whole frame — otherwise they
conflate "bright lit face" with "bright background, dark face" (a screen
showing a face) or with a dim room.
`LivenessInput` gains `face_bbox: Option<BoundingBox>`, and a
`face_region(frame)` helper that crops a single-channel (Gray8/IR) frame
to that box, clamped to frame bounds (SCRFD boxes can spill past the
edge), returning the region pixels + dimensions. Shared plumbing so each
face-region check doesn't re-implement clamping.
`IrCapture::liveness_input` populates `face_bbox` from the matching
outcome's detected face (None when no face was found, so a check falls
back to whole-frame or abstains). Tested: crop dimensions, out-of-bounds
clamping, and the no-bbox case.
Magnitude of inter-frame change is not a liveness signal. On real
hardware a live face scored 0.3-0.6 while a phone screen saturated at
1.0 (flicker reads as huge "motion") and a moving printed photo ranged
0-1 — i.e. the metric was anti-correlated with liveness, passing the
spoofs and nearly rejecting the genuine user. This matches the PAD
literature's consistent finding that liveness lives in the *structure*
of signals (spatial texture, motion geometry, spectral reflectance), not
the *magnitude* of any single quantity.
So `MicroMotionCheck` is removed rather than kept as misleading
near-useless code. `liveness_policy()` is now an empty no-op policy
(combined = 0.0, passes at the default threshold 0.0), pending the real
checks the research points to — a NIR-reflectance veto gate next, then a
structure-aware planar-homography motion check. The two-role framework
and all the capture/burst plumbing remain; only the wrong-feature check
goes. Planned-checks docs updated accordingly.
Liveness checks now declare a `CheckRole`: positive **evidence** of life,
or a spoof **veto**. The policy composes them differently:
evidence = noisy_or(evidence check scores) // raises confidence
combined = evidence × ∏(veto check scores) // veto caps it
This is needed because some signals are evidence of an *attack*, not of
life, and noisy-OR (which ignores low scores) would render them inert. A
veto check still outputs "confidence the subject is live" in 0..1, but
*low* when it detects an attack signature; multiplying caps the final
score, so a confident veto (≈0, "this is a screen") overrides even
strong evidence. An inapplicable veto isn't in the product and caps
nothing, preserving additivity for veto checks too.
`role()` defaults to `Evidence`, so existing checks are unaffected;
spoof detectors override it. `LivenessAssessment` gains `evidence`
(pre-cap) alongside `combined` (post-cap) for diagnostics. The old
`combine` is renamed `noisy_or` to name what it actually does now that
it's only the evidence half.
Proptested: noisy-OR bounded/monotonic/single-identity as before, plus
the defining veto property — adding a veto never raises the final score
— and the combined score stays in [0,1] for any evidence/veto mix.
(The first real checks land next: the micro-motion check is being
removed as a proven dead end — magnitude-of-change is anti-correlated
with liveness, per both measurement and the PAD literature — and
replaced by a NIR-reflectance veto gate.)
`test` now runs the liveness policy against the IR capture and reports
it. The IR modality captures inline (via the new `capture_ir` helper)
instead of through the RGB retry path, so the full burst is retained and
handed to the policy as a `LivenessInput`. The burst is sized with
`liveness_burst` so the micro-motion check has enough same-phase frames.
Output gains a `Liveness:` line with the combined score, the threshold,
the applicable/total check counts, and a per-check breakdown (including
`NotApplicable` lines, so it's visible why a sensor's ceiling is what it
is). A new `--liveness-threshold` flag (default 0.0) gates the verdict:
a passing identity match is downgraded to `Reject (liveness)` if the IR
capture didn't clear the threshold. At the default 0.0 this never
fires — liveness is observe-only until profiled, exactly the posture for
untuned heuristics.
Also simplifies the old `try_capture_face` into an RGB-only
`capture_rgb_face`: the IR branch and its `kind`/`ir_burst` params became
dead once IR moved to the inline burst-retaining path, so they're
removed rather than left as misleading cruft.
Liveness is wired on IR here, but the framework is modality-agnostic;
RGB liveness (retain RGB frames, run the same policy) is a documented
future step.
Micro-motion (and future multi-frame liveness checks) need several
same-illumination-phase frames, but the calibrated matching burst is
minimal — one strobe period, i.e. a single lit frame on a period-2
sensor. `liveness_burst(matching_burst, depth)` grows it to
`matching_burst * depth` so the burst spans `depth` periods and yields
`depth` same-phase frames (clamped to depth >= 1; saturating, no
overflow). `DEFAULT_LIVENESS_DEPTH` is 3.
Pure helper with unit tests; the `test` command wires it in when it runs
liveness (next commit). Matching-only paths keep using the minimal burst.
The first real liveness signal: scores involuntary frame-to-frame change
across the capture burst. A live subject always moves a little over the
~100-200ms a burst spans (head sway, micro-expression, blink); a printed
photo or a screen held up is rigid, differing only by sensor noise. So
mean per-pixel change is a model-free liveness signal — and it works on
*any* IR sensor, which is why it's the first check.
# Same-illumination comparison
The one subtlety is strobed sensors: their burst alternates lit/dark
frames, so diffing consecutive raw frames would read the strobe as huge
bogus "motion". The check therefore compares only same-illumination-phase
frames — those within `phase_band` of the lit frame's brightness. On a
constant sensor every frame qualifies (the grouping collapses to "use
all"), which is what makes the check universal. A test pins that the
strobe's dark frame is excluded.
# Applicability
Needs >= 2 same-phase frames; with fewer it returns `NotApplicable`, not
a bogus score. A minimal period-2 matching burst (one lit frame) is
therefore NotApplicable — liveness-aware burst sizing (next commit) is
what gives it enough same-phase frames. The additive model means this
just contributes nothing until then, never breaks anything.
# Scoring
Normalised mean-absolute per-pixel difference between consecutive
same-phase frames, saturating to 1.0 at `MicroMotionConfig::saturation`.
The constants are first-cut estimates to be tuned from real
live-vs-spoof captures (guided calibration), which is why liveness stays
observe-only (threshold 0.0) until then.
Proptested: total + bounded over arbitrary frames, and monotonic in
motion (more per-pixel change never lowers the score). Also records the
planned complementary checks (embedding-agreement, texture, differencing)
in the module docs.
`capture_ir_face` previously discarded every burst frame except the
chosen lit one. It now returns an `IrCapture` that owns the whole burst
plus the lit-frame index and an optional dark-frame index, with a
`liveness_input()` accessor producing a `LivenessInput` for assessment.
The dark frame is identified *from the data*, not from calibration, by
`detect_dark_frame`: the dimmest burst frame, reported (`Some`) only
when it's at least `STROBE_DARK_CONTRAST` (25/255) darker than the lit
frame — i.e. the sensor is actually strobing. On a constant-brightness
sensor all frames are similarly bright, so `dark_index` stays `None` and
any strobed-only liveness check gates itself off. Strobed differencing
stays additive, never required.
`detect_dark_frame` is deliberately a small isolated function with a
prominent KNOWN-LIMITATION note: its absolute brightness gap is only
valid under the (indoor) ambient conditions it was tuned in. Outdoors,
broadband sunlight floods both lit and dark frames with near-IR, shrinks
the absolute gap, and could disable differencing on a genuinely strobed
sensor. The proper fix (scale-invariant detection, or deferring the
strobed-vs-constant verdict to calibration) is not attempted yet — no
daylight IR captures exist to tune against. The decision is quarantined
here so the future change is local; callers see only the `Option<usize>`,
and a wrong `None` degrades safely.
Pure plumbing otherwise: `enroll` and `test` take `.outcome` and behave
exactly as before; no liveness check runs yet.
Introduces liveness / presentation-attack detection as a confidence axis
orthogonal to identity matching, structured so it works on any IR camera
and treats a strobed illuminator as an additive bonus rather than a
prerequisite.
# Shape
- `LivenessCheck` trait: each check inspects a `LivenessInput` (the full
captured burst, the chosen lit frame's index, and an optional dark
frame index for strobed sensors) and returns `CheckOutcome` — either
an `Applicable(score)` in 0..=1 or `NotApplicable`. A check that needs
hardware it didn't get (e.g. differencing on a constant sensor)
returns `NotApplicable`, never a low score, so adding it can't break
unsupported hardware.
- `LivenessPolicy` runs all checks and combines applicable scores via
noisy-OR (`1 - ∏(1 - sᵢ)`), producing a `LivenessAssessment` with the
combined score, the applicable count, and a per-check breakdown for
logging.
# Why noisy-OR, not a weighted average
Two properties the security model needs, both proptested:
1. No inflation: one applicable check of score `s` combines to exactly
`s`. A weighted average over *applicable* checks would renormalize a
lone weak signal up to full-range confidence, erasing the difference
between a constant sensor (few checks) and a strobed one (more). With
noisy-OR a `NotApplicable` check simply isn't in the product.
2. Monotonic: more applicable evidence never lowers the score, so a
strobed sensor's extra check can only *raise* its ceiling. Strobing
being more secure is thus structural, not a tuning artifact.
# No-op by default
An empty policy assesses `combined = 0.0`; `passes(threshold)` with the
default `threshold = 0.0` passes everything — exactly today's behavior,
liveness effectively off until checks are registered and a threshold is
chosen. A positive threshold with zero applicable checks fails closed.
Pure and total (clamps out-of-range / non-finite scores); no hardware,
no models. Real checks and the auth-path wiring land in follow-up
commits.
A targeted testing pass over the pure logic added across the IR-capture
and calibration work, plus one latent bug it surfaced.
# Bug fix: strobe period truncation
`classify_ir_mode` computed the strobe period as `(w[1] - w[0]) as u8`,
which *truncates* (wraps mod 256) when bright frames are more than 255
indices apart — a long, sparse burst could classify as a bogus tiny
period. Now uses a saturating `u8::try_from(...).unwrap_or(u8::MAX)`. A
regression test pins it (gap of 300 → 255, not 44), and was confirmed to
fail against the old code.
# Property tests
- `select_brightest`: result is an in-bounds true argmax; `Some` iff
the input is non-empty.
- `classify_ir_mode`: total (never panics) over arbitrary `&[f32]`
including NaN/inf/empty; a synthesized period-`p` strobe round-trips
to `Strobed { period: p }`; a sub-contrast stream is always
`Constant`; the result's `min_burst()` is always >= 1 and any strobed
period >= 2.
- Calibration store: `save_all`→`load` round-trips an arbitrary record
set (mirrors the enrollment store's headline property); `upsert`
leaves exactly one record per identity carrying the new mode.
- `resolve_calibration`: total; `Exact` implies an identical full
identity; any non-`None` match implies a same-card+kind record exists;
deterministic.
- `resolve_ir_camera` / `resolve_rgb_camera`: explicit path always wins;
total; an adopted camera is always of the matching kind; RGB adoption
happens exactly when there's a single path-bearing RGB camera.
# Stress test
Calibration store: 16 threads racing `upsert` on one file. The
read-modify-write isn't serialized (last-writer-wins on the record set,
as documented), so the test asserts only *integrity* — atomic
temp+rename means every post-race `load` succeeds on a structurally
valid, non-torn file and two consecutive loads agree.
RGB selection relied on a hardcoded `--rgb-camera` default of
`/dev/video0` — a guess that silently breaks when the RGB camera isn't
video0, and an inconsistency with IR, which auto-detects. Now both
modalities resolve the same way.
# core
`resolve_rgb_camera` + `RgbCameraResolution` mirror the IR resolver, with
RGB's mandatory-modality semantics: no opt-out, and the zero-camera case
is a hard `NoHardware` failure rather than a benign skip. Explicit path
wins; otherwise the single RGB-classified device is adopted, with zero
or several surfaced as `NoHardware` / `Ambiguous` for the caller to
render. Pure and unit-tested with synthetic snapshots, including that IR
cameras don't count toward RGB ambiguity.
# cli
`--rgb-camera` becomes optional (no default). `enroll` and `test` resolve
it through the shared `pipeline_runtime::resolve_required_rgb`, which
renders the outcome to a path or an actionable error ("pass --rgb-camera
<PATH>"). On a typical laptop with one RGB node this just works with no
flag; a machine with several RGB cameras now gets a clear "pick one"
error instead of silently grabbing video0.
Integrated webcams routinely expose their IR and RGB nodes under the
*same* V4L2 card name (the XPS 13 reports both as
`Integrated_Webcam_HD`). With calibration keyed on card + bus alone, an
IR record and the RGB sibling on the same bus were indistinguishable,
and the RGB node counted as a competing candidate that could make an IR
record's port-move resolution spuriously `Ambiguous`.
`CameraIdentity` gains a `kind` field, and `resolve_calibration` matches
on card + bus + kind: exact matches can't cross modalities, and the
same-card/different-bus (`MovedPort`) path filters candidates by kind so
a shared-card-name sibling of the other modality is ignored.
The field is `#[serde(default)]` to `Ir`, so calibration files written
before it existed — which only ever recorded IR cameras — still
deserialise. Documents the remaining limitation: two identical-model
cameras of the same kind on one machine are still distinguished only by
bus, and a same-kind duplicate that also moved ports falls back to a
mode-agnostic burst.
`test` built its `AuthMode` straight from `--ir-camera`, so omitting the
flag silently dropped to single-RGB auth even on a machine with a
working, auto-detectable IR camera — the same camera `enroll` and
`calibrate` find on their own. That mismatch is why dual-mode auth
appeared to require the flag.
`test` now resolves IR the same way `enroll` does, via
`resolve_ir_camera` (explicit flag > `--no-ir` opt-out > auto-detect):
an auto-detected or explicit IR camera runs dual-modality auth; no IR
hardware (or `--no-ir`) falls back to single RGB; multiple IR cameras
warn and fall back rather than guess. Adds `--no-ir` to `test` to mirror
`enroll`'s flag set so RGB-only is still reachable on dual-capable
hardware.
This also aligns `test` with what the M5 daemon's auth path must do —
resolve cameras from detection/config, not from a per-invocation flag.
Closes the calibration loop: `enroll` and `test` now read the calibration
store and size their IR capture burst from the detected
`IrCaptureMode::min_burst` instead of always using `DEFAULT_IR_BURST`. On
the strobed XPS 13 that drops the burst from 4 frames to the minimal 2,
shaving capture time; an uncalibrated or unmatched device transparently
keeps the safe default.
`pipeline_runtime::resolve_ir_burst` does the resolution via
`resolve_calibration`, handling every `CalibrationMatch`:
- `Exact` / `MovedPort` — use the calibrated burst. `MovedPort`
(device replugged to another port, card name still unique) also
refreshes the stored record's bus so the next run is an `Exact` hit.
- `Ambiguous` — several identically-named cameras connected; warn and
fall back to the default burst.
- `None` — uncalibrated; use the default silently.
Calibration being an optimization, every failure path (unreadable store,
ambiguous match, write-back failure on `MovedPort`) degrades to the
default burst with at most a warning, never an error.
`EnrollmentStore::root()` is added so the CLI can locate the sibling
calibration store in the same directory without re-resolving flags.
Measures an IR camera's illuminator emission mode once and persists it,
so subsequent IR captures can size their burst from the known mode
instead of always over-capturing. Gated on `v4l2` (it captures frames)
but not `inference` — it only measures per-frame brightness, never runs
detection.
Resolves the IR camera with the same precedence as `enroll` (explicit
flag > opt-out > auto-detect) so the two can't pick different cameras,
captures a 30-frame burst, classifies the brightness sequence via
`classify_ir_mode`, and upserts an `IrCalibration` record keyed by the
camera's card+bus identity into the calibration store (alongside the
enrollment store root). Uses `replace_by_card` so re-calibrating after a
port move doesn't leave a stale-bus record behind.
Prints the detected mode (Constant, or Strobed with its period) and the
observed brightness range so the result is legible and the measurement
auditable.
Calibration records how a given IR camera delivers illuminated frames so
capture can be optimized per device. It is purely an optimization and
capability layer: `capture_ir_face` already works without it (capture a
burst, keep the brightest), so a missing/unmatched/corrupt record just
falls back to the safe mode-agnostic burst — never an error. That's why
this store, unlike the enrollment store, needs no HMAC.
# Device identity (card + bus)
Records are keyed by `CameraIdentity { card, bus }`. Neither field alone
is a usable key: `/dev/videoN` renumbers across reboots, the card name
isn't unique across identical models, and the bus (port topology)
changes when a device is replugged elsewhere. `resolve_calibration`
matches leniently, returning a typed `CalibrationMatch`:
- `Exact` — card + bus both match.
- `MovedPort` — card matches, bus differs, and exactly one connected
camera has that card name, so the device just moved ports: safe to
adopt and refresh the stored bus.
- `Ambiguous` — card matches but several connected cameras share that
name, so we can't guess which: the caller should ask the user.
- `None` — no record; fall back to mode-agnostic capture.
This mirrors the decisions-in-core / rendering-in-CLI shape of
`camera::select::IrCameraResolution`.
# Store
`CalibrationStore` persists all records in one small CBOR file (the set
is tiny and the matching API wants it whole). Writes are atomic
(temp-file + fsync + rename), matching the enrollment store's crash
discipline. `upsert` replaces by exact identity; `replace_by_card` drops
a stale-bus record when a device moved ports.
# CameraInfo.bus
`CameraInfo` gains a `bus: Option<String>`, filled by `enumerate()` from
the V4L2 bus info and left `None` for mock/synthetic sources and the
by-path `open()` path. This is the input `CameraIdentity::from_info`
reads. Calibration shares the `v4l2` feature gate since it depends on the
backend's `IrCaptureMode`.
Wires both subcommands to the strobe-tolerant `capture_ir_face` so IR
enrollment and IR auth verification stop failing on dark strobe frames.
# Root cause
The IR sensor on the 2018 Dell XPS 13 strobes its illuminator: it emits
a strictly alternating lit/dark frame sequence (measured: lit ~76, dark
~11, period 2, indefinitely). A single IR shot therefore had a ~50%
chance of landing on a black frame, which is exactly the intermittent
"IR: no face detected" seen post-enrollment. `capture_ir_face` captures
a short burst and keeps the brightest frame, which is always the lit one
regardless of strobe phase.
# enroll
The IR modality now captures a `DEFAULT_IR_BURST` burst per sample via
`capture_ir_face` instead of a detect-on-every-frame retry loop, dumping
every burst frame under `--debug-dump-dir` (lit and dark alike) and
mapping an all-dark burst to `IrTooDark`. Drops the now-unused
`IR_DETECT_ATTEMPTS`.
# test
RGB keeps the bounded detection retry (good for blinks/pose); IR uses
the brightest-of-burst capture. A failing IR modality whose frames
stayed black reports that the illuminator likely never fired rather than
a bare "no face". `--debug-dump-dir` (added here) writes
`{rgb,ir}-attempt{n}.ppm`, so the auth path is diagnosable the same way
enrollment is.
Builds on the core IR-selection primitives to give the CLI capture
helpers the enroll/test commands need.
# IR capture (strobe-tolerant)
`capture_ir_face` captures a short *burst* of IR frames, picks the
brightest via `select_brightest`, and runs detection only on that. This
is the fix for the strobed XPS 13 sensor: because the emitter alternates
lit/dark frames, "the brightest frame in the burst" is the illuminated
one regardless of where in the strobe cycle capture began — no luck, no
per-frame detection guesswork. The burst length is `DEFAULT_IR_BURST`
(4) when the mode is unknown; a later calibration pass can size it
exactly (1 for constant, `period` for strobed).
# RGB bounded retry
`capture_one_face_retrying` (unchanged in spirit) retries detection
across a bounded number of frames so a blink or unsettled pose doesn't
fail a modality. The bound is deliberate — this runs on the auth/test
hot path, where an unbounded "loop until a face appears" would hang the
login flow whenever no one is at the camera. `RetryCaptureOutcome`
distinguishes `Found` from `NoFace { attempts, brightest }`.
# Open path + diagnostics
`open_camera` now uses `Warmup::Frames(1)` for both kinds — purely the
STREAMON priming discard, with IR frame selection delegated to
`capture_ir_face` rather than a (now-removed) brightness-gated warm-up.
`frame_mean_brightness` becomes a thin wrapper over core's
`frame_brightness`. `DARK_FRAME_MEAN_THRESHOLD` + `CliError::IrTooDark`
turn an all-dark burst (illuminator never fired) into an actionable
message instead of a bare "no face".
IR cameras with an active illuminator deliver illuminated frames in
ways that vary by hardware. Measuring the 2018 Dell XPS 13 IR node
showed it is not a slow-ramp emitter (an earlier theory) but a *strobed*
one: it is lit from its first frame and then alternates lit/dark frames
forever, the emitter pulsed every other frame.
That makes "discard frames until one is bright" actively wrong — under
strict alternation, stopping warm-up on a bright frame guarantees the
*next* captured frame is the dark twin. Frame selection, not stream
warm-up, is the right layer for this.
This commit provides the mode-agnostic primitives the capture layer
needs, all pure and unit-tested without hardware:
- `frame_brightness` — mean of raw frame bytes, a cheap luminance proxy
(the GREY IR bytes already are luminance).
- `select_brightest` — index of the brightest frame in a burst. Whatever
the emission pattern (constant, strobed, ramping), the brightest frame
in a long-enough burst is a usable illuminated one, so this alone is
enough for correct IR capture without knowing the mode.
- `IrCaptureMode` { `Constant`, `Strobed { period }` } + `classify_ir_mode`
— classify the emission pattern from a run of frame brightnesses. This
is an optimization/capability signal (size the capture burst minimally;
unlock future ambient-subtraction/liveness on strobed pairs), never
required for correctness.
`Warmup` loses its brightness-gated `UntilBright` variant and is now
purely a fixed priming-frame discard (`Frames(n)`), correct for every
sensor kind. `mean_brightness` moves out of `Warmup` to the free
`frame_brightness` function its several consumers share.
We're committing to a CLI design where the project stays ready for
additional UI layers (GUI, TUI, RPC client) without requiring the CLI
to be untangled. The rule that falls out, recorded in CONTRIBUTING.md
§ UI architecture: decisions live in `pareidolia-core` as pure
functions returning structured outcomes; rendering lives in
`pareidolia-cli`. Each match arm in a subcommand should be either
'translate this variant to its display string' (rendering) or 'call
this core function and dispatch on its outcome' (still rendering at
the CLI layer, with the actual decision made one frame down). It
should not be 'compute the next state inline'.
This commit extracts four meaningful instances of inline decision
logic from CLI commands into typed outcomes in core, and documents
the pattern so future contributors don't have to re-derive it.
# core: `camera::select::IrCameraResolution` + `resolve_ir_camera`
The IR-camera resolution that lived inline in `pareidolia enroll`
(explicit flag wins, then opt-out, then auto-detect with three
sub-branches) becomes a five-variant enum returned by a pure
function over `&[CameraInfo]`.
Unit-testable without V4L2: ten tests cover each branch and the
precedence ordering (explicit > opt-out, opt-out > auto-detect,
multi-IR collapses to Ambiguous, defensive handling of a path-less
`CameraInfo`). A future GUI would call `resolve_ir_camera` with a
list it owns and render the same five variants as widgets. The
daemon will use the same function with a hardware snapshot pulled
per auth request.
`enroll.rs` now contains only a `match` over the five variants
that picks the appropriate stderr notice and extracts the resolved
path.
# core: `enrollment::AuthMode`
Two-variant enum (`Single { rgb_camera }` / `Dual { rgb_camera,
ir_camera }`) for the configured authentication flow. CLI builds
one via `AuthMode::from_optional_ir(rgb, ir_opt)`; the daemon will
build its own from config + per-user persisted paths in M5. Both
will then call `authenticate` or `authenticate_dual` based on the
variant.
Replaces the inline `match &args.ir_camera` dispatch in
`pareidolia test`. The decision and the camera-path data now live
in one place instead of being threaded through args throughout the
flow.
# core: `enrollment::EnrollmentReadiness` + `from_load_result`
Three-variant classification of a `store.load(user)` result:
`NotEnrolled` (no enrollment file), `Empty` (file exists, zero
samples — distinct because the user *was* enrolled at some point),
`Ready(Enrollment)`.
`from_load_result` folds the `StoreError::NotEnrolled` variant
into a non-error `NotEnrolled` readiness state since 'no enrollment
yet' is normal flow control, not a failure. Genuine I/O / decode
errors stay as `Err`.
`pareidolia test` now calls this exactly once at the top of the
flow; the daemon (M5) will call it on every PAM auth request as
the canonical 'should we open the camera or short-circuit?' check.
Four unit tests cover the three readiness states plus error
propagation.
# core: `enrollment::EnrollmentSessionSummary`
Per-session counts (`new_rgb`, `new_ir`, `total_rgb`, `total_ir`,
`user`) returned by the enroll loop for rendering. CLI picks
between two text shapes based on `summary.had_ir()`; a GUI will
populate widgets from the same struct.
A small extraction, included alongside the larger three for two
reasons: it demonstrates the value-struct flavour of the pattern
(not every extraction needs to be an enum), and it cleans up the
last loose `if new_ir > 0` decision-vs-render mix in the enroll
flow.
# Documentation
CONTRIBUTING.md gains a `## UI architecture` section between
'Testing philosophy' and 'Security' that captures the convention,
points at `IrCameraResolution` as the canonical worked example,
lists what already follows the pattern and what doesn't yet, and
records the constraints we will *not* break (no interactive
prompts, no spinners/colour/TUI, no `init` wizards) — those would
make the CLI hostile to scripting and impossible for a non-terminal
UI to reuse anyway.
Two pieces of the dual-modality story were still missing after M4.7:
1. **Default behaviour didn't take advantage of available IR hardware.**
Users had to remember to pass `--ir-camera /dev/videoX` to enroll
the IR pool, and most never would. The recommended security posture
(dual-modality, defeats the photo-spoof attack) was opt-in rather
than opt-out.
2. **The daemon (M5) needs to know which physical device captured which
pool to reopen the same hardware on the PAM auth hot path** without
re-running the `enumerate()` heuristic for every auth request, and
without risking picking the wrong camera on systems with several.
# Auto-detect when --ir-camera is unset
`pareidolia enroll` now resolves the IR path via:
- `--ir-camera <PATH>` → use exactly that.
- `--no-ir` → skip IR; never auto-detect.
- Neither set → call `enumerate()`:
- 0 IR-classified cameras → silent RGB-only.
- 1 IR-classified camera → adopt it, print a one-line notice
naming the path and the `--no-ir` opt-out.
- 2+ IR-classified cameras → warn and skip IR (refuse to guess),
pointing the user at `--ir-camera` or `--no-ir`.
The single-IR auto-adopt is the common case (laptop built-in IR
sensor, or a single USB IR module). The multi-IR ambiguity branch
exists because silently picking the wrong sensor would pollute the
pool with samples from a different physical camera than the daemon
will later use at auth time, and the user would have no idea why
authentication felt unreliable.
`--no-ir` is mutually exclusive with `--ir-camera` (clap enforces).
Both flags are documented in `--help`.
# Persisted camera paths in Enrollment
`Enrollment` gains two optional fields:
pub rgb_camera_path: Option<PathBuf>,
pub ir_camera_path: Option<PathBuf>,
Both use `#[serde(default, skip_serializing_if = "Option::is_none")]`,
which has two consequences worth pinning explicitly:
- **Forward-compat for existing files**: enrollments written by
Pareidolia versions without these fields decode cleanly into the
new shape with `None` values. `encode_with_no_camera_paths_omits_keys_and_decode_yields_none`
verifies the on-disk payload is byte-identical to the old format
when paths are `None` (by asserting the field-name strings do not
appear in the encoded CBOR), so existing user enrollments survive
the upgrade untouched.
- **Round-trip preserved when set**: `encode_then_decode_preserves_persisted_camera_paths`
exercises the populated path through encode/decode and back.
The enroll loop writes the RGB path unconditionally (we always capture
RGB) and the IR path only when IR enrollment actually happened this
session, so a later `--no-ir` re-run doesn't clobber a previously
persisted IR path. Stale paths are harmless: the daemon will check
sample presence first and fall back to `enumerate()` if the persisted
device is gone.
# Resolution logic lives inline in enroll.rs for now
The IR-camera resolution match is currently inline in
`cli::commands::enroll::run`. A separate follow-up commit will lift
this kind of decision logic into `pareidolia-core` as structured
outcome types (the "decisions in core, rendering in CLI" pattern
we want to establish across all subcommands so a future GUI can sit
on the same primitives). Keeping the architectural refactor in its
own commit makes the user-visible feature here easy to review and
revert independently.
Single-modality face auth is identity-matching, not liveness detection.
ArcFace is trained to cluster the embedding space by identity; a sharp
photo of a face produces an embedding very close to the face's own,
because identity is what ArcFace was optimised to preserve. We
confirmed the practical impact: `pareidolia test` (RGB) passes when
shown a phone-screen photo of a relative, with similarity solidly
above the 0.6 threshold.
The same query against the IR camera, however, didn't even reach the
matcher — SCRFD failed to detect a face in the IR view of the phone
screen at all, because phone/laptop displays emit visible light but
almost no near-infrared. Printed photos behave similarly in most
cases. Requiring *both* modalities to pass independently therefore
closes the common presentation-attack vector without adding any
anti-spoof ML.
# Core: `authenticate_dual` + `DualMatchOutcome` + `SingleVerdict`
`Enrollment::authenticate_dual(rgb_query, ir_query, threshold)` takes
two `Option<&Embedding>` queries (one per modality) and returns a
`DualMatchOutcome`:
- `Pass { rgb, ir }` — both modalities cleared the threshold against
their respective enrolled pools. Returns the winning `MatchResult`
from each pool for audit / logging.
- `Reject { rgb, ir }` — at least one modality failed. Both
per-modality `SingleVerdict`s are preserved so the failure mode is
fully diagnosable (which side failed, and how).
- `NoEnrollment` — fires only when the enrollment has zero samples
total; per-modality emptiness collapses into `SingleVerdict::NoSamples`
on that side instead.
`SingleVerdict` carries the four states a per-modality decision can
land in: `Pass`, `BelowThreshold` (face found, similarity below
threshold), `NoFace` (caller didn't supply an embedding — capture
error, no face detected, modality disabled), `NoSamples` (enrollment
has no samples for this kind). The `Option` parameter shape lets the
caller express 'couldn't produce an embedding for this side' directly,
guaranteeing the matcher rejects rather than silently degrading to
single-modality.
# Tests
Eight new unit tests pin the full truth table (empty enrollment, both
queries present and matching, RGB pass + IR below threshold, missing
RGB query, both queries missing, RGB-only enrollment with IR queried,
both below threshold, is_pass helper). Two new property tests pin the
soundness invariants:
- `authenticate_dual_pass_iff_both_single_modality_pass` — the
primary contract: dual.is_pass() ⟺ both single-modality authenticate
calls would have passed. Callers can reason about
`authenticate_dual` as 'AND over single-modality outcomes'.
- `authenticate_dual_with_either_query_missing_never_passes_on_nonempty`
— a missing query on either side always forces Reject for any
non-empty enrollment, regardless of threshold.
One integration smoke test
(`public_dual_modality_authenticate_requires_both_pools_to_pass`)
covers the end-to-end public API including the Pass shape and the
None-query Reject shape.
# CLI: `pareidolia test` learns dual mode
The `conflicts_with = rgb_camera` mutex on `--ir-camera` is removed.
Mode selection becomes:
- `--ir-camera` unset → single RGB mode against the RGB pool
(unchanged from M4.6).
- `--ir-camera` set → dual mode: capture one frame from each sensor
serially (sibling-node cooldown between, like enroll), run
`authenticate_dual`. 'No face detected' on one side becomes
`SingleVerdict::NoFace` rather than a hard error, so the verdict
surfaces fully.
The dual output path prints both per-modality verdicts on reject
('RGB: pass (similarity 0.823 ≥ 0.600, sample 1) / IR: no face
detected in captured frame'), making it obvious which side blocked
auth. The previous IR-single-camera testing flow gives way to dual:
running `pareidolia test --ir-camera /dev/videoX` against the typical
RGB+IR enrollment is now a strictly stronger check than before.
# Documentation
A new `## Security` section in CONTRIBUTING.md explains the threat
model: what ArcFace matches (identity, not liveness), what dual-mode
defeats (phone-screen photos, most paper photos) and what it doesn't
(NIR-tuned prints, 3D masks, NIR-illuminated photographs), and the
separate concern of family-resemblance false-accepts. A defense
matrix summarises the cells.
Diagnosing camera or face-detection failures from a hardware-only
trace ('confidence 0.231' or 'no face detected') is painful because
the actual frame data has already been thrown away by the time the
user sees the error. There's no way to tell whether the camera
captured a black frame, a wrong exposure, a frame of someone else's
face, or a perfectly fine frame that SCRFD just couldn't handle.
When `pareidolia enroll --debug-dump-dir <PATH>` is set, every
captured (post-conversion, pre-detection) frame is written to that
directory as a binary PPM file before the detector runs. The dump
fires regardless of detection outcome, so failed captures still
leave the frame on disk for inspection. Files are named
`rgb-{i}.ppm` and `ir-{i}.ppm` with 1-based sample indices.
PPM (P6) was chosen instead of PNG because the goal is debug
output, not distribution: no compression, no extra dependencies,
universal viewer support (feh, eog, ImageMagick, GIMP), and a hex
dump of the file is still meaningful if a viewer isn't handy. The
format is a three-line text header followed by raw RGB bytes.
# Pipeline runtime split
To dump the frame *before* face detection runs (so failed
detections still leave a frame), `pipeline_runtime::capture_one_face`
is decomposed into:
- `capture_rgb8(source)` — capture + RGB8 conversion only
- `embed_best_face(frame, detector, embedder)` — detect, align,
embed, pick highest-confidence face
`capture_one_face` itself remains as a thin wrapper composing the
two, so the `test` subcommand and any future single-shot callers
don't need to change. Enrollment switches to the two-step form so
the optional dump can sit between capture and detection.
# Scope
Behaviour is unchanged when the flag is absent — no extra disk
writes, no extra log lines, no change to enrollment storage. The
flag is documented as a debugging tool, not a production knob; the
daemon path (M5) won't expose it.
`pareidolia --version` now prints e.g. `pareidolia 0.0.0 (a59ad019…)`,
so a quick check tells you whether the installed binary actually matches
the workspace you think you're building from. Previously the version
was just the package version, which on a pre-1.0 codebase (we've been
at 0.0.0 since M1) is functionally meaningless for 'is this current?'
questions.
# Wiring
- The flake's shellHook runs `jj log -r @ --no-graph -T commit_id`
(with a `git rev-parse HEAD` fallback for non-jj checkouts and
`echo unknown` for working copies outside any VCS) and exports the
result as `PAREIDOLIA_BUILD_REV` at shell-entry.
- The CLI lib reads the env var at compile time via `option_env!`,
defaulting to `"unknown"` when unset. A `VERSION_FULL` const
combines the package version and the rev via
`const_format::concatcp!` — pure compile-time string concat, no
runtime allocation, returns a true `&'static str` that clap's
`#[command(version = ...)]` attribute can use directly.
- No `build.rs` (we still keep that rule from the build-cache hygiene
doc; the rev comes in via env var, not a build script).
# Staleness window
The rev is captured at *shell-entry*, not at build time. A build done
immediately after `jj commit` reflects the new commit; a build done
later in the same shell session reflects whatever the rev was when
the shell was entered. Re-enter `nix develop` (or re-export the env
var) to refresh. Documented in the shellHook comment.
# Non-Nix builds
`cargo install` outside the devShell sees no `PAREIDOLIA_BUILD_REV`
env var, so the rev defaults to `unknown`. That's a feature: it
tells the user the build provenance isn't tracked. Setting the env
var manually before `cargo install` also works for anyone who wants
the rev without the devShell.
ArcFace is RGB-trained, so an IR-captured face embedding lives in a
different region of the 512-d space than the RGB embedding of the
same face. Comparing across modalities produces near-zero similarities
even for the same person — fail-safe but unusable. Making IR auth
actually work meant moving five pieces together: the matching layer
had to filter by modality, enroll had to capture both sensors,
`open_camera` had to choose the right pixel format for each kind,
the V4L2 stream had to be primed before the first capture would
return real data, and the dual-camera capture loop had to open each
sensor on demand rather than holding both simultaneously.
# Matching (core)
`Enrollment::best_match` and `Enrollment::authenticate` now take a
`CameraKind` parameter representing the live query's modality. Only
same-kind samples are considered for the best-of-N selection. The
property tests already establish that best-of-N is monotonically
improving in N; adding the modality filter doesn't change that
property within either pool.
A new `MatchOutcome::NoSamplesForModality { kind }` variant
distinguishes 'user has samples but not of this kind' from
`NoEnrollment` ('user has no samples at all'). They sort to the
same auth decision (no match possible) but have different
remediations: enroll vs enroll-for-this-modality. The PAM glue in M5
will route them differently in logs.
# Dual-camera enroll (cli)
`pareidolia enroll` gains `--rgb-camera` (default /dev/video0,
renamed from `--camera`) and `--ir-camera` (optional). When both
are set, each capture iteration pulls one frame from each sensor:
the RGB frame goes into the enrollment tagged Rgb, the IR frame
tagged Ir. Same user pose roughly across both, but slight motion
between them is fine — each modality's samples live in their own
pool and benefit from minor pose variation within that pool.
# Camera format selection by kind
`open_camera` picks the V4L2 FourCC by camera kind: YUYV for RGB
cameras, GREY for IR. Hardcoding YUYV across the board caused IR
captures to either negotiate to garbage or silently mislabel a GREY
stream as YUYV, which then propagated through the pipeline as
gibberish data and produced 'no face detected' for every IR sample.
Most laptop and USB IR sensors advertise GREY (8-bit grayscale) and
nothing else.
# Stream priming
`open_camera` captures and discards one frame immediately after
opening. Two unrelated problems both want the first frame thrown
away:
1. The `v4l` crate's `Stream::next()` runs `VIDIOC_STREAMON` on
the first call and returns a buffer slot without the QBUF/DQBUF
cycle that would fill it with real captured data. The first
frame from a fresh source is therefore whatever was in the mmap
buffer at REQBUFS time — typically zeros, sometimes garbage.
2. IR illuminators on UVC cameras are commonly gated on the device
entering streaming state and take a frame or two after STREAMON
to reach full brightness. The first captured IR frame can be
too dark for the detector even when the crate's behaviour is
perfect.
Priming once after open sidesteps both. The ~33 ms cost is paid
once per source, not per capture.
# Open-on-demand for dual-modality enroll
The original dual-camera enroll loop held both V4L2 sources open
for the whole session to amortise the per-open cost. This wedges
some UVC dual-sensor devices: sample 1 succeeds, then the next
DQBUF on either sibling video node blocks indefinitely. UVC
bandwidth is allocated statically at STREAMON and dual-sensor
devices vary in how gracefully they cope with two concurrent
sustained streams.
Cameras are now opened **per modality, per sample**: open RGB,
prime, capture, drop; sleep a sibling-open cooldown; open IR,
prime, capture, drop. The two streams are never alive at the same
time. Cost is ~1s per sample of open-overhead, paid only during
enrollment (a once-per-user operation), and the pattern matches
exactly what the daemon will do for PAM auth requests in M5.
# Test subcommand
`pareidolia test` gains the same flag pair, but mutually exclusive:
either you test against RGB (`--rgb-camera`, default) or IR
(`--ir-camera`), since a single test is a single match attempt.
The query's modality is now passed through to
`Enrollment::authenticate` and the matcher filters appropriately.
A new printable path handles `NoSamplesForModality` cleanly with
an actionable suggestion ('re-run `pareidolia enroll` with
--ir-camera to add them').
# Tests
Two new unit tests pin the modality filter:
- `best_match_filters_to_query_kind_ignoring_cross_modality_samples`
- `authenticate_with_only_other_modality_samples_yields_no_samples_for_modality`
One new utility (`Enrollment::count_by_kind`) with its own unit
test; used by the enroll CLI for the user-facing 'saved X RGB and Y
IR samples' summary.
One new integration smoke test:
- `public_cross_modality_query_surfaces_no_samples_for_modality`
All existing tests updated to pass `CameraKind::Rgb` to
`best_match`/`authenticate` (existing samples are tagged Rgb by
default in the test fixture, so behaviour is unchanged for them).
The module is named `enrollment` (American) and the subcommand is
`enroll`, but I'd drifted into British 'enrol' / 'enrolment' in a
handful of doc comments and arg descriptions across the enrollment
type docs, the CLI `enroll` and `test` subcommands, and their
feature-disabled fallbacks. Both spellings are valid English, but
mixing them in one codebase is just noise — standardising on the
American form matches the module/subcommand names that have been
in place since they were introduced.
Pure docs / arg-description change; no code behaviour affected.
The smallest possible end-to-end check that an enrollment actually
works: capture one frame, embed the face, match against the user's
existing enrollment, report Pass / Reject / Not enrolled. Lets users
validate their enrolment before any of the daemon / PAM / IPC plumbing
arrives in M5+, and provides an obvious triage step once the full
login path exists ('auth is failing — does `pareidolia test` pass?').
# Order of operations
Enrollment is loaded *first*. If the user isn't enrolled (or has zero
samples), report that and exit cleanly without ever touching the
camera or loading models. The 'am I enrolled' question is often the
actual one and shouldn't require waking the IR illuminator to answer.
# Graceful fallback without features
The CI build (--no-default-features) doesn't have v4l2 or inference,
so the real subcommand impl isn't compiled. The fallback in
commands.rs handles the no-enrollment case fully (no hardware needed
— just the store) and only surfaces FeatureDisabled when an enrolled
user would require an actual capture to match. This matches what users
expect: 'asking whether someone is enrolled' should always work; 'asking
whether *this face* matches them' requires the hardware integrations.
# Shared pipeline runtime
`enroll` and `test` had near-identical model-loading and capture
logic; factored into `pipeline_runtime` so each subcommand impl
stays focused on what's actually different (enroll appends N samples,
test matches one). `resolve_model_path`, `capture_one_face`,
`model_fingerprint`, and `open_camera` live there. `enroll`
refactored to use them; behaviour unchanged.
# Camera defaults to YUYV
`open_camera` explicitly requests YUYV format. USB webcams (the
BRIO included) commonly default to MJPG output, which we don't yet
decode — `camera::convert::to_rgb8` punted MJPEG to a later
milestone. YUYV is universally supported by UVC cameras and lands
through the existing convert path. When MJPEG decoding arrives the
preference can become a `--fourcc` flag.
# Tests
One integration smoke test covers the no-enrollment path against an
empty store through `pareidolia_cli::run` in-process. The full
Pass/Reject paths are user-driven hardware tests — running `test`
against the user's own face after a successful `enroll` is the
obvious manual validation step.
Three new subcommands that drive the enrollment store from the user
side:
# enroll
Runs the full pipeline locally: V4L2 capture → format convert → SCRFD
detection → affine align → ArcFace embed → append to store. The
daemon will eventually drive this same flow via IPC for the PAM auth
path, but enrolment is infrequent and user-initiated, so a
self-contained CLI is the simplest design and the matching algorithm
already handles the append-not-replace case via best-of-N monotonicity
(see enrollment.rs).
Loads any existing enrolment first and appends; this is the default
flow precisely because the matching layer's best-of-N is monotonically
improving in N, so re-running enrol on a day you look different
(lighting, beard, baggy eyes, glasses) just widens the set of poses the
matcher will accept. A future --replace flag is a clean addition.
Feature-gated on both v4l2 and inference. The CLI's default-features
now includes both, mirroring the daemon's shape. CI's
--no-default-features build picks up a fallback that surfaces
CliError::FeatureDisabled with the missing feature name so the message
is actionable.
Each enrolled sample stores a SHA-256 fingerprint of the SCRFD model
file. Matching against samples from a different model than the one
currently loaded is meaningless and would silently degrade accuracy;
M5+ will use this fingerprint to refuse such matches.
# list / remove
Both are feature-flag-free (only need the store, not models or
camera). `list` defaults to one-name-per-line output for scripting;
`--verbose` loads each enrollment to include the sample count (which
HMAC-verifies every file, so it's not free at scale). `remove`
deletes the whole enrollment by default; `--sample-id N` removes
just that sample.
# Shared store config
`--store-root` and `--key-file` are global args. Both default
to XDG_STATE_HOME (or $HOME/.local/state fallback). The key file is
auto-generated from /dev/urandom on first use with mode 0o600, which
keeps the "just works in nix develop" UX without burying the
biometric-data PII concerns under a default that silently writes to
`./pareidolia` or similar surprise paths.
# clap env feature
Workspace clap dep now includes the `env` feature so subcommand args
can fall back to environment variables (heavily used here for model
paths and store config). Pure Rust, no system deps.
# Tests
Two integration smoke tests cover the dispatch + store wiring against
in-process `run()`:
- list against an empty store root → succeeds (catches subcommand
wiring regressions before any data exists).
- remove of a never-enrolled user → succeeds (idempotency holds at
the CLI layer too).
Plus four unit tests on the store_config helper module covering store
root creation, key file generation + permissions, key file idempotency,
and wrong-sized key file rejection.
Real enrolment requires the user's face and a working camera, so it's
covered by manual testing rather than automation — appropriate for a
user-initiated subcommand.
The store's write path was direct fs::write; a crash mid-write could
leave a torn file, and the default umask was determining whether the
biometric-data file was world-readable. Both are now fixed.
# Atomic via write-temp-then-rename
`save` now:
1. Opens a uniquely-named temp file (`.target.tmp.{pid}.{nanos}`) in
the same directory as the target, using O_CREAT | O_EXCL so a name
collision surfaces as a hard error rather than silent overwrite.
2. Writes all bytes and `sync_all`s the file so data is on disk
before the rename.
3. `rename(temp, target)` — atomic on POSIX within a filesystem;
either the old file is in place or the new one is, never a torn
mix.
4. `sync_all`s the parent directory so the rename itself survives a
crash. Errors at this step are non-fatal: the data is durable, only
the directory entry might not be (we'll log once logging lands in
M5).
Temp files are cleaned up on error so the directory doesn't accumulate
`.tmp.` debris over partial writes.
# Owner-private file permissions
Files are now created with explicit mode 0o600 (owner read+write only)
via OpenOptionsExt::mode. The daemon's intended deployment runs as a
dedicated system user; with 0600 only that user (and root) can read
the enrolled face embeddings, which are PII.
# Tests
Three new tests pin the new guarantees:
- Mode-0o600 is set on every saved file (PermissionsExt-based check).
- No `.tmp.*` files leak after a successful run (multiple saves).
- 16 concurrent writers racing on the same user yield a loaded file
that matches exactly one of the writers' inputs — never a torn mix.
This is the atomic-rename guarantee in test form.
Adds the persistence layer for enrollments. Each enrolled user gets one
CBOR-encoded file under a configured root directory, wrapped in a fixed-
position header and HMAC-SHA256 so any out-of-band mutation surfaces as
IntegrityFailure on the next load.
# File format
offset size contents
0 4 magic "PARE" (ASCII, fixed)
4 1 version currently 1
5 3 reserved zero-padded
8 32 HMAC-SHA256 HMAC of bytes [40..] under the key
40 … payload CBOR-encoded `Enrollment`
The HMAC sits at a fixed offset *before* the payload deliberately:
verification is one constant-time slice compare with zero
deserialisation. Parsing untrusted CBOR ahead of integrity verification
would expose the daemon to whatever bugs a malicious file could
trigger in the deserialiser.
`verify_slice` is constant-time, which matters — a timing leak there
would let an attacker forge an HMAC byte by byte.
# Atomicity
`save` currently writes the target file directly; a crash mid-write
can leave a torn file. The atomic write-temp-then-rename path lands in
the next sub-milestone (M4.3) and won't change the format or HMAC
scheme.
# Username sanitisation
Usernames double as filenames, so they're validated against an
allowlist: ASCII alphanumerics plus `_`, `-`, `.`, no leading dot,
no `..`, max 64 bytes. Path traversal attempts surface as
InvalidUsername before the filesystem sees them.
# Tests
11 unit tests covering encode/decode round-trip, header validation
(magic, version, truncation), HMAC integrity (wrong key, payload
tampering, HMAC tampering), username validation (positive and negative
cases), and the EnrollmentStore filesystem operations (save+load,
NotEnrolled, list_users sorting, remove idempotency, on-disk tamper
detection).
3 property tests pin the headline guarantees:
- `decode(encode(e, k), k) == e` for arbitrary enrollments — the
format loses no information.
- Flipping any post-header byte triggers IntegrityFailure — what turns
'we have an HMAC' into 'we have integrity'.
- Cross-key decode always fails — keys are not interchangeable.
# Deps
serde + ciborium + hmac + sha2 join the workspace deps. All four are
pure Rust with no system requirements, so they don't need feature
gating; CI's --no-default-features build picks them up automatically.
First M4 slice. Pure data + math: enrolled samples, the per-user
enrollment that holds them, and the cosine-similarity-based matching
algorithm that drives the eventual PAM auth decision. The on-disk
store (HMAC, atomic writes) lands in subsequent commits; this slice
gives matching tests something to run against and pins the algorithmic
contract.
# Best-of-N, not mean-of-N
The matching layer takes the *best* of the N enrolled-sample
similarities, not their mean. Mean-of-N would make a frontal-only
enrollment fail on a slight head turn even though the user is clearly
recognisable from one of the angles already captured. Best-of-N matches
how real enrolment data is captured (multiple poses to span variation)
and how face recognition systems are typically operated.
# Match outcomes
`MatchOutcome` distinguishes three states: `Pass` (clears threshold),
`Reject` (below threshold, includes the best result so logs can show
how close), and `NoEnrollment` (no samples to match against). The
third variant is separated from `Reject` because the remediation
differs — the user needs to enrol, not retry — and the PAM glue will
want to route those cases differently when they cross the IPC boundary
in M5.
# Numeric edge cases pinned
- NaN similarities (dim-mismatched embeddings) are filtered out of
best-of-N selection rather than poisoning it. A mismatched-model
query against a populated enrollment surfaces as 'no valid candidate'
rather than 'best match was NaN'.
- Threshold comparison is `>=`: similarity exactly at the threshold
passes. The test `authenticate_at_exactly_the_threshold_passes_inclusively`
hand-crafts a similarity of 0.6 against threshold 0.6 to pin this.
# Tests
15 unit tests covering enrollment construction / push / remove,
best-match selection (single, multi-axis, NaN-skipping, mixed-validity),
threshold behaviour, and the MatchOutcome helpers.
5 property tests covering the invariants the matching layer relies on:
- Pushing a sample never decreases best-match similarity (B-o-N
monotonicity).
- Removing a sample never increases best-match similarity (mirror).
- Query identical to an enrolled sample yields similarity ≈ 1.0.
- Threshold at -∞ always passes any non-empty enrollment.
- Threshold above 1.0 never passes any normalised embedding.
Plus three cross-crate integration smoke tests for the public surface.
The SCRFD module's pure stages (NMS, letterbox preprocess, bilinear
resize, anchor decode) had hand-picked goldens but no property coverage.
Those are exactly the code paths where structural bugs silently degrade
detection quality without crashing — the kind property tests catch best.
# Coverage added
NMS (the largest gap):
- Output count is never greater than input count.
- Every pair of surviving detections has IoU ≤ threshold — the
postcondition NMS exists to enforce.
- The globally-best detection (highest confidence) always survives.
- Idempotency: running NMS on its own output produces the same set.
- Input-order independence: reversing the input order produces the same
surviving set.
letterbox_preprocess:
- Output tensor is always [1, 3, target, target] regardless of source
dimensions.
- det_scale is finite, positive, and produces source dimensions that
fit inside the target (within sub-pixel rounding).
bilinear_resize_rgb8:
- Output buffer length is always exactly dst_w · dst_h · 3.
- Solid colour preservation across arbitrary resize ratios (stronger
than the unit test which fixed one ratio).
decode_stride:
- Never panics on arbitrary tensor lengths or grid sizes. A malformed
model output should produce zero detections, never an index-out-of-
bounds panic that takes down the daemon.
# Performance
Letterbox proptests are bounded to 400×400 source dimensions because
each iteration runs a full bilinear resize. The 'output shape' and
'det_scale fit' properties are structural and don't gain coverage from
larger inputs; capping the bound keeps the test in single-digit seconds.
# Cost to CI
Zero: the proptests live inside the inference-feature-gated scrfd
module, so CI's --no-default-features build doesn't compile or run them.
The full-feature local suite picks them up via `just test-features`.
`pipeline::recognise` ties the three model-touching stages — Detector,
align_face, Embedder — into a single call. The output is a Vec of
(Detection, Embedding) pairs, one per face the detector found,
preserving the detector's ordering. This is what the daemon's PAM path
will eventually drive once enrollment + matching land in M4.
Pure orchestration code: no feature gate, no new runtime deps. It
operates on the existing trait + type surface, so it compiles in every
workspace configuration including --no-default-features.
# Errors abort the pipeline
Per-face partial successes (e.g. 'detected three faces, aligned two,
embedded one') would be hard to reason about in the matching layer, so
any stage error aborts the whole call with no results. The unit tests
pin this behaviour: detector errors, degenerate landmarks (align fails),
and embedder errors all surface as the original error variant with no
side effects.
# Golden test against real models
A new hardware-gated test runs the full pipeline against the InsightFace
sample face JPEG (t1.jpg, ~128 KB, MIT-licensed via their repo). It
asserts:
- ≥1 detection (the fixture has a clearly visible face; finding zero
would mean the detector regressed).
- Every detection's confidence clears the default 0.5 threshold.
- Every embedding is 512-d with L2 magnitude ≈ 1.0.
- Running the pipeline twice on the same frame yields bit-identical
embeddings (CPU EP determinism guarantee).
Stops short of pinning exact embedding values — that's the matching
layer's concern in M4, where similarity-against-stored-baseline becomes
the natural assertion.
# Fixture infrastructure
The fixture image is pinned in nix/models.nix with sha256, fetched at
devShell build time, and exposed via PAREIDOLIA_TEST_FACE so the
hardware test runs without manual setup inside `nix develop`. Outside
Nix, set the env var manually (the InsightFace URL is in
nix/models.nix).
`image` (jpeg + png decoders only, default-features off) joins the
workspace dev-dependencies for fixture loading; production code never
needs to decode images because the camera produces RGB8 directly.
The hardware-gated tests were panicking when their prerequisite (a
plugged-in camera, or a PAREIDOLIA_TEST_* env var pointing at a model
file) was missing. That made sense as a 'you ran this wrong' assertion
when the only invocation path was explicit `just test-hardware` from
inside `nix develop` with hardware present — but it falls over the
moment any of those conditions slip.
Concrete case: a USB unplug or driver hiccup leaves /dev/video* gone,
and `just test-hardware` fails noisily even though the other hardware
tests (model-loading, inference) would have run fine. The panic also
wrongly suggests something's broken when in fact the test simply lacks
its physical prerequisite.
Replace each panic with an `eprintln!` describing what's missing and
how to set it up, then early-return. Nextest treats the early return
as a pass — conventional behaviour for hardware tests in the Rust
ecosystem — and the suite stays green when hardware is temporarily
absent.
The eprintln output is captured by nextest's default profile, so the
skip messages would be invisible unless tests failed. `test-hardware`
now passes `--no-capture` so each test's stderr streams through in
real time, and each skip message is prefixed `warning:` so it stands
out.
While I was in there: the per-camera capture test was tolerantly
`continue`-ing on individual open errors but then *succeeding* even
when every camera failed, defeating the point of the test (saw this
when the BRIO firmware wedged into an EIO state after a previous run).
It now asserts at least one capture succeeded among the enumerated
devices, and inserts a 500ms cooldown between iterations because UVC
cameras commonly EIO on rapid open-close-open cycles.
UVC cameras commonly expose multiple /dev/video* nodes per physical
device: typically one or two real Video Capture interfaces plus matching
Metadata Capture siblings. Our enumerate() was accepting all of them by
filename pattern alone, so a Logitech BRIO (with video0/2 real + video1/3
metadata-only) showed up as four rows in `pareidolia cameras` and got
exercised four times by the capture hardware test.
The metadata-only nodes don't have VIDEO_CAPTURE capability, so opening
them for capture (set_format, REQBUFS, STREAMON) fails after the
firmware-touching ioctls have already woken the device. On the BRIO this
puts the IR illuminator into a 'warmed' state that lingers for several
seconds after the test ends — visible IR LEDs glowing between test runs
— and also causes control-query timeouts on subsequent v4l2-ctl
inspections.
Querying VIDIOC_QUERYCAP on the node (the device_caps field, which is
per-node rather than the union across the driver) and filtering by
VIDEO_CAPTURE excludes the metadata siblings without any extra fragility:
real capture devices keep advertising the cap; metadata ones don't.
After this, `pareidolia cameras` on a BRIO returns:
PATH NAME KIND
/dev/video2 Logitech BRIO ir
/dev/video0 Logitech BRIO rgb
instead of four rows where two were unusable noise.
Last real-model piece of the pipeline. Takes a 112×112 RGB8 chip produced
by `align_face`, runs a single ONNX forward pass through the ArcFace
glintr100 model, and returns a 512-d L2-normalised embedding ready for
the matching layer in M4.
Substantially simpler than the SCRFD detector: single input, single
output, no anchor decoding, no NMS. Preprocessing differs from SCRFD's
in one detail — ArcFace's training used `(pixel − 127.5) / 127.5`
(range [-1, 1]) where SCRFD used `/ 128.0` — and the embedder is strict
about input dimensions: 112×112 only, no rescaling on input. Wrong shapes
surface as `InvalidInputShape` before any inference runs.
The 512-d output goes straight through `Embedding::normalized` so cosine
similarity reduces to a dot product in M4's matching layer. The output
dimension is enforced after inference; a model returning the wrong
dimension surfaces as a Backend error rather than producing a malformed
embedding.
# Tests
4 pure unit tests covering the preprocessor (NCHW shape, the −1/+1
boundary mappings, per-channel separation into NCHW planes).
5 hardware-gated tests covering the embedder end-to-end via the real
ArcFace model:
- Non-RGB8 input is rejected before inference
- Wrong dimensions are rejected before inference
- Solid gray chip produces a 512-d unit-magnitude embedding
- Repeated inference is bit-for-bit deterministic (default CPU EP)
- Different inputs produce non-identical embeddings (sanity check)
Combined hardware suite now runs in ~16s (V4L2 capture dominates at ~11s;
all 5 ArcFace tests take ~5s combined including model load).
The pipeline stage between the detector and the embedder. Given detected
landmarks and the canonical ArcFace template, fits a similarity transform
(uniform scale + rotation + translation), inverts it, and warps the source
frame into a 112×112 chip.
# 2D-specific math
ArcFace alignment is a 2D-to-2D similarity transform, so we get a closed-
form least-squares fit without invoking SVD. The 2x2 rotation+scale can
be parameterised by a single (u, v) pair (u = s·cos θ, v = s·sin θ) and
solved as four sums over the centred points. This keeps the alignment
code free of any linear-algebra dependency — no nalgebra, no manual SVD.
# Behaviour at the edges
- Non-RGB8 frames return InvalidInputFormat rather than producing garbage.
- All-coincident or otherwise degenerate landmarks return a Backend
error (the linear system has no solution). The daemon should fall
through to password in this case rather than panic.
- Inverse-mapped samples that fall outside the source frame return
black pixels. This produces an aligned chip with some black border
when the face is near the source edge — a useful downstream signal
rather than a fatal error.
# Tests
19 unit tests covering the SimilarityTransform type (identity, scale,
rotation, inverse round-trip, singularity guard), the closed-form fit
(pure translation, pure scale, pure rotation, mismatched / empty / coincident
inputs), align_face (format validation, output shape, identity-warp
solid-colour preservation, metadata propagation, degenerate-landmark
error), and the bilinear sampler (out-of-bounds, NaN, exact-integer hits).
2 property tests guard the headline invariants — the fit-then-apply
reproduces any well-conditioned target within f32 tolerance, and the
inverse round-trips any non-singular transform within f32 tolerance —
so future micro-optimisations of the math can't silently drift.
First real Detector impl. Wires SCRFD-10G (loaded by OnnxSession from the
nix-pinned ONNX file) into the pipeline's Detector trait. Covers the full
flow: letterbox preprocess, normalised NCHW tensor, single inference call,
9-output anchor decoding across 3 stride levels, NMS, coordinate
rescaling.
Trait refinement: Detector::detect and Embedder::embed now take `&mut self`.
ONNX Runtime's Session::run is `&mut self` and the daemon owns one
detector per worker (never sharing concurrently), so this matches the
usage pattern without forcing interior mutability. As a bonus the
ScriptedDetector mock can now pop its queue properly instead of the
peek-and-pretend-to-mutate hack the previous `&self` signature forced.
Implementation notes:
- Letterbox uses the standard SCRFD layout: resized image at top-left,
zero-padded right/bottom, single scale factor. The OpenCV half-pixel
offset convention in the bilinear resampler matches reference Python
implementations, so detection coordinates compare correctly against
insightface goldens later.
- Bilinear resize is hand-rolled (~30 LOC) rather than pulling in the
`image` crate. We'll add `image` in M3.5 when fixture loading needs
it; not paying for it now keeps the inference feature's transitive dep
surface small.
- Decoder bails (returns no detections) rather than panicking when output
tensor shapes don't match the expected stride layout. A model variant
with different strides shouldn't take down the daemon, just report no
faces and let PAM fall through to password.
- Default thresholds (score 0.5, NMS IoU 0.4) match InsightFace
recommendations.
Tests:
- 18 unit tests on the pure helpers (NMS, anchor decoding, letterbox,
bilinear resize, coordinate rescaling). The decode test exercises the
anchor centre → bbox + landmarks math against hand-computed expected
outputs; the NMS tests cover score-order invariance, IoU-threshold
edge cases, and disjoint-box preservation.
- 1 hardware-gated end-to-end test loading the real SCRFD model and
running it on a synthetic black frame. Asserts no detections (no face
exists) and no panic; the runtime path is what matters here. Currently
~210ms wall-clock on CPU on the dev machine.
The real Detector and Embedder implementations need ONNX Runtime; this
commit wires up the dependency, the Nix devShell, the pinned model
weights, and a thin session wrapper they'll build on top. The actual
SCRFD detector and ArcFace embedder land in subsequent sub-milestones
(M3.3, M3.5).
Feature shape mirrors v4l2: `pareidolia-core` keeps `inference` opt-in
(default = []); the daemon enables it via its own default-on `inference`
feature that proxies to core. CI's --no-default-features keeps libv4l +
libonnxruntime out of the Nixery image while normal builds get the full
pipeline.
`ort`'s `load-dynamic` feature is the cleanest fit for Nix: the crate
dlopens libonnxruntime at runtime via ORT_DYLIB_PATH rather than linking
against it at build time. The devShell wires ORT_DYLIB_PATH to nixpkgs's
`.so` so both build and run work without bundling C++ libs.
`OnnxSession` exposes the underlying `ort::Session` directly rather
than wrapping every ort API behind a Pareidolia-flavoured alternative —
chasing ort's API as it evolves is more churn than the abstraction is
worth. The boundary is at error types: `ort::Error` is mapped to
`PipelineError::Backend` so the rest of the pipeline stays
backend-agnostic.
Model weights: the M6 plan had Nix-driven model fetching land alongside
the NixOS module, but the M3 sub-milestones need real ONNX files for
their hardware tests, so the fetching infrastructure moves forward.
`nix/models.nix` pins SCRFD-10G and ArcFace glintr100 from the
`immich-app/buffalo_l` HuggingFace mirror (a maintained mirror of the
InsightFace bundle, stable since 2023). The devShell exports
PAREIDOLIA_TEST_{MODEL,SCRFD,ARCFACE} pointing at the materialised store
paths so `just test-hardware` runs end-to-end with no manual setup.
The pinned weights are also exposed as flake packages
(`nix build .#scrfd-10g`) for ad-hoc use outside the devShell.
Foundation for the three-stage face recognition pipeline (detect → align →
embed). The traits exist now so the daemon, integration tests, and (later)
NixOS VM tests can be wired against them while the real ONNX-backed
implementations land in subsequent sub-milestones (M3.2 detector, M3.4
embedder).
Types worth flagging:
- `Embedding` is a Vec<f32> newtype with L2-normalisation built into the
preferred constructor (`Embedding::normalized`) and cosine similarity
as a plain dot product. Pre-normalising lets the matching layer skip
per-comparison magnitude divisions in the hot path. Magnitude-zero
inputs round-trip without dividing by zero so a degenerate embedder can
never panic the daemon.
- `Landmarks` fixes the canonical InsightFace order (left eye, right eye,
nose, mouth-left, mouth-right). The affine alignment in M3.3 depends on
this order being stable; nailing it down here means future code that
iterates landmarks (via `Landmarks::iter`) doesn't have to negotiate.
- `PipelineError` deliberately doesn't derive `Clone` — the scripted mock
implements re-emission with an explicit per-variant clone helper, so
adding a new variant in the future requires a conscious decision about
how it should propagate through scripted timelines.
Mock pair: `FixedDetector` / `FixedEmbedder` for the common 'return the
same thing on every call' pattern; `ScriptedDetector` for tests that need
a specific timeline. Property tests on the embedding math (magnitude
preservation under normalisation, cosine commutativity, never-NaN on
well-formed inputs) guard the invariants the matching layer in M4 will
rely on.
First CLI surface that actually does something. `pareidolia cameras` calls
`pareidolia_core::camera::v4l2::enumerate()` and renders the result as a
plain-text table with PATH / NAME / KIND columns. Read-only — enumeration
doesn't lock any device, so it's safe to run while the daemon or other apps
are using a camera.
clap drives argument parsing (derive style) so future subcommands compose
cleanly. `try_parse_from` rather than `parse_from` keeps the library
side from calling `process::exit` on --help / --version; we map clap's
exit-code-0 errors to Ok(()) at the dispatch point. main.rs now passes
`args_os()` as-is (program name included) rather than skipping it, which
matches what clap expects.
The CLI mirrors the daemon's feature shape: `v4l2` default-on, proxies to
core's `v4l2`. CI's `--no-default-features` disables it, in which case
the subcommand surfaces a clean error pointing at the build-time choice
rather than silently returning empty output.
Subcommand impls live in per-command submodules so each command's parsing
+ output + tests stay colocated and the surface of any single file stays
small. `format_table` is split out as a pure function so unit tests can
pin the output shape without touching stdout.
There's a discoverable gap between 'just ci-fast' (what's actually gated)
and 'every check a contributor should run before pushing'. Folding ci-fast
and ci-slow into a single 'just test' makes the pre-push expectation
explicit without forcing contributors to remember both names.
Hardware tests stay separate ('just test-hardware') so the recipe runs
identically whether or not the developer has a camera or other dev hardware
attached.
The v4l2 module is opt-in via the `v4l2` feature on pareidolia-core, and
its hardware-touching tests are `#[ignore]`d. Neither convention is
discoverable without docs, and 'just test-hardware' is the kind of recipe
contributors will forget exists if it isn't surfaced in CONTRIBUTING.
Also adds `test-v4l2` for running the v4l2 module's hardware-free tests
(FourCC mapping, enumerate-on-empty-host) with the feature enabled.
First hardware-touching FrameSource. Wraps the `v4l` crate to expose Linux
V4L2 capture through the same trait the mocks use. The two public entry
points cover the only operations the daemon needs:
- `enumerate()` walks /dev/video* and returns the cameras it can see.
- `V4l2FrameSource::open()` configures a specific device for capture.
# Exclusivity
V4L2 capture devices are exclusive while buffers are allocated: any other
process trying to open the same /dev/video* node gets EBUSY. The daemon's
intended pattern is therefore open-on-demand (per PAM auth request) and
drop-immediately-after so the camera stays available to other applications.
Drop cleanup must actually run — the v4l crate's `Stream<'a>` is
conservatively lifetime-bound to its `&Device`, so we use `self_cell` to
own both together and drop them in the right order (Stream first, then
Device).
# Feature shape
V4L2 is required for Pareidolia to actually run on a real system, so the
daemon's default feature set includes `v4l2` and the binary picks it up
out of the box. `pareidolia-core` keeps `v4l2` opt-in (default = []) so
consumers that only need the pipeline (mocks, format conversions) don't pay
the libv4l / bindgen build cost. CI passes `--no-default-features` to
disable v4l2 everywhere, which avoids needing libv4l in its Nixery image.
The workspace.dependencies entries set default-features = false on all the
internal crates so daemon's v4l2 doesn't transitively re-enable itself for
unrelated consumers (integration tests, the PAM module) through the dep
graph — a quirk of `cargo --workspace` feature unification.
A heuristic guesses IR vs RGB by checking whether the device advertises any
colour FourCCs (YUYV / RGB3 / MJPG). It's intentionally crude; the daemon's
config can override per device.
Rust 1.88 stabilised `u32::is_multiple_of` and Clippy 1.90's `manual_is_multiple_of`
lint promotes it as the preferred form over `x % n == 0`. The behaviour is
identical; the lint fires now because the toolchain bump exposed it.
A transitive dep of the soon-to-land v4l backend (`home` via `bindgen`)
declares MSRV 1.88, so 1.85 no longer resolves a complete dep graph. Bumping
to 1.90 gives us a small buffer above that floor without committing to the
absolute latest release.
The `v4l` crate generates its FFI layer via bindgen at build time, which
needs libclang plus the surrounding C / kernel-UAPI header search paths
set up correctly. `rustPlatform.bindgenHook` from nixpkgs is the
idiomatic way to provide all of that — it wires up LIBCLANG_PATH and
BINDGEN_EXTRA_CLANG_ARGS via the standard stdenv mechanism. Adding it to
nativeBuildInputs means the devShell handles bindgen out of the box.
`v4l-utils` covers libv4l itself. CI workflows aren't updated here because
the v4l backend lands feature-gated; CI keeps building the default-no-features
profile and so doesn't need any of this.
End-to-end check that capture and conversion compose through the public
re-export path the daemon and CLI will actually use: pull a Gray8 frame from
a mock source, run it through `camera::convert::to_rgb8`, verify the output
shape and that metadata (camera kind) propagates across the boundary. Catches
the regression where convert is exported but the path it's exported under
silently changes.
The downstream inference pipeline eats RGB8 exclusively after alignment, so
the dominant operation downstream of capture is 'turn whatever the camera
gave us into RGB8'. This module is that conversion surface.
Supported source formats: Rgb8 (identity), Gray8 (single-channel replicated),
Gray16 LE (high byte taken then replicated — sufficient for face recognition
input even when the low byte carries useful information), and Yuyv (BT.601
full-range, the JFIF flavour most consumer webcams produce). MJPEG returns
`UnsupportedFormat` until a JPEG decoder is wired in.
Correctness over throughput: YUYV uses f32 arithmetic for legibility. The
slowest target machine (a 2018 i5 XPS) easily absorbs the cost at typical
webcam resolutions; fixed-point is a profiler-led decision, not a blind one.
`MalformedBuffer` is added to `FrameError` so a corrupt capture surfaces
as a clean typed error all the way up to PAM (which then falls through to
password) rather than panicking the daemon.
Property tests guard the structural invariants — output length always matches
declared dimensions, grayscale conversions always replicate, and crucially,
the converter never panics on arbitrary buffer / dimension combinations.
That last property is what justifies trusting it on real V4L2 input where
malformed frames are a realistic failure mode.
Rust 1.75 predates the stabilisation of edition 2024, and the modern crates.io
graph (proptest's transitive deps in particular) increasingly requires it.
Bumping the pin to 1.85 — the first stable with edition 2024 — gets us onto
a current toolchain without committing to bleeding-edge versions.
`rust-version` in Cargo.toml is bumped in lockstep so MSRV stays honest.
`#[non_exhaustive]` is a real semver attribute, not just an annotation: it
prevents downstream crates from exhaustively matching on an enum, forcing
them to write wildcard arms. The intent is to let library authors add
variants in future versions without breaking consumers.
For Pareidolia today every consumer of these enums is another crate in this
workspace, so the trade-off inverts: wildcard arms in our own match
statements would silently absorb future variants instead of producing the
compile errors we actually want (e.g. 'you added Nv12 to PixelFormat but
the converter in daemon still doesn't handle it').
The flag existed only so the empty M1 skeleton didn't fail nextest's
default 'no tests is an error' behaviour. With real unit tests now present
in `pareidolia-core`, the guard is misleading: it would silently hide a
future regression where the test target stops emitting any tests at all.
Unit tests inside `pareidolia-core` reach in via `super::` and so can't
catch the case where the public re-export surface accidentally hides a type
a downstream crate needs. These cross-crate tests are explicitly that
backstop: they exercise the same mocks but only through the public path the
daemon and CLI will use.
Lives as a module under the single integration test binary, per the workspace
convention documented in CONTRIBUTING.md.
Establishes the camera abstraction the rest of M2 (pixel format conversions,
V4L2 backend, camera enumeration) fills in.
Decoupling the inference pipeline and daemon from V4L2 via a trait — and
shipping mock implementations alongside the trait itself — means every
downstream consumer is testable without hardware. The mocks live outside
`#[cfg(test)]` because integration tests, the daemon's own test harness,
and (eventually) NixOS VM tests all need them too.
Three mock flavours cover the test surface:
- StaticFrameSource for the common 'same frame every time' case
- ScriptedFrameSource for scenarios that need a specific timeline of
successes and failures
- FailingFrameSource for fault injection (camera unplugged, busy, etc.)
`thiserror` is pulled into `pareidolia-core` to keep `FrameError`'s
variants declarative.
Adds a "Liveness detection (model-free PAD)" section to CONTRIBUTING.md
covering: the evidence/veto composition (noisy-OR capped by veto product,
why not a weighted average, veto-only and fail-closed semantics, the
NotApplicable additivity that makes strobing structurally more secure);
the NIR reflectance gate; the removed motion-magnitude dead end and its
intended planar-homography replacement; the mandatory observe-only-until-
tuned posture and the `calibrate-liveness` workflow; and honest scope
(casual vs determined attacks, ISO/IEC 30107-3 per-attack evaluation).
Updates the "future milestone" PAD note to reflect what now exists, and
extends the defense matrix with a tuned-liveness column.
The guided live-vs-spoof profiling workflow that turns the liveness
checks' guessed thresholds into data-derived ones. Incremental and
GUI-shaped — each invocation does one thing, accumulating into a
persisted profile:
- `capture <class> [--label .. --append --conditions ..]` — capture a
labeled batch (live / printed-photo / screen / other). Opens the IR
camera, runs the same detect → liveness-policy pipeline auth uses,
harvests each check's sub-score `detail` into a feature vector, and
folds the batch into the profile (replacing the class, or appending).
- `status` — per-class sample counts.
- `report` — per-feature live-vs-spoof separation (means ± std, d-prime).
- `suggest-threshold` — per-feature threshold separating live from each
spoof, with the false-reject / false-accept it would incur.
- `reset [<class>]` — clear all, or one class.
Each verb is one core operation, so a future GUI can drive the same
flow — and because core ingests caller-supplied frames/features, a GUI
that owns the camera for preview can feed its own frames rather than
contend for the device.
Features are namespaced by check (e.g. `nir_reflectance.brightness`) so
multiple checks' sub-scores don't collide. A no-face capture (e.g. a
screen that's dark in IR) is skipped and counted — and an all-skipped
batch errors with a note that no-face is itself the dual-modality
defense. Gated on v4l2+inference with an arg-compatible fallback;
verified the --no-default-features build still compiles.
`LivenessProfileStore`: a single-file CBOR store for the accumulating
guided-calibration profile. Like the camera calibration store and unlike
the enrollment store, no HMAC — it holds tuning data, not secrets, and
liveness degrades gracefully on an absent/unreadable profile. Atomic
writes (temp + fsync + rename) so a crash never leaves a torn file;
versioned (refuses unknown future versions). `load` returns None when
absent (uncalibrated, normal), `load_or_default` for read-modify-write
verbs, plus `save`/`remove`.
Tested: missing-is-None, save/load round-trip, idempotent remove, no
temp-file leakage, an arbitrary-profile round-trip proptest, and a
16-thread concurrent-writer stress test asserting the file is never torn
(every post-race load decodes; last-writer-wins on content).
Property tests over the profile analysis:
- FeatureStats: min <= mean <= max, std >= 0, exact count, for any
finite value set.
- suggest_threshold: reported error rates are always valid fractions in
[0,1]; and — the headline invariant — the chosen threshold *minimises*
total error (live-reject + spoof-accept) over the sample set: no other
observed cut point does better. That's the guarantee that the
suggester actually picks the best separating threshold.
- separation: its sign follows the class-mean ordering; analysis never
panics on arbitrary data.
The data side of guided liveness calibration: an accumulating profile of
labeled samples plus pure analysis over it. No hardware — a sample is a
named feature vector (the CheckEvaluation `detail` a check produces) with
a class label, so the caller supplies samples (CLI from V4L2, a future
GUI from frames it owns). This keeps the camera-ownership boundary clean:
core never opens a device.
- `SampleClass`: Live / PrintedPhoto / Screen / Other(label) — flexible
per-attack-type labeling for ISO-30107-3-style reporting.
- `LivenessSample`: { class, features: Vec<(name, value)>, timestamp }.
Stores computed features, not raw frames — privacy (no face images in
a non-HMAC file), size, and check-agnostic extensibility. Format
versioned so raw-frame retention could be added later.
- `LivenessProfile`: accumulate via `replace_class` (default re-capture)
or `append_batch` (--append), plus `clear_class`, `class_counts`,
`feature_names`. Carries a free-form `conditions` note since a profile
is only valid for similar capture conditions.
- Analysis: `FeatureStats`, `separation` (d-prime-like class separation
per feature), and `suggest_threshold` (sweeps cut points to minimise
live-reject + spoof-accept, orienting correctly whether live reads
higher or lower than the spoof). Suggestions only — never auto-applied.
Unit-tested: accumulation semantics, stats, separation magnitude, and
threshold suggestion on cleanly-separable / overlapping / inverted-sense
data.
A policy with only veto checks and no evidence check — exactly our
current NIR-gate-only config — always scored combined = 0, because
combined = noisy_or(evidence) × ∏(veto) and noisy_or([]) = 0. So a live
face the NIR veto scored 0.94 came out as 0.000. (Surfaced in testing: a
clearly-live face logged nir_reflectance 0.94 but Liveness combined
0.000.)
`assess` now composes by *what applied*:
- nothing applicable (no evidence, no veto) → 0.0, so a positive
threshold fails closed (operator enabled enforcement but nothing ran);
- veto(s) but no evidence → the veto product alone (a veto-only policy
reduces to the vetoes' verdict — the fix);
- evidence present → noisy-OR of evidence, capped by vetoes.
Because present evidence caps *further* than vetoes alone, the veto-only
neutral baseline never weakens security; it only stops an empty
noisy-OR from zeroing a working veto. `evidence` is still reported as
the raw noisy-OR for diagnostics.
Tests: veto-only reduces to the veto score (and still zeroes a spoof),
and "nothing applicable" stays fail-closed-at-positive-threshold,
distinct from veto-only.
A check's single combined score hides *why* it landed there — e.g. a
genuine face scoring 0.69 on the NIR gate could be limited by any one of
its brightness/structure/smoothness sub-scores, and you can't tune or
trust a threshold without knowing which.
`LivenessCheck::evaluate` now returns a `CheckEvaluation { outcome,
detail }`, where `detail` is a list of named `(sub-score, value)` pairs —
purely diagnostic, never affecting composition. `CheckScore` carries it
through to the assessment, and `test`'s liveness output prints the
breakdown indented under each check.
The NIR gate reports its three sub-scores (brightness/structure/
smoothness) plus the raw measurements (mean/std/roughness), so a
borderline live result is interpretable and guided calibration has the
per-feature data it needs per attack type.
Chose the explicit return-struct refactor over a recompute-on-demand
`detail()` method: single evaluation, detail is first-class. Checks with
no breakdown use `CheckEvaluation::bare`.
The strongest, cheapest model-free liveness signal on a NIR sensor, and
the research-recommended first check. A single-frame veto on the face
region exploiting physics RGB can't:
- Screens emit ~no near-IR → a displayed face is dark and flat under the
sensor → vetoed.
- Skin has a characteristic NIR reflectance and sub-surface scattering →
a real face is bright with rich but diffusely-smooth structure; paper
reflects differently (flatter, or halftone-textured).
`NirReflectanceGate` (role: Veto) combines three face-region features
into a confidence-live score, taking their *minimum* so an attack need
fail only one to be vetoed and a genuine face must satisfy all:
1. brightness ≥ `min_mean` (screens too dark);
2. structural contrast (std) ≥ `min_std` (flat surfaces fail);
3. diffuse smoothness — normalised mean |Laplacian| ≤ `max_roughness`
(print halftone / screen grid add high-frequency texture).
It measures the detected face region (via the plumbed bbox), not the
whole frame, so it doesn't confuse a dim room with a screen; it abstains
(`NotApplicable`) when there's no face box rather than veto on a weak
signal. Registered in `test`'s policy.
`face_region` is extended to handle `Rgb8` as well as `Gray8`: the IR
capture path delivers RGB8 with the grey value triplicated into R/G/B,
so a single channel is the intensity. Without this the gate saw RGB8
frames and always returned NotApplicable (the `n/a` it reported in
practice) — so this is what makes the gate actually run.
Tested: dark/flat screen-like region vetoes (~0), bright smoothly-shaded
region passes, high-frequency-textured region is penalised, score in
[0,1], and RGB8 intensity extraction. Thresholds are first-cut estimates
to be tuned by guided calibration; liveness stays observe-only
(threshold 0.0) until then.
Face-region liveness checks (the NIR reflectance gate next) need to
measure *where the face is*, not the whole frame — otherwise they
conflate "bright lit face" with "bright background, dark face" (a screen
showing a face) or with a dim room.
`LivenessInput` gains `face_bbox: Option<BoundingBox>`, and a
`face_region(frame)` helper that crops a single-channel (Gray8/IR) frame
to that box, clamped to frame bounds (SCRFD boxes can spill past the
edge), returning the region pixels + dimensions. Shared plumbing so each
face-region check doesn't re-implement clamping.
`IrCapture::liveness_input` populates `face_bbox` from the matching
outcome's detected face (None when no face was found, so a check falls
back to whole-frame or abstains). Tested: crop dimensions, out-of-bounds
clamping, and the no-bbox case.
Magnitude of inter-frame change is not a liveness signal. On real
hardware a live face scored 0.3-0.6 while a phone screen saturated at
1.0 (flicker reads as huge "motion") and a moving printed photo ranged
0-1 — i.e. the metric was anti-correlated with liveness, passing the
spoofs and nearly rejecting the genuine user. This matches the PAD
literature's consistent finding that liveness lives in the *structure*
of signals (spatial texture, motion geometry, spectral reflectance), not
the *magnitude* of any single quantity.
So `MicroMotionCheck` is removed rather than kept as misleading
near-useless code. `liveness_policy()` is now an empty no-op policy
(combined = 0.0, passes at the default threshold 0.0), pending the real
checks the research points to — a NIR-reflectance veto gate next, then a
structure-aware planar-homography motion check. The two-role framework
and all the capture/burst plumbing remain; only the wrong-feature check
goes. Planned-checks docs updated accordingly.
Liveness checks now declare a `CheckRole`: positive **evidence** of life,
or a spoof **veto**. The policy composes them differently:
evidence = noisy_or(evidence check scores) // raises confidence
combined = evidence × ∏(veto check scores) // veto caps it
This is needed because some signals are evidence of an *attack*, not of
life, and noisy-OR (which ignores low scores) would render them inert. A
veto check still outputs "confidence the subject is live" in 0..1, but
*low* when it detects an attack signature; multiplying caps the final
score, so a confident veto (≈0, "this is a screen") overrides even
strong evidence. An inapplicable veto isn't in the product and caps
nothing, preserving additivity for veto checks too.
`role()` defaults to `Evidence`, so existing checks are unaffected;
spoof detectors override it. `LivenessAssessment` gains `evidence`
(pre-cap) alongside `combined` (post-cap) for diagnostics. The old
`combine` is renamed `noisy_or` to name what it actually does now that
it's only the evidence half.
Proptested: noisy-OR bounded/monotonic/single-identity as before, plus
the defining veto property — adding a veto never raises the final score
— and the combined score stays in [0,1] for any evidence/veto mix.
(The first real checks land next: the micro-motion check is being
removed as a proven dead end — magnitude-of-change is anti-correlated
with liveness, per both measurement and the PAD literature — and
replaced by a NIR-reflectance veto gate.)
`test` now runs the liveness policy against the IR capture and reports
it. The IR modality captures inline (via the new `capture_ir` helper)
instead of through the RGB retry path, so the full burst is retained and
handed to the policy as a `LivenessInput`. The burst is sized with
`liveness_burst` so the micro-motion check has enough same-phase frames.
Output gains a `Liveness:` line with the combined score, the threshold,
the applicable/total check counts, and a per-check breakdown (including
`NotApplicable` lines, so it's visible why a sensor's ceiling is what it
is). A new `--liveness-threshold` flag (default 0.0) gates the verdict:
a passing identity match is downgraded to `Reject (liveness)` if the IR
capture didn't clear the threshold. At the default 0.0 this never
fires — liveness is observe-only until profiled, exactly the posture for
untuned heuristics.
Also simplifies the old `try_capture_face` into an RGB-only
`capture_rgb_face`: the IR branch and its `kind`/`ir_burst` params became
dead once IR moved to the inline burst-retaining path, so they're
removed rather than left as misleading cruft.
Liveness is wired on IR here, but the framework is modality-agnostic;
RGB liveness (retain RGB frames, run the same policy) is a documented
future step.
Micro-motion (and future multi-frame liveness checks) need several
same-illumination-phase frames, but the calibrated matching burst is
minimal — one strobe period, i.e. a single lit frame on a period-2
sensor. `liveness_burst(matching_burst, depth)` grows it to
`matching_burst * depth` so the burst spans `depth` periods and yields
`depth` same-phase frames (clamped to depth >= 1; saturating, no
overflow). `DEFAULT_LIVENESS_DEPTH` is 3.
Pure helper with unit tests; the `test` command wires it in when it runs
liveness (next commit). Matching-only paths keep using the minimal burst.
The first real liveness signal: scores involuntary frame-to-frame change
across the capture burst. A live subject always moves a little over the
~100-200ms a burst spans (head sway, micro-expression, blink); a printed
photo or a screen held up is rigid, differing only by sensor noise. So
mean per-pixel change is a model-free liveness signal — and it works on
*any* IR sensor, which is why it's the first check.
# Same-illumination comparison
The one subtlety is strobed sensors: their burst alternates lit/dark
frames, so diffing consecutive raw frames would read the strobe as huge
bogus "motion". The check therefore compares only same-illumination-phase
frames — those within `phase_band` of the lit frame's brightness. On a
constant sensor every frame qualifies (the grouping collapses to "use
all"), which is what makes the check universal. A test pins that the
strobe's dark frame is excluded.
# Applicability
Needs >= 2 same-phase frames; with fewer it returns `NotApplicable`, not
a bogus score. A minimal period-2 matching burst (one lit frame) is
therefore NotApplicable — liveness-aware burst sizing (next commit) is
what gives it enough same-phase frames. The additive model means this
just contributes nothing until then, never breaks anything.
# Scoring
Normalised mean-absolute per-pixel difference between consecutive
same-phase frames, saturating to 1.0 at `MicroMotionConfig::saturation`.
The constants are first-cut estimates to be tuned from real
live-vs-spoof captures (guided calibration), which is why liveness stays
observe-only (threshold 0.0) until then.
Proptested: total + bounded over arbitrary frames, and monotonic in
motion (more per-pixel change never lowers the score). Also records the
planned complementary checks (embedding-agreement, texture, differencing)
in the module docs.
`capture_ir_face` previously discarded every burst frame except the
chosen lit one. It now returns an `IrCapture` that owns the whole burst
plus the lit-frame index and an optional dark-frame index, with a
`liveness_input()` accessor producing a `LivenessInput` for assessment.
The dark frame is identified *from the data*, not from calibration, by
`detect_dark_frame`: the dimmest burst frame, reported (`Some`) only
when it's at least `STROBE_DARK_CONTRAST` (25/255) darker than the lit
frame — i.e. the sensor is actually strobing. On a constant-brightness
sensor all frames are similarly bright, so `dark_index` stays `None` and
any strobed-only liveness check gates itself off. Strobed differencing
stays additive, never required.
`detect_dark_frame` is deliberately a small isolated function with a
prominent KNOWN-LIMITATION note: its absolute brightness gap is only
valid under the (indoor) ambient conditions it was tuned in. Outdoors,
broadband sunlight floods both lit and dark frames with near-IR, shrinks
the absolute gap, and could disable differencing on a genuinely strobed
sensor. The proper fix (scale-invariant detection, or deferring the
strobed-vs-constant verdict to calibration) is not attempted yet — no
daylight IR captures exist to tune against. The decision is quarantined
here so the future change is local; callers see only the `Option<usize>`,
and a wrong `None` degrades safely.
Pure plumbing otherwise: `enroll` and `test` take `.outcome` and behave
exactly as before; no liveness check runs yet.
Introduces liveness / presentation-attack detection as a confidence axis
orthogonal to identity matching, structured so it works on any IR camera
and treats a strobed illuminator as an additive bonus rather than a
prerequisite.
# Shape
- `LivenessCheck` trait: each check inspects a `LivenessInput` (the full
captured burst, the chosen lit frame's index, and an optional dark
frame index for strobed sensors) and returns `CheckOutcome` — either
an `Applicable(score)` in 0..=1 or `NotApplicable`. A check that needs
hardware it didn't get (e.g. differencing on a constant sensor)
returns `NotApplicable`, never a low score, so adding it can't break
unsupported hardware.
- `LivenessPolicy` runs all checks and combines applicable scores via
noisy-OR (`1 - ∏(1 - sᵢ)`), producing a `LivenessAssessment` with the
combined score, the applicable count, and a per-check breakdown for
logging.
# Why noisy-OR, not a weighted average
Two properties the security model needs, both proptested:
1. No inflation: one applicable check of score `s` combines to exactly
`s`. A weighted average over *applicable* checks would renormalize a
lone weak signal up to full-range confidence, erasing the difference
between a constant sensor (few checks) and a strobed one (more). With
noisy-OR a `NotApplicable` check simply isn't in the product.
2. Monotonic: more applicable evidence never lowers the score, so a
strobed sensor's extra check can only *raise* its ceiling. Strobing
being more secure is thus structural, not a tuning artifact.
# No-op by default
An empty policy assesses `combined = 0.0`; `passes(threshold)` with the
default `threshold = 0.0` passes everything — exactly today's behavior,
liveness effectively off until checks are registered and a threshold is
chosen. A positive threshold with zero applicable checks fails closed.
Pure and total (clamps out-of-range / non-finite scores); no hardware,
no models. Real checks and the auth-path wiring land in follow-up
commits.
A targeted testing pass over the pure logic added across the IR-capture
and calibration work, plus one latent bug it surfaced.
# Bug fix: strobe period truncation
`classify_ir_mode` computed the strobe period as `(w[1] - w[0]) as u8`,
which *truncates* (wraps mod 256) when bright frames are more than 255
indices apart — a long, sparse burst could classify as a bogus tiny
period. Now uses a saturating `u8::try_from(...).unwrap_or(u8::MAX)`. A
regression test pins it (gap of 300 → 255, not 44), and was confirmed to
fail against the old code.
# Property tests
- `select_brightest`: result is an in-bounds true argmax; `Some` iff
the input is non-empty.
- `classify_ir_mode`: total (never panics) over arbitrary `&[f32]`
including NaN/inf/empty; a synthesized period-`p` strobe round-trips
to `Strobed { period: p }`; a sub-contrast stream is always
`Constant`; the result's `min_burst()` is always >= 1 and any strobed
period >= 2.
- Calibration store: `save_all`→`load` round-trips an arbitrary record
set (mirrors the enrollment store's headline property); `upsert`
leaves exactly one record per identity carrying the new mode.
- `resolve_calibration`: total; `Exact` implies an identical full
identity; any non-`None` match implies a same-card+kind record exists;
deterministic.
- `resolve_ir_camera` / `resolve_rgb_camera`: explicit path always wins;
total; an adopted camera is always of the matching kind; RGB adoption
happens exactly when there's a single path-bearing RGB camera.
# Stress test
Calibration store: 16 threads racing `upsert` on one file. The
read-modify-write isn't serialized (last-writer-wins on the record set,
as documented), so the test asserts only *integrity* — atomic
temp+rename means every post-race `load` succeeds on a structurally
valid, non-torn file and two consecutive loads agree.
RGB selection relied on a hardcoded `--rgb-camera` default of
`/dev/video0` — a guess that silently breaks when the RGB camera isn't
video0, and an inconsistency with IR, which auto-detects. Now both
modalities resolve the same way.
# core
`resolve_rgb_camera` + `RgbCameraResolution` mirror the IR resolver, with
RGB's mandatory-modality semantics: no opt-out, and the zero-camera case
is a hard `NoHardware` failure rather than a benign skip. Explicit path
wins; otherwise the single RGB-classified device is adopted, with zero
or several surfaced as `NoHardware` / `Ambiguous` for the caller to
render. Pure and unit-tested with synthetic snapshots, including that IR
cameras don't count toward RGB ambiguity.
# cli
`--rgb-camera` becomes optional (no default). `enroll` and `test` resolve
it through the shared `pipeline_runtime::resolve_required_rgb`, which
renders the outcome to a path or an actionable error ("pass --rgb-camera
<PATH>"). On a typical laptop with one RGB node this just works with no
flag; a machine with several RGB cameras now gets a clear "pick one"
error instead of silently grabbing video0.
Integrated webcams routinely expose their IR and RGB nodes under the
*same* V4L2 card name (the XPS 13 reports both as
`Integrated_Webcam_HD`). With calibration keyed on card + bus alone, an
IR record and the RGB sibling on the same bus were indistinguishable,
and the RGB node counted as a competing candidate that could make an IR
record's port-move resolution spuriously `Ambiguous`.
`CameraIdentity` gains a `kind` field, and `resolve_calibration` matches
on card + bus + kind: exact matches can't cross modalities, and the
same-card/different-bus (`MovedPort`) path filters candidates by kind so
a shared-card-name sibling of the other modality is ignored.
The field is `#[serde(default)]` to `Ir`, so calibration files written
before it existed — which only ever recorded IR cameras — still
deserialise. Documents the remaining limitation: two identical-model
cameras of the same kind on one machine are still distinguished only by
bus, and a same-kind duplicate that also moved ports falls back to a
mode-agnostic burst.
`test` built its `AuthMode` straight from `--ir-camera`, so omitting the
flag silently dropped to single-RGB auth even on a machine with a
working, auto-detectable IR camera — the same camera `enroll` and
`calibrate` find on their own. That mismatch is why dual-mode auth
appeared to require the flag.
`test` now resolves IR the same way `enroll` does, via
`resolve_ir_camera` (explicit flag > `--no-ir` opt-out > auto-detect):
an auto-detected or explicit IR camera runs dual-modality auth; no IR
hardware (or `--no-ir`) falls back to single RGB; multiple IR cameras
warn and fall back rather than guess. Adds `--no-ir` to `test` to mirror
`enroll`'s flag set so RGB-only is still reachable on dual-capable
hardware.
This also aligns `test` with what the M5 daemon's auth path must do —
resolve cameras from detection/config, not from a per-invocation flag.
Closes the calibration loop: `enroll` and `test` now read the calibration
store and size their IR capture burst from the detected
`IrCaptureMode::min_burst` instead of always using `DEFAULT_IR_BURST`. On
the strobed XPS 13 that drops the burst from 4 frames to the minimal 2,
shaving capture time; an uncalibrated or unmatched device transparently
keeps the safe default.
`pipeline_runtime::resolve_ir_burst` does the resolution via
`resolve_calibration`, handling every `CalibrationMatch`:
- `Exact` / `MovedPort` — use the calibrated burst. `MovedPort`
(device replugged to another port, card name still unique) also
refreshes the stored record's bus so the next run is an `Exact` hit.
- `Ambiguous` — several identically-named cameras connected; warn and
fall back to the default burst.
- `None` — uncalibrated; use the default silently.
Calibration being an optimization, every failure path (unreadable store,
ambiguous match, write-back failure on `MovedPort`) degrades to the
default burst with at most a warning, never an error.
`EnrollmentStore::root()` is added so the CLI can locate the sibling
calibration store in the same directory without re-resolving flags.
Measures an IR camera's illuminator emission mode once and persists it,
so subsequent IR captures can size their burst from the known mode
instead of always over-capturing. Gated on `v4l2` (it captures frames)
but not `inference` — it only measures per-frame brightness, never runs
detection.
Resolves the IR camera with the same precedence as `enroll` (explicit
flag > opt-out > auto-detect) so the two can't pick different cameras,
captures a 30-frame burst, classifies the brightness sequence via
`classify_ir_mode`, and upserts an `IrCalibration` record keyed by the
camera's card+bus identity into the calibration store (alongside the
enrollment store root). Uses `replace_by_card` so re-calibrating after a
port move doesn't leave a stale-bus record behind.
Prints the detected mode (Constant, or Strobed with its period) and the
observed brightness range so the result is legible and the measurement
auditable.
Calibration records how a given IR camera delivers illuminated frames so
capture can be optimized per device. It is purely an optimization and
capability layer: `capture_ir_face` already works without it (capture a
burst, keep the brightest), so a missing/unmatched/corrupt record just
falls back to the safe mode-agnostic burst — never an error. That's why
this store, unlike the enrollment store, needs no HMAC.
# Device identity (card + bus)
Records are keyed by `CameraIdentity { card, bus }`. Neither field alone
is a usable key: `/dev/videoN` renumbers across reboots, the card name
isn't unique across identical models, and the bus (port topology)
changes when a device is replugged elsewhere. `resolve_calibration`
matches leniently, returning a typed `CalibrationMatch`:
- `Exact` — card + bus both match.
- `MovedPort` — card matches, bus differs, and exactly one connected
camera has that card name, so the device just moved ports: safe to
adopt and refresh the stored bus.
- `Ambiguous` — card matches but several connected cameras share that
name, so we can't guess which: the caller should ask the user.
- `None` — no record; fall back to mode-agnostic capture.
This mirrors the decisions-in-core / rendering-in-CLI shape of
`camera::select::IrCameraResolution`.
# Store
`CalibrationStore` persists all records in one small CBOR file (the set
is tiny and the matching API wants it whole). Writes are atomic
(temp-file + fsync + rename), matching the enrollment store's crash
discipline. `upsert` replaces by exact identity; `replace_by_card` drops
a stale-bus record when a device moved ports.
# CameraInfo.bus
`CameraInfo` gains a `bus: Option<String>`, filled by `enumerate()` from
the V4L2 bus info and left `None` for mock/synthetic sources and the
by-path `open()` path. This is the input `CameraIdentity::from_info`
reads. Calibration shares the `v4l2` feature gate since it depends on the
backend's `IrCaptureMode`.
Wires both subcommands to the strobe-tolerant `capture_ir_face` so IR
enrollment and IR auth verification stop failing on dark strobe frames.
# Root cause
The IR sensor on the 2018 Dell XPS 13 strobes its illuminator: it emits
a strictly alternating lit/dark frame sequence (measured: lit ~76, dark
~11, period 2, indefinitely). A single IR shot therefore had a ~50%
chance of landing on a black frame, which is exactly the intermittent
"IR: no face detected" seen post-enrollment. `capture_ir_face` captures
a short burst and keeps the brightest frame, which is always the lit one
regardless of strobe phase.
# enroll
The IR modality now captures a `DEFAULT_IR_BURST` burst per sample via
`capture_ir_face` instead of a detect-on-every-frame retry loop, dumping
every burst frame under `--debug-dump-dir` (lit and dark alike) and
mapping an all-dark burst to `IrTooDark`. Drops the now-unused
`IR_DETECT_ATTEMPTS`.
# test
RGB keeps the bounded detection retry (good for blinks/pose); IR uses
the brightest-of-burst capture. A failing IR modality whose frames
stayed black reports that the illuminator likely never fired rather than
a bare "no face". `--debug-dump-dir` (added here) writes
`{rgb,ir}-attempt{n}.ppm`, so the auth path is diagnosable the same way
enrollment is.
Builds on the core IR-selection primitives to give the CLI capture
helpers the enroll/test commands need.
# IR capture (strobe-tolerant)
`capture_ir_face` captures a short *burst* of IR frames, picks the
brightest via `select_brightest`, and runs detection only on that. This
is the fix for the strobed XPS 13 sensor: because the emitter alternates
lit/dark frames, "the brightest frame in the burst" is the illuminated
one regardless of where in the strobe cycle capture began — no luck, no
per-frame detection guesswork. The burst length is `DEFAULT_IR_BURST`
(4) when the mode is unknown; a later calibration pass can size it
exactly (1 for constant, `period` for strobed).
# RGB bounded retry
`capture_one_face_retrying` (unchanged in spirit) retries detection
across a bounded number of frames so a blink or unsettled pose doesn't
fail a modality. The bound is deliberate — this runs on the auth/test
hot path, where an unbounded "loop until a face appears" would hang the
login flow whenever no one is at the camera. `RetryCaptureOutcome`
distinguishes `Found` from `NoFace { attempts, brightest }`.
# Open path + diagnostics
`open_camera` now uses `Warmup::Frames(1)` for both kinds — purely the
STREAMON priming discard, with IR frame selection delegated to
`capture_ir_face` rather than a (now-removed) brightness-gated warm-up.
`frame_mean_brightness` becomes a thin wrapper over core's
`frame_brightness`. `DARK_FRAME_MEAN_THRESHOLD` + `CliError::IrTooDark`
turn an all-dark burst (illuminator never fired) into an actionable
message instead of a bare "no face".
IR cameras with an active illuminator deliver illuminated frames in
ways that vary by hardware. Measuring the 2018 Dell XPS 13 IR node
showed it is not a slow-ramp emitter (an earlier theory) but a *strobed*
one: it is lit from its first frame and then alternates lit/dark frames
forever, the emitter pulsed every other frame.
That makes "discard frames until one is bright" actively wrong — under
strict alternation, stopping warm-up on a bright frame guarantees the
*next* captured frame is the dark twin. Frame selection, not stream
warm-up, is the right layer for this.
This commit provides the mode-agnostic primitives the capture layer
needs, all pure and unit-tested without hardware:
- `frame_brightness` — mean of raw frame bytes, a cheap luminance proxy
(the GREY IR bytes already are luminance).
- `select_brightest` — index of the brightest frame in a burst. Whatever
the emission pattern (constant, strobed, ramping), the brightest frame
in a long-enough burst is a usable illuminated one, so this alone is
enough for correct IR capture without knowing the mode.
- `IrCaptureMode` { `Constant`, `Strobed { period }` } + `classify_ir_mode`
— classify the emission pattern from a run of frame brightnesses. This
is an optimization/capability signal (size the capture burst minimally;
unlock future ambient-subtraction/liveness on strobed pairs), never
required for correctness.
`Warmup` loses its brightness-gated `UntilBright` variant and is now
purely a fixed priming-frame discard (`Frames(n)`), correct for every
sensor kind. `mean_brightness` moves out of `Warmup` to the free
`frame_brightness` function its several consumers share.
We're committing to a CLI design where the project stays ready for
additional UI layers (GUI, TUI, RPC client) without requiring the CLI
to be untangled. The rule that falls out, recorded in CONTRIBUTING.md
§ UI architecture: decisions live in `pareidolia-core` as pure
functions returning structured outcomes; rendering lives in
`pareidolia-cli`. Each match arm in a subcommand should be either
'translate this variant to its display string' (rendering) or 'call
this core function and dispatch on its outcome' (still rendering at
the CLI layer, with the actual decision made one frame down). It
should not be 'compute the next state inline'.
This commit extracts four meaningful instances of inline decision
logic from CLI commands into typed outcomes in core, and documents
the pattern so future contributors don't have to re-derive it.
# core: `camera::select::IrCameraResolution` + `resolve_ir_camera`
The IR-camera resolution that lived inline in `pareidolia enroll`
(explicit flag wins, then opt-out, then auto-detect with three
sub-branches) becomes a five-variant enum returned by a pure
function over `&[CameraInfo]`.
Unit-testable without V4L2: ten tests cover each branch and the
precedence ordering (explicit > opt-out, opt-out > auto-detect,
multi-IR collapses to Ambiguous, defensive handling of a path-less
`CameraInfo`). A future GUI would call `resolve_ir_camera` with a
list it owns and render the same five variants as widgets. The
daemon will use the same function with a hardware snapshot pulled
per auth request.
`enroll.rs` now contains only a `match` over the five variants
that picks the appropriate stderr notice and extracts the resolved
path.
# core: `enrollment::AuthMode`
Two-variant enum (`Single { rgb_camera }` / `Dual { rgb_camera,
ir_camera }`) for the configured authentication flow. CLI builds
one via `AuthMode::from_optional_ir(rgb, ir_opt)`; the daemon will
build its own from config + per-user persisted paths in M5. Both
will then call `authenticate` or `authenticate_dual` based on the
variant.
Replaces the inline `match &args.ir_camera` dispatch in
`pareidolia test`. The decision and the camera-path data now live
in one place instead of being threaded through args throughout the
flow.
# core: `enrollment::EnrollmentReadiness` + `from_load_result`
Three-variant classification of a `store.load(user)` result:
`NotEnrolled` (no enrollment file), `Empty` (file exists, zero
samples — distinct because the user *was* enrolled at some point),
`Ready(Enrollment)`.
`from_load_result` folds the `StoreError::NotEnrolled` variant
into a non-error `NotEnrolled` readiness state since 'no enrollment
yet' is normal flow control, not a failure. Genuine I/O / decode
errors stay as `Err`.
`pareidolia test` now calls this exactly once at the top of the
flow; the daemon (M5) will call it on every PAM auth request as
the canonical 'should we open the camera or short-circuit?' check.
Four unit tests cover the three readiness states plus error
propagation.
# core: `enrollment::EnrollmentSessionSummary`
Per-session counts (`new_rgb`, `new_ir`, `total_rgb`, `total_ir`,
`user`) returned by the enroll loop for rendering. CLI picks
between two text shapes based on `summary.had_ir()`; a GUI will
populate widgets from the same struct.
A small extraction, included alongside the larger three for two
reasons: it demonstrates the value-struct flavour of the pattern
(not every extraction needs to be an enum), and it cleans up the
last loose `if new_ir > 0` decision-vs-render mix in the enroll
flow.
# Documentation
CONTRIBUTING.md gains a `## UI architecture` section between
'Testing philosophy' and 'Security' that captures the convention,
points at `IrCameraResolution` as the canonical worked example,
lists what already follows the pattern and what doesn't yet, and
records the constraints we will *not* break (no interactive
prompts, no spinners/colour/TUI, no `init` wizards) — those would
make the CLI hostile to scripting and impossible for a non-terminal
UI to reuse anyway.
Two pieces of the dual-modality story were still missing after M4.7:
1. **Default behaviour didn't take advantage of available IR hardware.**
Users had to remember to pass `--ir-camera /dev/videoX` to enroll
the IR pool, and most never would. The recommended security posture
(dual-modality, defeats the photo-spoof attack) was opt-in rather
than opt-out.
2. **The daemon (M5) needs to know which physical device captured which
pool to reopen the same hardware on the PAM auth hot path** without
re-running the `enumerate()` heuristic for every auth request, and
without risking picking the wrong camera on systems with several.
# Auto-detect when --ir-camera is unset
`pareidolia enroll` now resolves the IR path via:
- `--ir-camera <PATH>` → use exactly that.
- `--no-ir` → skip IR; never auto-detect.
- Neither set → call `enumerate()`:
- 0 IR-classified cameras → silent RGB-only.
- 1 IR-classified camera → adopt it, print a one-line notice
naming the path and the `--no-ir` opt-out.
- 2+ IR-classified cameras → warn and skip IR (refuse to guess),
pointing the user at `--ir-camera` or `--no-ir`.
The single-IR auto-adopt is the common case (laptop built-in IR
sensor, or a single USB IR module). The multi-IR ambiguity branch
exists because silently picking the wrong sensor would pollute the
pool with samples from a different physical camera than the daemon
will later use at auth time, and the user would have no idea why
authentication felt unreliable.
`--no-ir` is mutually exclusive with `--ir-camera` (clap enforces).
Both flags are documented in `--help`.
# Persisted camera paths in Enrollment
`Enrollment` gains two optional fields:
pub rgb_camera_path: Option<PathBuf>,
pub ir_camera_path: Option<PathBuf>,
Both use `#[serde(default, skip_serializing_if = "Option::is_none")]`,
which has two consequences worth pinning explicitly:
- **Forward-compat for existing files**: enrollments written by
Pareidolia versions without these fields decode cleanly into the
new shape with `None` values. `encode_with_no_camera_paths_omits_keys_and_decode_yields_none`
verifies the on-disk payload is byte-identical to the old format
when paths are `None` (by asserting the field-name strings do not
appear in the encoded CBOR), so existing user enrollments survive
the upgrade untouched.
- **Round-trip preserved when set**: `encode_then_decode_preserves_persisted_camera_paths`
exercises the populated path through encode/decode and back.
The enroll loop writes the RGB path unconditionally (we always capture
RGB) and the IR path only when IR enrollment actually happened this
session, so a later `--no-ir` re-run doesn't clobber a previously
persisted IR path. Stale paths are harmless: the daemon will check
sample presence first and fall back to `enumerate()` if the persisted
device is gone.
# Resolution logic lives inline in enroll.rs for now
The IR-camera resolution match is currently inline in
`cli::commands::enroll::run`. A separate follow-up commit will lift
this kind of decision logic into `pareidolia-core` as structured
outcome types (the "decisions in core, rendering in CLI" pattern
we want to establish across all subcommands so a future GUI can sit
on the same primitives). Keeping the architectural refactor in its
own commit makes the user-visible feature here easy to review and
revert independently.
Single-modality face auth is identity-matching, not liveness detection.
ArcFace is trained to cluster the embedding space by identity; a sharp
photo of a face produces an embedding very close to the face's own,
because identity is what ArcFace was optimised to preserve. We
confirmed the practical impact: `pareidolia test` (RGB) passes when
shown a phone-screen photo of a relative, with similarity solidly
above the 0.6 threshold.
The same query against the IR camera, however, didn't even reach the
matcher — SCRFD failed to detect a face in the IR view of the phone
screen at all, because phone/laptop displays emit visible light but
almost no near-infrared. Printed photos behave similarly in most
cases. Requiring *both* modalities to pass independently therefore
closes the common presentation-attack vector without adding any
anti-spoof ML.
# Core: `authenticate_dual` + `DualMatchOutcome` + `SingleVerdict`
`Enrollment::authenticate_dual(rgb_query, ir_query, threshold)` takes
two `Option<&Embedding>` queries (one per modality) and returns a
`DualMatchOutcome`:
- `Pass { rgb, ir }` — both modalities cleared the threshold against
their respective enrolled pools. Returns the winning `MatchResult`
from each pool for audit / logging.
- `Reject { rgb, ir }` — at least one modality failed. Both
per-modality `SingleVerdict`s are preserved so the failure mode is
fully diagnosable (which side failed, and how).
- `NoEnrollment` — fires only when the enrollment has zero samples
total; per-modality emptiness collapses into `SingleVerdict::NoSamples`
on that side instead.
`SingleVerdict` carries the four states a per-modality decision can
land in: `Pass`, `BelowThreshold` (face found, similarity below
threshold), `NoFace` (caller didn't supply an embedding — capture
error, no face detected, modality disabled), `NoSamples` (enrollment
has no samples for this kind). The `Option` parameter shape lets the
caller express 'couldn't produce an embedding for this side' directly,
guaranteeing the matcher rejects rather than silently degrading to
single-modality.
# Tests
Eight new unit tests pin the full truth table (empty enrollment, both
queries present and matching, RGB pass + IR below threshold, missing
RGB query, both queries missing, RGB-only enrollment with IR queried,
both below threshold, is_pass helper). Two new property tests pin the
soundness invariants:
- `authenticate_dual_pass_iff_both_single_modality_pass` — the
primary contract: dual.is_pass() ⟺ both single-modality authenticate
calls would have passed. Callers can reason about
`authenticate_dual` as 'AND over single-modality outcomes'.
- `authenticate_dual_with_either_query_missing_never_passes_on_nonempty`
— a missing query on either side always forces Reject for any
non-empty enrollment, regardless of threshold.
One integration smoke test
(`public_dual_modality_authenticate_requires_both_pools_to_pass`)
covers the end-to-end public API including the Pass shape and the
None-query Reject shape.
# CLI: `pareidolia test` learns dual mode
The `conflicts_with = rgb_camera` mutex on `--ir-camera` is removed.
Mode selection becomes:
- `--ir-camera` unset → single RGB mode against the RGB pool
(unchanged from M4.6).
- `--ir-camera` set → dual mode: capture one frame from each sensor
serially (sibling-node cooldown between, like enroll), run
`authenticate_dual`. 'No face detected' on one side becomes
`SingleVerdict::NoFace` rather than a hard error, so the verdict
surfaces fully.
The dual output path prints both per-modality verdicts on reject
('RGB: pass (similarity 0.823 ≥ 0.600, sample 1) / IR: no face
detected in captured frame'), making it obvious which side blocked
auth. The previous IR-single-camera testing flow gives way to dual:
running `pareidolia test --ir-camera /dev/videoX` against the typical
RGB+IR enrollment is now a strictly stronger check than before.
# Documentation
A new `## Security` section in CONTRIBUTING.md explains the threat
model: what ArcFace matches (identity, not liveness), what dual-mode
defeats (phone-screen photos, most paper photos) and what it doesn't
(NIR-tuned prints, 3D masks, NIR-illuminated photographs), and the
separate concern of family-resemblance false-accepts. A defense
matrix summarises the cells.
Diagnosing camera or face-detection failures from a hardware-only
trace ('confidence 0.231' or 'no face detected') is painful because
the actual frame data has already been thrown away by the time the
user sees the error. There's no way to tell whether the camera
captured a black frame, a wrong exposure, a frame of someone else's
face, or a perfectly fine frame that SCRFD just couldn't handle.
When `pareidolia enroll --debug-dump-dir <PATH>` is set, every
captured (post-conversion, pre-detection) frame is written to that
directory as a binary PPM file before the detector runs. The dump
fires regardless of detection outcome, so failed captures still
leave the frame on disk for inspection. Files are named
`rgb-{i}.ppm` and `ir-{i}.ppm` with 1-based sample indices.
PPM (P6) was chosen instead of PNG because the goal is debug
output, not distribution: no compression, no extra dependencies,
universal viewer support (feh, eog, ImageMagick, GIMP), and a hex
dump of the file is still meaningful if a viewer isn't handy. The
format is a three-line text header followed by raw RGB bytes.
# Pipeline runtime split
To dump the frame *before* face detection runs (so failed
detections still leave a frame), `pipeline_runtime::capture_one_face`
is decomposed into:
- `capture_rgb8(source)` — capture + RGB8 conversion only
- `embed_best_face(frame, detector, embedder)` — detect, align,
embed, pick highest-confidence face
`capture_one_face` itself remains as a thin wrapper composing the
two, so the `test` subcommand and any future single-shot callers
don't need to change. Enrollment switches to the two-step form so
the optional dump can sit between capture and detection.
# Scope
Behaviour is unchanged when the flag is absent — no extra disk
writes, no extra log lines, no change to enrollment storage. The
flag is documented as a debugging tool, not a production knob; the
daemon path (M5) won't expose it.
`pareidolia --version` now prints e.g. `pareidolia 0.0.0 (a59ad019…)`,
so a quick check tells you whether the installed binary actually matches
the workspace you think you're building from. Previously the version
was just the package version, which on a pre-1.0 codebase (we've been
at 0.0.0 since M1) is functionally meaningless for 'is this current?'
questions.
# Wiring
- The flake's shellHook runs `jj log -r @ --no-graph -T commit_id`
(with a `git rev-parse HEAD` fallback for non-jj checkouts and
`echo unknown` for working copies outside any VCS) and exports the
result as `PAREIDOLIA_BUILD_REV` at shell-entry.
- The CLI lib reads the env var at compile time via `option_env!`,
defaulting to `"unknown"` when unset. A `VERSION_FULL` const
combines the package version and the rev via
`const_format::concatcp!` — pure compile-time string concat, no
runtime allocation, returns a true `&'static str` that clap's
`#[command(version = ...)]` attribute can use directly.
- No `build.rs` (we still keep that rule from the build-cache hygiene
doc; the rev comes in via env var, not a build script).
# Staleness window
The rev is captured at *shell-entry*, not at build time. A build done
immediately after `jj commit` reflects the new commit; a build done
later in the same shell session reflects whatever the rev was when
the shell was entered. Re-enter `nix develop` (or re-export the env
var) to refresh. Documented in the shellHook comment.
# Non-Nix builds
`cargo install` outside the devShell sees no `PAREIDOLIA_BUILD_REV`
env var, so the rev defaults to `unknown`. That's a feature: it
tells the user the build provenance isn't tracked. Setting the env
var manually before `cargo install` also works for anyone who wants
the rev without the devShell.
ArcFace is RGB-trained, so an IR-captured face embedding lives in a
different region of the 512-d space than the RGB embedding of the
same face. Comparing across modalities produces near-zero similarities
even for the same person — fail-safe but unusable. Making IR auth
actually work meant moving five pieces together: the matching layer
had to filter by modality, enroll had to capture both sensors,
`open_camera` had to choose the right pixel format for each kind,
the V4L2 stream had to be primed before the first capture would
return real data, and the dual-camera capture loop had to open each
sensor on demand rather than holding both simultaneously.
# Matching (core)
`Enrollment::best_match` and `Enrollment::authenticate` now take a
`CameraKind` parameter representing the live query's modality. Only
same-kind samples are considered for the best-of-N selection. The
property tests already establish that best-of-N is monotonically
improving in N; adding the modality filter doesn't change that
property within either pool.
A new `MatchOutcome::NoSamplesForModality { kind }` variant
distinguishes 'user has samples but not of this kind' from
`NoEnrollment` ('user has no samples at all'). They sort to the
same auth decision (no match possible) but have different
remediations: enroll vs enroll-for-this-modality. The PAM glue in M5
will route them differently in logs.
# Dual-camera enroll (cli)
`pareidolia enroll` gains `--rgb-camera` (default /dev/video0,
renamed from `--camera`) and `--ir-camera` (optional). When both
are set, each capture iteration pulls one frame from each sensor:
the RGB frame goes into the enrollment tagged Rgb, the IR frame
tagged Ir. Same user pose roughly across both, but slight motion
between them is fine — each modality's samples live in their own
pool and benefit from minor pose variation within that pool.
# Camera format selection by kind
`open_camera` picks the V4L2 FourCC by camera kind: YUYV for RGB
cameras, GREY for IR. Hardcoding YUYV across the board caused IR
captures to either negotiate to garbage or silently mislabel a GREY
stream as YUYV, which then propagated through the pipeline as
gibberish data and produced 'no face detected' for every IR sample.
Most laptop and USB IR sensors advertise GREY (8-bit grayscale) and
nothing else.
# Stream priming
`open_camera` captures and discards one frame immediately after
opening. Two unrelated problems both want the first frame thrown
away:
1. The `v4l` crate's `Stream::next()` runs `VIDIOC_STREAMON` on
the first call and returns a buffer slot without the QBUF/DQBUF
cycle that would fill it with real captured data. The first
frame from a fresh source is therefore whatever was in the mmap
buffer at REQBUFS time — typically zeros, sometimes garbage.
2. IR illuminators on UVC cameras are commonly gated on the device
entering streaming state and take a frame or two after STREAMON
to reach full brightness. The first captured IR frame can be
too dark for the detector even when the crate's behaviour is
perfect.
Priming once after open sidesteps both. The ~33 ms cost is paid
once per source, not per capture.
# Open-on-demand for dual-modality enroll
The original dual-camera enroll loop held both V4L2 sources open
for the whole session to amortise the per-open cost. This wedges
some UVC dual-sensor devices: sample 1 succeeds, then the next
DQBUF on either sibling video node blocks indefinitely. UVC
bandwidth is allocated statically at STREAMON and dual-sensor
devices vary in how gracefully they cope with two concurrent
sustained streams.
Cameras are now opened **per modality, per sample**: open RGB,
prime, capture, drop; sleep a sibling-open cooldown; open IR,
prime, capture, drop. The two streams are never alive at the same
time. Cost is ~1s per sample of open-overhead, paid only during
enrollment (a once-per-user operation), and the pattern matches
exactly what the daemon will do for PAM auth requests in M5.
# Test subcommand
`pareidolia test` gains the same flag pair, but mutually exclusive:
either you test against RGB (`--rgb-camera`, default) or IR
(`--ir-camera`), since a single test is a single match attempt.
The query's modality is now passed through to
`Enrollment::authenticate` and the matcher filters appropriately.
A new printable path handles `NoSamplesForModality` cleanly with
an actionable suggestion ('re-run `pareidolia enroll` with
--ir-camera to add them').
# Tests
Two new unit tests pin the modality filter:
- `best_match_filters_to_query_kind_ignoring_cross_modality_samples`
- `authenticate_with_only_other_modality_samples_yields_no_samples_for_modality`
One new utility (`Enrollment::count_by_kind`) with its own unit
test; used by the enroll CLI for the user-facing 'saved X RGB and Y
IR samples' summary.
One new integration smoke test:
- `public_cross_modality_query_surfaces_no_samples_for_modality`
All existing tests updated to pass `CameraKind::Rgb` to
`best_match`/`authenticate` (existing samples are tagged Rgb by
default in the test fixture, so behaviour is unchanged for them).
The module is named `enrollment` (American) and the subcommand is
`enroll`, but I'd drifted into British 'enrol' / 'enrolment' in a
handful of doc comments and arg descriptions across the enrollment
type docs, the CLI `enroll` and `test` subcommands, and their
feature-disabled fallbacks. Both spellings are valid English, but
mixing them in one codebase is just noise — standardising on the
American form matches the module/subcommand names that have been
in place since they were introduced.
Pure docs / arg-description change; no code behaviour affected.
The smallest possible end-to-end check that an enrollment actually
works: capture one frame, embed the face, match against the user's
existing enrollment, report Pass / Reject / Not enrolled. Lets users
validate their enrolment before any of the daemon / PAM / IPC plumbing
arrives in M5+, and provides an obvious triage step once the full
login path exists ('auth is failing — does `pareidolia test` pass?').
# Order of operations
Enrollment is loaded *first*. If the user isn't enrolled (or has zero
samples), report that and exit cleanly without ever touching the
camera or loading models. The 'am I enrolled' question is often the
actual one and shouldn't require waking the IR illuminator to answer.
# Graceful fallback without features
The CI build (--no-default-features) doesn't have v4l2 or inference,
so the real subcommand impl isn't compiled. The fallback in
commands.rs handles the no-enrollment case fully (no hardware needed
— just the store) and only surfaces FeatureDisabled when an enrolled
user would require an actual capture to match. This matches what users
expect: 'asking whether someone is enrolled' should always work; 'asking
whether *this face* matches them' requires the hardware integrations.
# Shared pipeline runtime
`enroll` and `test` had near-identical model-loading and capture
logic; factored into `pipeline_runtime` so each subcommand impl
stays focused on what's actually different (enroll appends N samples,
test matches one). `resolve_model_path`, `capture_one_face`,
`model_fingerprint`, and `open_camera` live there. `enroll`
refactored to use them; behaviour unchanged.
# Camera defaults to YUYV
`open_camera` explicitly requests YUYV format. USB webcams (the
BRIO included) commonly default to MJPG output, which we don't yet
decode — `camera::convert::to_rgb8` punted MJPEG to a later
milestone. YUYV is universally supported by UVC cameras and lands
through the existing convert path. When MJPEG decoding arrives the
preference can become a `--fourcc` flag.
# Tests
One integration smoke test covers the no-enrollment path against an
empty store through `pareidolia_cli::run` in-process. The full
Pass/Reject paths are user-driven hardware tests — running `test`
against the user's own face after a successful `enroll` is the
obvious manual validation step.
Three new subcommands that drive the enrollment store from the user
side:
# enroll
Runs the full pipeline locally: V4L2 capture → format convert → SCRFD
detection → affine align → ArcFace embed → append to store. The
daemon will eventually drive this same flow via IPC for the PAM auth
path, but enrolment is infrequent and user-initiated, so a
self-contained CLI is the simplest design and the matching algorithm
already handles the append-not-replace case via best-of-N monotonicity
(see enrollment.rs).
Loads any existing enrolment first and appends; this is the default
flow precisely because the matching layer's best-of-N is monotonically
improving in N, so re-running enrol on a day you look different
(lighting, beard, baggy eyes, glasses) just widens the set of poses the
matcher will accept. A future --replace flag is a clean addition.
Feature-gated on both v4l2 and inference. The CLI's default-features
now includes both, mirroring the daemon's shape. CI's
--no-default-features build picks up a fallback that surfaces
CliError::FeatureDisabled with the missing feature name so the message
is actionable.
Each enrolled sample stores a SHA-256 fingerprint of the SCRFD model
file. Matching against samples from a different model than the one
currently loaded is meaningless and would silently degrade accuracy;
M5+ will use this fingerprint to refuse such matches.
# list / remove
Both are feature-flag-free (only need the store, not models or
camera). `list` defaults to one-name-per-line output for scripting;
`--verbose` loads each enrollment to include the sample count (which
HMAC-verifies every file, so it's not free at scale). `remove`
deletes the whole enrollment by default; `--sample-id N` removes
just that sample.
# Shared store config
`--store-root` and `--key-file` are global args. Both default
to XDG_STATE_HOME (or $HOME/.local/state fallback). The key file is
auto-generated from /dev/urandom on first use with mode 0o600, which
keeps the "just works in nix develop" UX without burying the
biometric-data PII concerns under a default that silently writes to
`./pareidolia` or similar surprise paths.
# clap env feature
Workspace clap dep now includes the `env` feature so subcommand args
can fall back to environment variables (heavily used here for model
paths and store config). Pure Rust, no system deps.
# Tests
Two integration smoke tests cover the dispatch + store wiring against
in-process `run()`:
- list against an empty store root → succeeds (catches subcommand
wiring regressions before any data exists).
- remove of a never-enrolled user → succeeds (idempotency holds at
the CLI layer too).
Plus four unit tests on the store_config helper module covering store
root creation, key file generation + permissions, key file idempotency,
and wrong-sized key file rejection.
Real enrolment requires the user's face and a working camera, so it's
covered by manual testing rather than automation — appropriate for a
user-initiated subcommand.
The store's write path was direct fs::write; a crash mid-write could
leave a torn file, and the default umask was determining whether the
biometric-data file was world-readable. Both are now fixed.
# Atomic via write-temp-then-rename
`save` now:
1. Opens a uniquely-named temp file (`.target.tmp.{pid}.{nanos}`) in
the same directory as the target, using O_CREAT | O_EXCL so a name
collision surfaces as a hard error rather than silent overwrite.
2. Writes all bytes and `sync_all`s the file so data is on disk
before the rename.
3. `rename(temp, target)` — atomic on POSIX within a filesystem;
either the old file is in place or the new one is, never a torn
mix.
4. `sync_all`s the parent directory so the rename itself survives a
crash. Errors at this step are non-fatal: the data is durable, only
the directory entry might not be (we'll log once logging lands in
M5).
Temp files are cleaned up on error so the directory doesn't accumulate
`.tmp.` debris over partial writes.
# Owner-private file permissions
Files are now created with explicit mode 0o600 (owner read+write only)
via OpenOptionsExt::mode. The daemon's intended deployment runs as a
dedicated system user; with 0600 only that user (and root) can read
the enrolled face embeddings, which are PII.
# Tests
Three new tests pin the new guarantees:
- Mode-0o600 is set on every saved file (PermissionsExt-based check).
- No `.tmp.*` files leak after a successful run (multiple saves).
- 16 concurrent writers racing on the same user yield a loaded file
that matches exactly one of the writers' inputs — never a torn mix.
This is the atomic-rename guarantee in test form.
Adds the persistence layer for enrollments. Each enrolled user gets one
CBOR-encoded file under a configured root directory, wrapped in a fixed-
position header and HMAC-SHA256 so any out-of-band mutation surfaces as
IntegrityFailure on the next load.
# File format
offset size contents
0 4 magic "PARE" (ASCII, fixed)
4 1 version currently 1
5 3 reserved zero-padded
8 32 HMAC-SHA256 HMAC of bytes [40..] under the key
40 … payload CBOR-encoded `Enrollment`
The HMAC sits at a fixed offset *before* the payload deliberately:
verification is one constant-time slice compare with zero
deserialisation. Parsing untrusted CBOR ahead of integrity verification
would expose the daemon to whatever bugs a malicious file could
trigger in the deserialiser.
`verify_slice` is constant-time, which matters — a timing leak there
would let an attacker forge an HMAC byte by byte.
# Atomicity
`save` currently writes the target file directly; a crash mid-write
can leave a torn file. The atomic write-temp-then-rename path lands in
the next sub-milestone (M4.3) and won't change the format or HMAC
scheme.
# Username sanitisation
Usernames double as filenames, so they're validated against an
allowlist: ASCII alphanumerics plus `_`, `-`, `.`, no leading dot,
no `..`, max 64 bytes. Path traversal attempts surface as
InvalidUsername before the filesystem sees them.
# Tests
11 unit tests covering encode/decode round-trip, header validation
(magic, version, truncation), HMAC integrity (wrong key, payload
tampering, HMAC tampering), username validation (positive and negative
cases), and the EnrollmentStore filesystem operations (save+load,
NotEnrolled, list_users sorting, remove idempotency, on-disk tamper
detection).
3 property tests pin the headline guarantees:
- `decode(encode(e, k), k) == e` for arbitrary enrollments — the
format loses no information.
- Flipping any post-header byte triggers IntegrityFailure — what turns
'we have an HMAC' into 'we have integrity'.
- Cross-key decode always fails — keys are not interchangeable.
# Deps
serde + ciborium + hmac + sha2 join the workspace deps. All four are
pure Rust with no system requirements, so they don't need feature
gating; CI's --no-default-features build picks them up automatically.
First M4 slice. Pure data + math: enrolled samples, the per-user
enrollment that holds them, and the cosine-similarity-based matching
algorithm that drives the eventual PAM auth decision. The on-disk
store (HMAC, atomic writes) lands in subsequent commits; this slice
gives matching tests something to run against and pins the algorithmic
contract.
# Best-of-N, not mean-of-N
The matching layer takes the *best* of the N enrolled-sample
similarities, not their mean. Mean-of-N would make a frontal-only
enrollment fail on a slight head turn even though the user is clearly
recognisable from one of the angles already captured. Best-of-N matches
how real enrolment data is captured (multiple poses to span variation)
and how face recognition systems are typically operated.
# Match outcomes
`MatchOutcome` distinguishes three states: `Pass` (clears threshold),
`Reject` (below threshold, includes the best result so logs can show
how close), and `NoEnrollment` (no samples to match against). The
third variant is separated from `Reject` because the remediation
differs — the user needs to enrol, not retry — and the PAM glue will
want to route those cases differently when they cross the IPC boundary
in M5.
# Numeric edge cases pinned
- NaN similarities (dim-mismatched embeddings) are filtered out of
best-of-N selection rather than poisoning it. A mismatched-model
query against a populated enrollment surfaces as 'no valid candidate'
rather than 'best match was NaN'.
- Threshold comparison is `>=`: similarity exactly at the threshold
passes. The test `authenticate_at_exactly_the_threshold_passes_inclusively`
hand-crafts a similarity of 0.6 against threshold 0.6 to pin this.
# Tests
15 unit tests covering enrollment construction / push / remove,
best-match selection (single, multi-axis, NaN-skipping, mixed-validity),
threshold behaviour, and the MatchOutcome helpers.
5 property tests covering the invariants the matching layer relies on:
- Pushing a sample never decreases best-match similarity (B-o-N
monotonicity).
- Removing a sample never increases best-match similarity (mirror).
- Query identical to an enrolled sample yields similarity ≈ 1.0.
- Threshold at -∞ always passes any non-empty enrollment.
- Threshold above 1.0 never passes any normalised embedding.
Plus three cross-crate integration smoke tests for the public surface.
The SCRFD module's pure stages (NMS, letterbox preprocess, bilinear
resize, anchor decode) had hand-picked goldens but no property coverage.
Those are exactly the code paths where structural bugs silently degrade
detection quality without crashing — the kind property tests catch best.
# Coverage added
NMS (the largest gap):
- Output count is never greater than input count.
- Every pair of surviving detections has IoU ≤ threshold — the
postcondition NMS exists to enforce.
- The globally-best detection (highest confidence) always survives.
- Idempotency: running NMS on its own output produces the same set.
- Input-order independence: reversing the input order produces the same
surviving set.
letterbox_preprocess:
- Output tensor is always [1, 3, target, target] regardless of source
dimensions.
- det_scale is finite, positive, and produces source dimensions that
fit inside the target (within sub-pixel rounding).
bilinear_resize_rgb8:
- Output buffer length is always exactly dst_w · dst_h · 3.
- Solid colour preservation across arbitrary resize ratios (stronger
than the unit test which fixed one ratio).
decode_stride:
- Never panics on arbitrary tensor lengths or grid sizes. A malformed
model output should produce zero detections, never an index-out-of-
bounds panic that takes down the daemon.
# Performance
Letterbox proptests are bounded to 400×400 source dimensions because
each iteration runs a full bilinear resize. The 'output shape' and
'det_scale fit' properties are structural and don't gain coverage from
larger inputs; capping the bound keeps the test in single-digit seconds.
# Cost to CI
Zero: the proptests live inside the inference-feature-gated scrfd
module, so CI's --no-default-features build doesn't compile or run them.
The full-feature local suite picks them up via `just test-features`.
`pipeline::recognise` ties the three model-touching stages — Detector,
align_face, Embedder — into a single call. The output is a Vec of
(Detection, Embedding) pairs, one per face the detector found,
preserving the detector's ordering. This is what the daemon's PAM path
will eventually drive once enrollment + matching land in M4.
Pure orchestration code: no feature gate, no new runtime deps. It
operates on the existing trait + type surface, so it compiles in every
workspace configuration including --no-default-features.
# Errors abort the pipeline
Per-face partial successes (e.g. 'detected three faces, aligned two,
embedded one') would be hard to reason about in the matching layer, so
any stage error aborts the whole call with no results. The unit tests
pin this behaviour: detector errors, degenerate landmarks (align fails),
and embedder errors all surface as the original error variant with no
side effects.
# Golden test against real models
A new hardware-gated test runs the full pipeline against the InsightFace
sample face JPEG (t1.jpg, ~128 KB, MIT-licensed via their repo). It
asserts:
- ≥1 detection (the fixture has a clearly visible face; finding zero
would mean the detector regressed).
- Every detection's confidence clears the default 0.5 threshold.
- Every embedding is 512-d with L2 magnitude ≈ 1.0.
- Running the pipeline twice on the same frame yields bit-identical
embeddings (CPU EP determinism guarantee).
Stops short of pinning exact embedding values — that's the matching
layer's concern in M4, where similarity-against-stored-baseline becomes
the natural assertion.
# Fixture infrastructure
The fixture image is pinned in nix/models.nix with sha256, fetched at
devShell build time, and exposed via PAREIDOLIA_TEST_FACE so the
hardware test runs without manual setup inside `nix develop`. Outside
Nix, set the env var manually (the InsightFace URL is in
nix/models.nix).
`image` (jpeg + png decoders only, default-features off) joins the
workspace dev-dependencies for fixture loading; production code never
needs to decode images because the camera produces RGB8 directly.
The hardware-gated tests were panicking when their prerequisite (a
plugged-in camera, or a PAREIDOLIA_TEST_* env var pointing at a model
file) was missing. That made sense as a 'you ran this wrong' assertion
when the only invocation path was explicit `just test-hardware` from
inside `nix develop` with hardware present — but it falls over the
moment any of those conditions slip.
Concrete case: a USB unplug or driver hiccup leaves /dev/video* gone,
and `just test-hardware` fails noisily even though the other hardware
tests (model-loading, inference) would have run fine. The panic also
wrongly suggests something's broken when in fact the test simply lacks
its physical prerequisite.
Replace each panic with an `eprintln!` describing what's missing and
how to set it up, then early-return. Nextest treats the early return
as a pass — conventional behaviour for hardware tests in the Rust
ecosystem — and the suite stays green when hardware is temporarily
absent.
The eprintln output is captured by nextest's default profile, so the
skip messages would be invisible unless tests failed. `test-hardware`
now passes `--no-capture` so each test's stderr streams through in
real time, and each skip message is prefixed `warning:` so it stands
out.
While I was in there: the per-camera capture test was tolerantly
`continue`-ing on individual open errors but then *succeeding* even
when every camera failed, defeating the point of the test (saw this
when the BRIO firmware wedged into an EIO state after a previous run).
It now asserts at least one capture succeeded among the enumerated
devices, and inserts a 500ms cooldown between iterations because UVC
cameras commonly EIO on rapid open-close-open cycles.
UVC cameras commonly expose multiple /dev/video* nodes per physical
device: typically one or two real Video Capture interfaces plus matching
Metadata Capture siblings. Our enumerate() was accepting all of them by
filename pattern alone, so a Logitech BRIO (with video0/2 real + video1/3
metadata-only) showed up as four rows in `pareidolia cameras` and got
exercised four times by the capture hardware test.
The metadata-only nodes don't have VIDEO_CAPTURE capability, so opening
them for capture (set_format, REQBUFS, STREAMON) fails after the
firmware-touching ioctls have already woken the device. On the BRIO this
puts the IR illuminator into a 'warmed' state that lingers for several
seconds after the test ends — visible IR LEDs glowing between test runs
— and also causes control-query timeouts on subsequent v4l2-ctl
inspections.
Querying VIDIOC_QUERYCAP on the node (the device_caps field, which is
per-node rather than the union across the driver) and filtering by
VIDEO_CAPTURE excludes the metadata siblings without any extra fragility:
real capture devices keep advertising the cap; metadata ones don't.
After this, `pareidolia cameras` on a BRIO returns:
PATH NAME KIND
/dev/video2 Logitech BRIO ir
/dev/video0 Logitech BRIO rgb
instead of four rows where two were unusable noise.
Last real-model piece of the pipeline. Takes a 112×112 RGB8 chip produced
by `align_face`, runs a single ONNX forward pass through the ArcFace
glintr100 model, and returns a 512-d L2-normalised embedding ready for
the matching layer in M4.
Substantially simpler than the SCRFD detector: single input, single
output, no anchor decoding, no NMS. Preprocessing differs from SCRFD's
in one detail — ArcFace's training used `(pixel − 127.5) / 127.5`
(range [-1, 1]) where SCRFD used `/ 128.0` — and the embedder is strict
about input dimensions: 112×112 only, no rescaling on input. Wrong shapes
surface as `InvalidInputShape` before any inference runs.
The 512-d output goes straight through `Embedding::normalized` so cosine
similarity reduces to a dot product in M4's matching layer. The output
dimension is enforced after inference; a model returning the wrong
dimension surfaces as a Backend error rather than producing a malformed
embedding.
# Tests
4 pure unit tests covering the preprocessor (NCHW shape, the −1/+1
boundary mappings, per-channel separation into NCHW planes).
5 hardware-gated tests covering the embedder end-to-end via the real
ArcFace model:
- Non-RGB8 input is rejected before inference
- Wrong dimensions are rejected before inference
- Solid gray chip produces a 512-d unit-magnitude embedding
- Repeated inference is bit-for-bit deterministic (default CPU EP)
- Different inputs produce non-identical embeddings (sanity check)
Combined hardware suite now runs in ~16s (V4L2 capture dominates at ~11s;
all 5 ArcFace tests take ~5s combined including model load).
The pipeline stage between the detector and the embedder. Given detected
landmarks and the canonical ArcFace template, fits a similarity transform
(uniform scale + rotation + translation), inverts it, and warps the source
frame into a 112×112 chip.
# 2D-specific math
ArcFace alignment is a 2D-to-2D similarity transform, so we get a closed-
form least-squares fit without invoking SVD. The 2x2 rotation+scale can
be parameterised by a single (u, v) pair (u = s·cos θ, v = s·sin θ) and
solved as four sums over the centred points. This keeps the alignment
code free of any linear-algebra dependency — no nalgebra, no manual SVD.
# Behaviour at the edges
- Non-RGB8 frames return InvalidInputFormat rather than producing garbage.
- All-coincident or otherwise degenerate landmarks return a Backend
error (the linear system has no solution). The daemon should fall
through to password in this case rather than panic.
- Inverse-mapped samples that fall outside the source frame return
black pixels. This produces an aligned chip with some black border
when the face is near the source edge — a useful downstream signal
rather than a fatal error.
# Tests
19 unit tests covering the SimilarityTransform type (identity, scale,
rotation, inverse round-trip, singularity guard), the closed-form fit
(pure translation, pure scale, pure rotation, mismatched / empty / coincident
inputs), align_face (format validation, output shape, identity-warp
solid-colour preservation, metadata propagation, degenerate-landmark
error), and the bilinear sampler (out-of-bounds, NaN, exact-integer hits).
2 property tests guard the headline invariants — the fit-then-apply
reproduces any well-conditioned target within f32 tolerance, and the
inverse round-trips any non-singular transform within f32 tolerance —
so future micro-optimisations of the math can't silently drift.
First real Detector impl. Wires SCRFD-10G (loaded by OnnxSession from the
nix-pinned ONNX file) into the pipeline's Detector trait. Covers the full
flow: letterbox preprocess, normalised NCHW tensor, single inference call,
9-output anchor decoding across 3 stride levels, NMS, coordinate
rescaling.
Trait refinement: Detector::detect and Embedder::embed now take `&mut self`.
ONNX Runtime's Session::run is `&mut self` and the daemon owns one
detector per worker (never sharing concurrently), so this matches the
usage pattern without forcing interior mutability. As a bonus the
ScriptedDetector mock can now pop its queue properly instead of the
peek-and-pretend-to-mutate hack the previous `&self` signature forced.
Implementation notes:
- Letterbox uses the standard SCRFD layout: resized image at top-left,
zero-padded right/bottom, single scale factor. The OpenCV half-pixel
offset convention in the bilinear resampler matches reference Python
implementations, so detection coordinates compare correctly against
insightface goldens later.
- Bilinear resize is hand-rolled (~30 LOC) rather than pulling in the
`image` crate. We'll add `image` in M3.5 when fixture loading needs
it; not paying for it now keeps the inference feature's transitive dep
surface small.
- Decoder bails (returns no detections) rather than panicking when output
tensor shapes don't match the expected stride layout. A model variant
with different strides shouldn't take down the daemon, just report no
faces and let PAM fall through to password.
- Default thresholds (score 0.5, NMS IoU 0.4) match InsightFace
recommendations.
Tests:
- 18 unit tests on the pure helpers (NMS, anchor decoding, letterbox,
bilinear resize, coordinate rescaling). The decode test exercises the
anchor centre → bbox + landmarks math against hand-computed expected
outputs; the NMS tests cover score-order invariance, IoU-threshold
edge cases, and disjoint-box preservation.
- 1 hardware-gated end-to-end test loading the real SCRFD model and
running it on a synthetic black frame. Asserts no detections (no face
exists) and no panic; the runtime path is what matters here. Currently
~210ms wall-clock on CPU on the dev machine.
The real Detector and Embedder implementations need ONNX Runtime; this
commit wires up the dependency, the Nix devShell, the pinned model
weights, and a thin session wrapper they'll build on top. The actual
SCRFD detector and ArcFace embedder land in subsequent sub-milestones
(M3.3, M3.5).
Feature shape mirrors v4l2: `pareidolia-core` keeps `inference` opt-in
(default = []); the daemon enables it via its own default-on `inference`
feature that proxies to core. CI's --no-default-features keeps libv4l +
libonnxruntime out of the Nixery image while normal builds get the full
pipeline.
`ort`'s `load-dynamic` feature is the cleanest fit for Nix: the crate
dlopens libonnxruntime at runtime via ORT_DYLIB_PATH rather than linking
against it at build time. The devShell wires ORT_DYLIB_PATH to nixpkgs's
`.so` so both build and run work without bundling C++ libs.
`OnnxSession` exposes the underlying `ort::Session` directly rather
than wrapping every ort API behind a Pareidolia-flavoured alternative —
chasing ort's API as it evolves is more churn than the abstraction is
worth. The boundary is at error types: `ort::Error` is mapped to
`PipelineError::Backend` so the rest of the pipeline stays
backend-agnostic.
Model weights: the M6 plan had Nix-driven model fetching land alongside
the NixOS module, but the M3 sub-milestones need real ONNX files for
their hardware tests, so the fetching infrastructure moves forward.
`nix/models.nix` pins SCRFD-10G and ArcFace glintr100 from the
`immich-app/buffalo_l` HuggingFace mirror (a maintained mirror of the
InsightFace bundle, stable since 2023). The devShell exports
PAREIDOLIA_TEST_{MODEL,SCRFD,ARCFACE} pointing at the materialised store
paths so `just test-hardware` runs end-to-end with no manual setup.
The pinned weights are also exposed as flake packages
(`nix build .#scrfd-10g`) for ad-hoc use outside the devShell.
Foundation for the three-stage face recognition pipeline (detect → align →
embed). The traits exist now so the daemon, integration tests, and (later)
NixOS VM tests can be wired against them while the real ONNX-backed
implementations land in subsequent sub-milestones (M3.2 detector, M3.4
embedder).
Types worth flagging:
- `Embedding` is a Vec<f32> newtype with L2-normalisation built into the
preferred constructor (`Embedding::normalized`) and cosine similarity
as a plain dot product. Pre-normalising lets the matching layer skip
per-comparison magnitude divisions in the hot path. Magnitude-zero
inputs round-trip without dividing by zero so a degenerate embedder can
never panic the daemon.
- `Landmarks` fixes the canonical InsightFace order (left eye, right eye,
nose, mouth-left, mouth-right). The affine alignment in M3.3 depends on
this order being stable; nailing it down here means future code that
iterates landmarks (via `Landmarks::iter`) doesn't have to negotiate.
- `PipelineError` deliberately doesn't derive `Clone` — the scripted mock
implements re-emission with an explicit per-variant clone helper, so
adding a new variant in the future requires a conscious decision about
how it should propagate through scripted timelines.
Mock pair: `FixedDetector` / `FixedEmbedder` for the common 'return the
same thing on every call' pattern; `ScriptedDetector` for tests that need
a specific timeline. Property tests on the embedding math (magnitude
preservation under normalisation, cosine commutativity, never-NaN on
well-formed inputs) guard the invariants the matching layer in M4 will
rely on.
First CLI surface that actually does something. `pareidolia cameras` calls
`pareidolia_core::camera::v4l2::enumerate()` and renders the result as a
plain-text table with PATH / NAME / KIND columns. Read-only — enumeration
doesn't lock any device, so it's safe to run while the daemon or other apps
are using a camera.
clap drives argument parsing (derive style) so future subcommands compose
cleanly. `try_parse_from` rather than `parse_from` keeps the library
side from calling `process::exit` on --help / --version; we map clap's
exit-code-0 errors to Ok(()) at the dispatch point. main.rs now passes
`args_os()` as-is (program name included) rather than skipping it, which
matches what clap expects.
The CLI mirrors the daemon's feature shape: `v4l2` default-on, proxies to
core's `v4l2`. CI's `--no-default-features` disables it, in which case
the subcommand surfaces a clean error pointing at the build-time choice
rather than silently returning empty output.
Subcommand impls live in per-command submodules so each command's parsing
+ output + tests stay colocated and the surface of any single file stays
small. `format_table` is split out as a pure function so unit tests can
pin the output shape without touching stdout.
There's a discoverable gap between 'just ci-fast' (what's actually gated)
and 'every check a contributor should run before pushing'. Folding ci-fast
and ci-slow into a single 'just test' makes the pre-push expectation
explicit without forcing contributors to remember both names.
Hardware tests stay separate ('just test-hardware') so the recipe runs
identically whether or not the developer has a camera or other dev hardware
attached.
The v4l2 module is opt-in via the `v4l2` feature on pareidolia-core, and
its hardware-touching tests are `#[ignore]`d. Neither convention is
discoverable without docs, and 'just test-hardware' is the kind of recipe
contributors will forget exists if it isn't surfaced in CONTRIBUTING.
Also adds `test-v4l2` for running the v4l2 module's hardware-free tests
(FourCC mapping, enumerate-on-empty-host) with the feature enabled.
First hardware-touching FrameSource. Wraps the `v4l` crate to expose Linux
V4L2 capture through the same trait the mocks use. The two public entry
points cover the only operations the daemon needs:
- `enumerate()` walks /dev/video* and returns the cameras it can see.
- `V4l2FrameSource::open()` configures a specific device for capture.
# Exclusivity
V4L2 capture devices are exclusive while buffers are allocated: any other
process trying to open the same /dev/video* node gets EBUSY. The daemon's
intended pattern is therefore open-on-demand (per PAM auth request) and
drop-immediately-after so the camera stays available to other applications.
Drop cleanup must actually run — the v4l crate's `Stream<'a>` is
conservatively lifetime-bound to its `&Device`, so we use `self_cell` to
own both together and drop them in the right order (Stream first, then
Device).
# Feature shape
V4L2 is required for Pareidolia to actually run on a real system, so the
daemon's default feature set includes `v4l2` and the binary picks it up
out of the box. `pareidolia-core` keeps `v4l2` opt-in (default = []) so
consumers that only need the pipeline (mocks, format conversions) don't pay
the libv4l / bindgen build cost. CI passes `--no-default-features` to
disable v4l2 everywhere, which avoids needing libv4l in its Nixery image.
The workspace.dependencies entries set default-features = false on all the
internal crates so daemon's v4l2 doesn't transitively re-enable itself for
unrelated consumers (integration tests, the PAM module) through the dep
graph — a quirk of `cargo --workspace` feature unification.
A heuristic guesses IR vs RGB by checking whether the device advertises any
colour FourCCs (YUYV / RGB3 / MJPG). It's intentionally crude; the daemon's
config can override per device.
A transitive dep of the soon-to-land v4l backend (`home` via `bindgen`)
declares MSRV 1.88, so 1.85 no longer resolves a complete dep graph. Bumping
to 1.90 gives us a small buffer above that floor without committing to the
absolute latest release.
The `v4l` crate generates its FFI layer via bindgen at build time, which
needs libclang plus the surrounding C / kernel-UAPI header search paths
set up correctly. `rustPlatform.bindgenHook` from nixpkgs is the
idiomatic way to provide all of that — it wires up LIBCLANG_PATH and
BINDGEN_EXTRA_CLANG_ARGS via the standard stdenv mechanism. Adding it to
nativeBuildInputs means the devShell handles bindgen out of the box.
`v4l-utils` covers libv4l itself. CI workflows aren't updated here because
the v4l backend lands feature-gated; CI keeps building the default-no-features
profile and so doesn't need any of this.
End-to-end check that capture and conversion compose through the public
re-export path the daemon and CLI will actually use: pull a Gray8 frame from
a mock source, run it through `camera::convert::to_rgb8`, verify the output
shape and that metadata (camera kind) propagates across the boundary. Catches
the regression where convert is exported but the path it's exported under
silently changes.
The downstream inference pipeline eats RGB8 exclusively after alignment, so
the dominant operation downstream of capture is 'turn whatever the camera
gave us into RGB8'. This module is that conversion surface.
Supported source formats: Rgb8 (identity), Gray8 (single-channel replicated),
Gray16 LE (high byte taken then replicated — sufficient for face recognition
input even when the low byte carries useful information), and Yuyv (BT.601
full-range, the JFIF flavour most consumer webcams produce). MJPEG returns
`UnsupportedFormat` until a JPEG decoder is wired in.
Correctness over throughput: YUYV uses f32 arithmetic for legibility. The
slowest target machine (a 2018 i5 XPS) easily absorbs the cost at typical
webcam resolutions; fixed-point is a profiler-led decision, not a blind one.
`MalformedBuffer` is added to `FrameError` so a corrupt capture surfaces
as a clean typed error all the way up to PAM (which then falls through to
password) rather than panicking the daemon.
Property tests guard the structural invariants — output length always matches
declared dimensions, grayscale conversions always replicate, and crucially,
the converter never panics on arbitrary buffer / dimension combinations.
That last property is what justifies trusting it on real V4L2 input where
malformed frames are a realistic failure mode.
Rust 1.75 predates the stabilisation of edition 2024, and the modern crates.io
graph (proptest's transitive deps in particular) increasingly requires it.
Bumping the pin to 1.85 — the first stable with edition 2024 — gets us onto
a current toolchain without committing to bleeding-edge versions.
`rust-version` in Cargo.toml is bumped in lockstep so MSRV stays honest.
`#[non_exhaustive]` is a real semver attribute, not just an annotation: it
prevents downstream crates from exhaustively matching on an enum, forcing
them to write wildcard arms. The intent is to let library authors add
variants in future versions without breaking consumers.
For Pareidolia today every consumer of these enums is another crate in this
workspace, so the trade-off inverts: wildcard arms in our own match
statements would silently absorb future variants instead of producing the
compile errors we actually want (e.g. 'you added Nv12 to PixelFormat but
the converter in daemon still doesn't handle it').
Unit tests inside `pareidolia-core` reach in via `super::` and so can't
catch the case where the public re-export surface accidentally hides a type
a downstream crate needs. These cross-crate tests are explicitly that
backstop: they exercise the same mocks but only through the public path the
daemon and CLI will use.
Lives as a module under the single integration test binary, per the workspace
convention documented in CONTRIBUTING.md.
Establishes the camera abstraction the rest of M2 (pixel format conversions,
V4L2 backend, camera enumeration) fills in.
Decoupling the inference pipeline and daemon from V4L2 via a trait — and
shipping mock implementations alongside the trait itself — means every
downstream consumer is testable without hardware. The mocks live outside
`#[cfg(test)]` because integration tests, the daemon's own test harness,
and (eventually) NixOS VM tests all need them too.
Three mock flavours cover the test surface:
- StaticFrameSource for the common 'same frame every time' case
- ScriptedFrameSource for scenarios that need a specific timeline of
successes and failures
- FailingFrameSource for fault injection (camera unplugged, busy, etc.)
`thiserror` is pulled into `pareidolia-core` to keep `FrameError`'s
variants declarative.