Pareidolia#
Pareidolia is a facial recognition login service for Linux.
License#
Pareidolia is distributed under the MIT and Apache 2.0 licenses.
UVC cameras commonly expose multiple /dev/video* nodes per physical
device: typically one or two real Video Capture interfaces plus matching
Metadata Capture siblings. Our enumerate() was accepting all of them by
filename pattern alone, so a Logitech BRIO (with video0/2 real + video1/3
metadata-only) showed up as four rows in `pareidolia cameras` and got
exercised four times by the capture hardware test.
The metadata-only nodes don't have VIDEO_CAPTURE capability, so opening
them for capture (set_format, REQBUFS, STREAMON) fails after the
firmware-touching ioctls have already woken the device. On the BRIO this
puts the IR illuminator into a 'warmed' state that lingers for several
seconds after the test ends — visible IR LEDs glowing between test runs
— and also causes control-query timeouts on subsequent v4l2-ctl
inspections.
Querying VIDIOC_QUERYCAP on the node (the device_caps field, which is
per-node rather than the union across the driver) and filtering by
VIDEO_CAPTURE excludes the metadata siblings without any extra fragility:
real capture devices keep advertising the cap; metadata ones don't.
After this, `pareidolia cameras` on a BRIO returns:
PATH NAME KIND
/dev/video2 Logitech BRIO ir
/dev/video0 Logitech BRIO rgb
instead of four rows where two were unusable noise.
Last real-model piece of the pipeline. Takes a 112×112 RGB8 chip produced
by `align_face`, runs a single ONNX forward pass through the ArcFace
glintr100 model, and returns a 512-d L2-normalised embedding ready for
the matching layer in M4.
Substantially simpler than the SCRFD detector: single input, single
output, no anchor decoding, no NMS. Preprocessing differs from SCRFD's
in one detail — ArcFace's training used `(pixel − 127.5) / 127.5`
(range [-1, 1]) where SCRFD used `/ 128.0` — and the embedder is strict
about input dimensions: 112×112 only, no rescaling on input. Wrong shapes
surface as `InvalidInputShape` before any inference runs.
The 512-d output goes straight through `Embedding::normalized` so cosine
similarity reduces to a dot product in M4's matching layer. The output
dimension is enforced after inference; a model returning the wrong
dimension surfaces as a Backend error rather than producing a malformed
embedding.
# Tests
4 pure unit tests covering the preprocessor (NCHW shape, the −1/+1
boundary mappings, per-channel separation into NCHW planes).
5 hardware-gated tests covering the embedder end-to-end via the real
ArcFace model:
- Non-RGB8 input is rejected before inference
- Wrong dimensions are rejected before inference
- Solid gray chip produces a 512-d unit-magnitude embedding
- Repeated inference is bit-for-bit deterministic (default CPU EP)
- Different inputs produce non-identical embeddings (sanity check)
Combined hardware suite now runs in ~16s (V4L2 capture dominates at ~11s;
all 5 ArcFace tests take ~5s combined including model load).
The pipeline stage between the detector and the embedder. Given detected
landmarks and the canonical ArcFace template, fits a similarity transform
(uniform scale + rotation + translation), inverts it, and warps the source
frame into a 112×112 chip.
# 2D-specific math
ArcFace alignment is a 2D-to-2D similarity transform, so we get a closed-
form least-squares fit without invoking SVD. The 2x2 rotation+scale can
be parameterised by a single (u, v) pair (u = s·cos θ, v = s·sin θ) and
solved as four sums over the centred points. This keeps the alignment
code free of any linear-algebra dependency — no nalgebra, no manual SVD.
# Behaviour at the edges
- Non-RGB8 frames return InvalidInputFormat rather than producing garbage.
- All-coincident or otherwise degenerate landmarks return a Backend
error (the linear system has no solution). The daemon should fall
through to password in this case rather than panic.
- Inverse-mapped samples that fall outside the source frame return
black pixels. This produces an aligned chip with some black border
when the face is near the source edge — a useful downstream signal
rather than a fatal error.
# Tests
19 unit tests covering the SimilarityTransform type (identity, scale,
rotation, inverse round-trip, singularity guard), the closed-form fit
(pure translation, pure scale, pure rotation, mismatched / empty / coincident
inputs), align_face (format validation, output shape, identity-warp
solid-colour preservation, metadata propagation, degenerate-landmark
error), and the bilinear sampler (out-of-bounds, NaN, exact-integer hits).
2 property tests guard the headline invariants — the fit-then-apply
reproduces any well-conditioned target within f32 tolerance, and the
inverse round-trips any non-singular transform within f32 tolerance —
so future micro-optimisations of the math can't silently drift.
First real Detector impl. Wires SCRFD-10G (loaded by OnnxSession from the
nix-pinned ONNX file) into the pipeline's Detector trait. Covers the full
flow: letterbox preprocess, normalised NCHW tensor, single inference call,
9-output anchor decoding across 3 stride levels, NMS, coordinate
rescaling.
Trait refinement: Detector::detect and Embedder::embed now take `&mut self`.
ONNX Runtime's Session::run is `&mut self` and the daemon owns one
detector per worker (never sharing concurrently), so this matches the
usage pattern without forcing interior mutability. As a bonus the
ScriptedDetector mock can now pop its queue properly instead of the
peek-and-pretend-to-mutate hack the previous `&self` signature forced.
Implementation notes:
- Letterbox uses the standard SCRFD layout: resized image at top-left,
zero-padded right/bottom, single scale factor. The OpenCV half-pixel
offset convention in the bilinear resampler matches reference Python
implementations, so detection coordinates compare correctly against
insightface goldens later.
- Bilinear resize is hand-rolled (~30 LOC) rather than pulling in the
`image` crate. We'll add `image` in M3.5 when fixture loading needs
it; not paying for it now keeps the inference feature's transitive dep
surface small.
- Decoder bails (returns no detections) rather than panicking when output
tensor shapes don't match the expected stride layout. A model variant
with different strides shouldn't take down the daemon, just report no
faces and let PAM fall through to password.
- Default thresholds (score 0.5, NMS IoU 0.4) match InsightFace
recommendations.
Tests:
- 18 unit tests on the pure helpers (NMS, anchor decoding, letterbox,
bilinear resize, coordinate rescaling). The decode test exercises the
anchor centre → bbox + landmarks math against hand-computed expected
outputs; the NMS tests cover score-order invariance, IoU-threshold
edge cases, and disjoint-box preservation.
- 1 hardware-gated end-to-end test loading the real SCRFD model and
running it on a synthetic black frame. Asserts no detections (no face
exists) and no panic; the runtime path is what matters here. Currently
~210ms wall-clock on CPU on the dev machine.
The real Detector and Embedder implementations need ONNX Runtime; this
commit wires up the dependency, the Nix devShell, the pinned model
weights, and a thin session wrapper they'll build on top. The actual
SCRFD detector and ArcFace embedder land in subsequent sub-milestones
(M3.3, M3.5).
Feature shape mirrors v4l2: `pareidolia-core` keeps `inference` opt-in
(default = []); the daemon enables it via its own default-on `inference`
feature that proxies to core. CI's --no-default-features keeps libv4l +
libonnxruntime out of the Nixery image while normal builds get the full
pipeline.
`ort`'s `load-dynamic` feature is the cleanest fit for Nix: the crate
dlopens libonnxruntime at runtime via ORT_DYLIB_PATH rather than linking
against it at build time. The devShell wires ORT_DYLIB_PATH to nixpkgs's
`.so` so both build and run work without bundling C++ libs.
`OnnxSession` exposes the underlying `ort::Session` directly rather
than wrapping every ort API behind a Pareidolia-flavoured alternative —
chasing ort's API as it evolves is more churn than the abstraction is
worth. The boundary is at error types: `ort::Error` is mapped to
`PipelineError::Backend` so the rest of the pipeline stays
backend-agnostic.
Model weights: the M6 plan had Nix-driven model fetching land alongside
the NixOS module, but the M3 sub-milestones need real ONNX files for
their hardware tests, so the fetching infrastructure moves forward.
`nix/models.nix` pins SCRFD-10G and ArcFace glintr100 from the
`immich-app/buffalo_l` HuggingFace mirror (a maintained mirror of the
InsightFace bundle, stable since 2023). The devShell exports
PAREIDOLIA_TEST_{MODEL,SCRFD,ARCFACE} pointing at the materialised store
paths so `just test-hardware` runs end-to-end with no manual setup.
The pinned weights are also exposed as flake packages
(`nix build .#scrfd-10g`) for ad-hoc use outside the devShell.
Pareidolia is a facial recognition login service for Linux.
Pareidolia is distributed under the MIT and Apache 2.0 licenses.