fix: deterministic documentation-figure rendering (end the flaky figures job) (#131)

* fix: make documentation-figure rendering deterministic across machines

The "Documentation figures up to date" CI job was intermittently failing on the
heavy compute figures (excitation_signals, multiple_shock, sii_vocal_efforts,
sottek_specific_loudness, tonality_roughness_demo): their multi-threaded
numerical backends reorder floating-point reductions depending on the available
core count, so the CI runner produced byte-different SVG/PNG output than the
committed (single-threaded) baseline, and a plain re-run usually "fixed" it.

Pin the numerical thread pools to a single thread so rendering is reproducible
regardless of the host core count:

- generate_graphs.py / generate_diagrams.py set OMP/MKL/OPENBLAS/NUMEXPR/NUMBA/
VECLIB thread counts to 1 before the numeric backends import (covers direct
invocation).
- The Makefile `graphs` target also exports those plus PYTHONHASHSEED=0 before
the interpreter starts (PYTHONHASHSEED cannot be set from within the process);
CI runs `make graphs`, so it inherits this with no workflow change.

Regenerating every figure under these settings produces a zero-byte diff against
the committed images, confirming the current baseline is already the
single-threaded canonical output — so no figure files change here.

* fix: compare documentation figures within tolerance, not byte-exact

The "Documentation figures up to date" job regenerated .github/images with
`make graphs` and required a byte-identical result. That is unachievable on
GitHub's hardware-heterogeneous runner fleet: the pinned software stack
computes a few plotted path coordinates ~1 ULP apart depending on which CPU
microarchitecture (SIMD kernel) the run lands on, so the job failed
intermittently even though the figures were numerically and visually
identical. Single-thread pinning removes intra-machine reduction races but
cannot remove this cross-CPU last-bit difference.

Replace the byte diff with scripts/check_figures.py, which compares the
freshly regenerated figures against the committed versions within a tolerance:

* SVG -- non-numeric structure (elements, text, colours, ordering) must match
exactly; every numeric token must agree within an absolute/relative
tolerance. A moved element or relabelled axis fails; a last-bit
coordinate wobble passes.
* PNG -- identical dimensions, at most a small number of pixels changed beyond
a level threshold (catches localised changes a global RMS dilutes),
and a bounded RMS (catches broad changes the pixel count misses).
* other -- exact byte compare. Added/removed files always fail.

The thread-pinning determinism work from the previous commit is retained (it
still removes genuine intra-machine flakiness); this makes the check robust to
the cross-machine floating-point non-determinism it cannot eliminate.

* fix: harden figure check against hash-id drift and orphan figures

Two robustness gaps found reviewing the tolerance-aware figure check:

* SVG clip-path / marker ids are SHA256 hashes of the underlying float
geometry (matplotlib backend_svg._make_id). A cross-CPU 1-ULP wobble in that
geometry avalanches the whole hash, so the id text would change and trip the
strict structural gate -- the very drift the check exists to tolerate. Now
canonicalise every id and its url(#...)/href="#..." references to
first-appearance placeholders before comparing, so equivalent documents match
regardless of hash text while a genuine id add/remove/reorder/repoint fails.

* The generators only overwrite files, never delete, so a figure that is no
longer produced would survive on disk byte-identical to HEAD and pass the
"committed figure no longer generated" check. Clear the generated SVG/PNG at
the start of the `graphs` target before regenerating (animations *.gif/*.webm
are from the separate `animations` target and are preserved). A full
clear+regenerate reproduces the committed tree exactly, confirming no
committed figure is orphaned.

authored by

José M. Requena Plens and committed by

GitHub 2 weeks ago (Jul 11, 2026, 11:09 AM +0200) eb46d2c8 8345151a

+267 -33

6 changed files

Expand all

.github

workflows

python-app.yml

Makefile

requirements-figures.txt

scripts

check_figures.py

generate_diagrams.py

generate_graphs.py

+17 -2

Makefile

··· 17 17 PNPM = pnpm 18 18 endif 19 19 20 + # Deterministic figure rendering: pin numerical thread pools to one thread and 21 + # fix the hash seed BEFORE the interpreter starts, so multi-threaded reductions 22 + # and set ordering cannot perturb the committed SVG/PNG bytes across machines 23 + # (this is what made the heavy compute figures flaky on CI). The scripts also 24 + # set the thread vars internally; PYTHONHASHSEED can only be set from here. 25 + FIGURE_ENV = OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \ 26 + NUMEXPR_NUM_THREADS=1 NUMBA_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 \ 27 + PYTHONHASHSEED=0 28 + 20 29 install: 21 30 $(PYTHON) -m pip install --upgrade pip 22 31 $(PYTHON) -m pip install -r requirements.txt ··· 43 52 @if [ -f .env ]; then export $$(cat .env | xargs) && $(PNPM) exec sonar-scanner; else $(PNPM) exec sonar-scanner; fi 44 53 45 54 graphs: 46 - $(PYTHON) scripts/generate_graphs.py 47 - $(PYTHON) scripts/generate_diagrams.py 55 + # Clear the generated SVG/PNG first so a figure that is no longer produced 56 + # is actually removed (the generators only overwrite, never delete, so a 57 + # stale orphan would otherwise survive and slip past the staleness check). 58 + # Animations (*.gif/*.webm) come from the separate `animations` target and 59 + # are deliberately preserved. 60 + find .github/images -maxdepth 1 -type f $ -name '*.svg' -o -name '*.png' $ -delete 61 + $(FIGURE_ENV) $(PYTHON) scripts/generate_graphs.py 62 + $(FIGURE_ENV) $(PYTHON) scripts/generate_diagrams.py 48 63 49 64 # Regenerate the Tier-1 documentation animations (WebM for the site, GIF for 50 65 # the GitHub docs). Kept out of `graphs`/CI because the ffmpeg encoding is slow

+9 -6

requirements-figures.txt

··· 1 - # Locked rendering + compute stack for the reproducible-figures CI check 1 + # Locked rendering + compute stack for the figures CI check 2 2 # (the "Documentation figures up to date" job in python-app.yml). 3 3 # 4 - # The committed .github/images figures are byte-reproducible only against a 5 - # fixed stack: matplotlib/fonttools/pillow fix SVG text layout and PNG 6 - # encoding, while numpy/scipy/contourpy fix the *computed* data and therefore 7 - # the plotted path coordinates. Pinning only matplotlib is not enough -- a new 8 - # numpy or fonttools release silently shifts a few figures. 4 + # This pins the figure *structure*, not the last bit of every coordinate: 5 + # matplotlib/fonttools/pillow fix SVG text layout and PNG encoding, while 6 + # numpy/scipy/contourpy fix the computed data. Pinning only matplotlib is not 7 + # enough -- a new numpy or fonttools release changes element ordering, labels 8 + # or dimensions, which the structural comparison in scripts/check_figures.py 9 + # treats as a real change. Cross-CPU ~1-ULP coordinate drift, by contrast, is 10 + # absorbed by that script's numeric tolerance, so the stack does not have to 11 + # byte-match the runner hardware. 9 12 # 10 13 # Bump these together with a fresh `make graphs` regeneration (generate the 11 14 # figures with exactly this stack, then commit both). Generated from the

+195

scripts/check_figures.py

··· 1 + # Copyright (c) 2026. Jose M. Requena-Plens 2 + """Tolerance-aware staleness check for the committed documentation figures. 3 + 4 + The ``Documentation figures up to date`` CI job regenerates ``.github/images`` 5 + with ``make graphs`` and must confirm the result still matches what is 6 + committed. A byte-exact ``git diff`` cannot do this reliably: GitHub's runner 7 + fleet is hardware-heterogeneous, so the same pinned software stack computes a 8 + handful of plotted path coordinates ~1 ULP apart depending on which CPU 9 + microarchitecture the run lands on (a different SIMD kernel in the numpy/BLAS 10 + stack). That sub-pixel drift is numerically and visually irrelevant, yet a 11 + byte diff flags the figure as stale, which made the job intermittently fail. 12 + 13 + This script compares the freshly regenerated working-tree figures against the 14 + committed versions (``git HEAD``) within a tolerance instead: 15 + 16 + * **SVG** -- the non-numeric structure (elements, text, colours, ordering) 17 + must be identical, and every numeric token must agree within an absolute or 18 + relative tolerance. A moved element, changed label or new path fails; a 19 + last-bit coordinate wobble passes. 20 + * **PNG** -- identical dimensions, and *both* (a) at most ``PNG_MAX_SIG_PIXELS`` 21 + pixels whose per-channel difference exceeds ``PNG_LEVEL_TOL`` and (b) a 22 + per-pixel root-mean-square difference below ``PNG_RMS_TOL``. The pixel count 23 + catches a *localised* change (a moved line, a relabelled axis) that a global 24 + RMS would dilute in a large image; the RMS catches a *broad* change (a 25 + recoloured background) that few-but-everywhere pixels would slip past the 26 + count. Cross-CPU sub-ULP coordinate drift changes neither meaningfully. 27 + * **Anything else** -- exact byte compare. 28 + 29 + Added or removed files always fail: a new figure must be committed, and a 30 + figure that is no longer generated must be removed from the tree. 31 + 32 + The check is intentionally strict about *structure* and lenient only about the 33 + *last digits of numbers*, so it still catches every real figure change while 34 + being immune to cross-CPU floating-point non-determinism. 35 + """ 36 + 37 + from __future__ import annotations 38 + 39 + import io 40 + import re 41 + import subprocess # noqa: S404 (fixed argv, no shell, trusted git invocation) 42 + import sys 43 + from pathlib import Path 44 + 45 + import numpy as np 46 + from PIL import Image 47 + 48 + IMG_DIR = ".github/images" 49 + 50 + # Numeric tokens differing by less than this (absolute, in SVG user units, or 51 + # relative for large magnitudes) are treated as equal. A 1-ULP coordinate 52 + # wobble is ~1e-6; a real edit moves a coordinate by whole units. 53 + SVG_ABS_TOL = 1e-2 54 + SVG_REL_TOL = 1e-4 55 + 56 + # A pixel counts as "meaningfully changed" if any channel differs by more than 57 + # PNG_LEVEL_TOL (0..255). Cross-CPU drift perturbs a coordinate by ~1e-6 units, 58 + # i.e. a ~1e-5-pixel geometric shift, whose anti-aliasing effect rounds to at 59 + # most a level or two on a few edge pixels -- far below this threshold. 60 + PNG_LEVEL_TOL = 12 61 + # Allowed number of meaningfully-changed pixels. A real edit moves a plotted 62 + # line or glyph, changing hundreds to thousands of edge pixels; noise changes 63 + # a handful at most. 64 + PNG_MAX_SIG_PIXELS = 100 65 + # Broad-change guard: maximum root-mean-square per-channel difference. Catches 66 + # a change spread thinly over the whole image (e.g. a recoloured background) 67 + # that stays individually under PNG_LEVEL_TOL. 68 + PNG_RMS_TOL = 2.0 69 + 70 + # Integers and decimals, with optional sign and exponent. ``split``/``findall`` 71 + # with this pattern partition a file into fixed text and numeric values. 72 + _TOKEN = re.compile(rb"-?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?") 73 + 74 + # matplotlib derives clip-path / marker / collection ids by hashing the literal 75 + # ``str()`` of the underlying float geometry (see backend_svg._make_id). A 76 + # cross-CPU 1-ULP wobble in that geometry avalanches the whole SHA256, so the 77 + # id *text* would change even though the figure is numerically identical -- the 78 + # exact drift this check exists to tolerate. Canonicalising every id and its 79 + # ``url(#...)`` / ``href="#..."`` references to placeholders numbered by first 80 + # appearance removes the hash text from the structural comparison while still 81 + # catching a genuine change (an id added, removed, reordered, or a reference 82 + # repointed changes the placeholder sequence). 83 + _IDREF = re.compile(rb'(\bid="|url\(#|xlink:href="#|\bhref="#)([A-Za-z_][\w.:\-]*)') 84 + 85 + 86 + def _committed(path: str) -> bytes | None: 87 + """Return the bytes of ``path`` as committed at HEAD, or ``None``.""" 88 + result = subprocess.run( # noqa: S603 89 + ["git", "show", f"HEAD:{path}"], # noqa: S607 90 + capture_output=True, 91 + ) 92 + return result.stdout if result.returncode == 0 else None 93 + 94 + 95 + def _tracked_files() -> set[str]: 96 + """Return the set of ``.github/images`` paths tracked at HEAD.""" 97 + result = subprocess.run( # noqa: S603 98 + ["git", "ls-tree", "-r", "--name-only", "HEAD", IMG_DIR], # noqa: S607 99 + capture_output=True, 100 + text=True, 101 + check=True, 102 + ) 103 + return {line for line in result.stdout.splitlines() if line} 104 + 105 + 106 + def _canonicalize_ids(data: bytes) -> bytes: 107 + """Rewrite hash-derived ids/references to first-appearance placeholders.""" 108 + mapping: dict[bytes, bytes] = {} 109 + 110 + def repl(match: re.Match[bytes]) -> bytes: 111 + token = match.group(2) 112 + placeholder = mapping.setdefault(token, b"ID%d" % len(mapping)) 113 + return match.group(1) + placeholder 114 + 115 + return _IDREF.sub(repl, data) 116 + 117 + 118 + def _svg_within_tolerance(old: bytes, new: bytes) -> bool: 119 + """True if two SVGs match structurally and numerically within tolerance.""" 120 + old = _canonicalize_ids(old) 121 + new = _canonicalize_ids(new) 122 + if _TOKEN.split(old) != _TOKEN.split(new): 123 + return False # non-numeric structure (elements/text/colours) differs 124 + old_nums = _TOKEN.findall(old) 125 + new_nums = _TOKEN.findall(new) 126 + if len(old_nums) != len(new_nums): 127 + return False 128 + for o_tok, n_tok in zip(old_nums, new_nums): 129 + o_val = float(o_tok) 130 + n_val = float(n_tok) 131 + limit = max(SVG_ABS_TOL, SVG_REL_TOL * max(abs(o_val), abs(n_val))) 132 + if abs(o_val - n_val) > limit: 133 + return False 134 + return True 135 + 136 + 137 + def _png_problem(old: bytes, new: bytes) -> str | None: 138 + """Describe how two PNGs differ beyond tolerance, or ``None`` if within it.""" 139 + a = np.asarray(Image.open(io.BytesIO(old)).convert("RGBA"), dtype=np.float64) 140 + b = np.asarray(Image.open(io.BytesIO(new)).convert("RGBA"), dtype=np.float64) 141 + if a.shape != b.shape: 142 + return f"dimensions changed {a.shape} != {b.shape}" 143 + diff = np.abs(a - b) 144 + sig_pixels = int(np.count_nonzero(diff.max(axis=-1) > PNG_LEVEL_TOL)) 145 + if sig_pixels > PNG_MAX_SIG_PIXELS: 146 + return f"{sig_pixels} pixels changed by >{PNG_LEVEL_TOL} (> {PNG_MAX_SIG_PIXELS})" 147 + rms = float(np.sqrt(np.mean(diff**2))) 148 + if rms > PNG_RMS_TOL: 149 + return f"RMS {rms:.3f} > {PNG_RMS_TOL}" 150 + return None 151 + 152 + 153 + def main() -> int: 154 + disk = {str(p) for p in Path(IMG_DIR).rglob("*") if p.is_file()} 155 + tracked = _tracked_files() 156 + problems: list[str] = [] 157 + 158 + for path in sorted(disk - tracked): 159 + problems.append(f"new figure not committed: {path}") 160 + for path in sorted(tracked - disk): 161 + problems.append(f"committed figure no longer generated: {path}") 162 + 163 + for path in sorted(disk & tracked): 164 + new = Path(path).read_bytes() 165 + old = _committed(path) 166 + if old is None: 167 + problems.append(f"cannot read committed {path}") 168 + continue 169 + if old == new: 170 + continue # byte-identical: fast path, no tolerance needed 171 + if path.endswith(".svg"): 172 + if not _svg_within_tolerance(old, new): 173 + problems.append(f"SVG changed beyond tolerance: {path}") 174 + elif path.endswith(".png"): 175 + reason = _png_problem(old, new) 176 + if reason is not None: 177 + problems.append(f"PNG changed beyond tolerance ({reason}): {path}") 178 + else: 179 + problems.append(f"asset changed: {path}") 180 + 181 + if problems: 182 + print( 183 + "::error::.github/images is out of date - " 184 + "run 'make graphs' and commit the result." 185 + ) 186 + for problem in problems: 187 + print(f" - {problem}") 188 + return 1 189 + 190 + print(f"All {len(disk & tracked)} committed figures match within tolerance.") 191 + return 0 192 + 193 + 194 + if __name__ == "__main__": 195 + sys.exit(main())

+11 -2

scripts/generate_diagrams.py

··· 11 11 from __future__ import annotations 12 12 13 13 import os 14 - from collections.abc import Callable 15 - from dataclasses import dataclass 14 + 15 + # Deterministic output: pin numerical thread pools to a single thread before any 16 + # numeric backend initializes (see generate_graphs.py for the rationale). 17 + for _threads_var in ( 18 + "OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", 19 + "NUMEXPR_NUM_THREADS", "NUMBA_NUM_THREADS", "VECLIB_MAXIMUM_THREADS", 20 + ): 21 + os.environ.setdefault(_threads_var, "1") 22 + 23 + from collections.abc import Callable # noqa: E402 24 + from dataclasses import dataclass # noqa: E402 16 25 17 26 18 27 @dataclass(frozen=True)

+19 -7

scripts/generate_graphs.py

··· 1 1 import os 2 2 import sys 3 - from functools import lru_cache 4 - from typing import Any, Literal 5 3 6 - import matplotlib.pyplot as plt 7 - import matplotlib.ticker as mticker 8 - import numpy as np 9 - from scipy import signal as scipy_signal 4 + # Deterministic figure output: pin every numerical thread pool to a single 5 + # thread BEFORE numpy/scipy/numba import their backends, so multi-threaded 6 + # reductions cannot reorder floating-point sums and perturb the rendered bytes 7 + # across machines (the CI "Documentation figures" job runs on a different core 8 + # count than a dev box, which is what made the heavy compute figures flaky). 9 + for _threads_var in ( 10 + "OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", 11 + "NUMEXPR_NUM_THREADS", "NUMBA_NUM_THREADS", "VECLIB_MAXIMUM_THREADS", 12 + ): 13 + os.environ.setdefault(_threads_var, "1") 14 + 15 + from functools import lru_cache # noqa: E402 16 + from typing import Any, Literal # noqa: E402 17 + 18 + import matplotlib.pyplot as plt # noqa: E402 19 + import matplotlib.ticker as mticker # noqa: E402 20 + import numpy as np # noqa: E402 21 + from scipy import signal as scipy_signal # noqa: E402 10 22 11 23 # Add src to path to use the local package 12 24 sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "src"))) 13 25 14 - from phonometry import OctaveFilterBank 26 + from phonometry import OctaveFilterBank # noqa: E402 15 27 16 28 # Constants for professional styling 17 29 # ---------------------------------------------------------------------------

+16 -16

.github/workflows/python-app.yml

··· 63 63 exit 1 64 64 fi 65 65 66 - # The committed documentation figures (.github/images) must equal a fresh 67 - # `make graphs` run. Byte reproducibility needs the exact rendering + compute 68 - # stack the committed files were generated with: matplotlib/fonttools/pillow 69 - # fix SVG text layout and PNG encoding, numpy/scipy fix the computed data (and 70 - # therefore the plotted path coordinates). That full stack is pinned in 71 - # requirements-figures.txt -- pinning matplotlib alone is not enough, a newer 72 - # numpy or fonttools silently shifts a few figures. Bump that file together 73 - # with a fresh `make graphs` regeneration. 66 + # The committed documentation figures (.github/images) must match a fresh 67 + # `make graphs` run. The rendering + compute stack is pinned in 68 + # requirements-figures.txt so SVG *structure* (elements, text, colours) and 69 + # PNG dimensions/encoding are stable -- matplotlib/fonttools/pillow fix the 70 + # layout, numpy/scipy fix the computed data. The comparison itself 71 + # (scripts/check_figures.py) is tolerance-aware rather than a byte diff: 72 + # GitHub's heterogeneous runner CPUs shift a few path coordinates ~1 ULP, 73 + # which is visually irrelevant but breaks a byte compare. Bump the pinned 74 + # stack together with a fresh `make graphs` regeneration. 74 75 figures: 75 76 name: Documentation figures up to date 76 77 runs-on: ubuntu-latest ··· 92 93 - name: Regenerate figures and diagrams 93 94 run: make graphs 94 95 - name: Fail if any committed figure is stale 95 - run: | 96 - # Stage first so the diff also catches added/removed files, not just 97 - # in-place edits (plain `git diff` ignores untracked output). 98 - git add -A -- .github/images 99 - if ! git diff --cached --exit-code -- .github/images; then 100 - echo "::error::.github/images is out of date - run 'make graphs' and commit the result." 101 - exit 1 102 - fi 96 + # Tolerance-aware compare instead of a byte diff: GitHub's runner fleet is 97 + # hardware-heterogeneous, so the pinned stack computes a few path 98 + # coordinates ~1 ULP apart depending on the CPU the run lands on. That 99 + # sub-pixel drift is irrelevant but breaks a byte diff; the script checks 100 + # SVG structure + numeric tolerance and PNG RMS instead, so real figure 101 + # changes still fail while cross-CPU noise passes. See scripts/check_figures.py. 102 + run: python scripts/check_figures.py 103 103 104 104 tests: 105 105 runs-on: ${{ matrix.os }}

Configure Feed

Configure Feed