Commits
Operator visual verdict on iter 107: algpseudocodex blocks in ag-0018
and uts-001F (1st alg) render with 'chaos chars' on the right-hand-side
commands and comments. Root cause: tectonic's XDV emits cmdXGlyphArray
with FONT-SPECIFIC glyph indices into the LM font's glyph table;
redirecting font lookup to cmcsc10.pfb without remapping indices makes
dvisvgm pull wrong characters from CM's table. The pseudo block (2nd
alg) was unaffected by chance — its specific characters happen to map
to the same indices in both font families.
Iter 108: shrink rewrite_dvi.py to ONLY the pdf:pagesize -> xyz:pagesize
replacement (iter 69 baseline). Drop the LM->CM substitution entirely.
Residual difference: tngl uses LM glyph shapes via native XDV references,
gh-pages uses CM via Type1; both render correctly but with slightly
different appearances. The 'space after a/b' collapse in pseudo
block also accepted as residual — not fixable via DVI rewrite (would
require font-aware glyph table remapping for every cm/lm font pair).
Refs #2
[AGENT: kael]
iter 106 confirmed python failed on every shim invocation with
'IndentationError'. iter 107's first attempt used a quoted heredoc
('PYEOF') to write the python to a file, but the 0-indented python
lines exited the YAML block scalar and broke pipeline parsing
('could not find expected':'').
iter 107 (v2): use 'printf %s\n' with each python line as a
single-quoted argument. All lines stay within the YAML block scalar
(6-space leading); printf assembles them with newlines into
/tmp/bin/rewrite_dvi.py. Tested locally that the generated file
py_compiles cleanly.
The shim then invokes 'python3 /tmp/bin/rewrite_dvi.py $tmp' (file,
not -c). Same length-preserving LM->CM substitution.
[AGENT: kael]
iter 105 deployed byte-identical to iter 102 — strong signal the
LM-to-CM python substitution never actually ran. Add per-invocation
log lines:
1. on entry to the shim's XDV branch (echo to /tmp/iter106-shim.log)
2. inside python after the for-loop (size_d, size_d2, hits=[lm_names])
3. on python failure (separate echo)
Then stash the first 50 lines of /tmp/iter106-shim.log into
output/forest/_debug-shim.log so I can read it via 'git clone -b site'
without spindle TUI access.
[AGENT: kael]
iter 103/104 render workflows failed in 2-5 minutes with the same
narrowing logic that iter 102 succeeded with. Suspecting either
(a) f-string assertion behavior under set -e in the shim, or
(b) unicode chars in the python -c comments getting mangled at the
heredoc layer.
Replace assert + f-string with a silent 'if len(lm) == len(cm)'
length check. Rewrite comments to pure ASCII (no →, no LM->CM
arrow). ALSO fix a length bug discovered while typing:
lmromancaps10-regular is 21 bytes, not 22; iter 102 had
cmcsc10 + 15 NULs = 22 bytes which would have failed the
assertion silently (and may have left the DVI unmodified for
that pattern, explaining why iter 102 looked successful).
[AGENT: kael]
iter 103 render failed at 2m3s — too fast to be code-related. Likely
transient Spindle infrastructure issue. Empty commit to retrigger
the pipeline; if iter 104 also fails fast, the issue is in the iter
103 code change and I'll dig further.
[AGENT: kael]
Operator visual verdict on iter 102: 2nd algorithm in uts-001F is
"almost all good" except space after 'a' / 'b' is collapsed in
'a is largest' / 'b is largest'. The collapse comes from substituting
cmr10 / cmbx10 for the body+bold text — tectonic computed
inter-glyph positions using lmroman10-regular widths, so swapping
to CM glyph metrics narrows the space character and inter-word
gaps. Revert those two substitutions; keep only the small-caps and
italic ones (lmromancaps10-regular → cmcsc10, lmroman10-italic →
cmti10) which the operator confirmed visually correct.
Refs #2
[AGENT: kael]
Operator wants gh-pages parity. Per iter 14 (nf<N> is NOT a fallback)
the actual residual difference is tectonic emitting Latin Modern
native font names in XDV (lmroman10-italic etc.) where gh-pages'
lualatex emits Computer Modern (cmti10 etc.). Both render, but with
slightly different glyph shapes.
Surgical fix in the dvisvgm shim, before piping the buffered DVI to
dvisvgm:
cm_subs = [
(b'lmromancaps10-regular', b'cmcsc10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0'), # 22 bytes
(b'lmroman10-italic', b'cmti10\0\0\0\0\0\0\0\0\0\0'), # 16 bytes
(b'lmroman10-bold', b'cmbx10\0\0\0\0\0\0\0\0'), # 14 bytes
(b'lmroman10-regular', b'cmr10\0\0\0\0\0\0\0\0\0\0\0\0'), # 17 bytes
]
Length-preserving rewrites so DVI offsets / length-prefix bytes stay
intact. dvisvgm reads psname_len bytes into a C++ string ('cmcsc10\0\0\0...');
FileSystem::exists(name.c_str()) sees only 'cmcsc10' (C-string null
terminator stops the read). Symlinks 'cmcsc10' → 'cmcsc10.pfb'
(extensionless, like iter 94 did for the LM OTFs) so the cwd lookup
resolves.
If this works the SVG will show font-family:cmcsc10 / cmti10 / cmbx10
/ cmr10 (gh-pages-equivalent) instead of nf-N. Visible byte/glyph
diff for ag-0018 and uts-001F's pseudo block.
Refs #2
[AGENT: kael]
Iter 100 created OTF symlinks via 'find TEXLIVE_TEXMF' but I have no
visibility into whether TEXLIVE_TEXMF was set or how many symlinks
landed. Add env + symlink summary to deployed _debug-staging.log so I
can read the actual state via clone.
Refs #2
[AGENT: kael]
iter 99 set TEXMF/TEXMFDIST env vars but dvisvgm STILL emits nf-fonts
for lmroman/cmcsc/cmti. Hypothesis: dvisvgm's libkpathsea (linked at
build time against texlive.bin.core) ignores TEXMF env var because it
was already initialized with a different config.
iter 100 bypasses kpathsea entirely for OTF/TTF: walk forest-texlive's
TEXMF (TEXLIVE_TEXMF/fonts/opentype/) and create symlinks in PWD for
every .otf / .ttf, both with extension AND without. dvisvgm's
FileFinder._additionalDirs lookup (which starts with cwd and walks
through stat()) will find them before falling back to its broken
libkpathsea — same approach that works for cm*.pfb / rm-*.tfm.
Refs #2
[AGENT: kael]
iter 98 evidence (the stashed kpsewhich verification ran inside the
build step) showed forest-texlive's bin/kpsewhich resolves
cmcsc10.pfb / lmr10.pfb / etc. through
/nix/store/<hash>-texlive-combined-2025-texmfdist/...
But dvisvgm uses its OWN libkpathsea (linked at build time against
texlive.bin.core) and doesn't pick that path up from PATH alone.
Fix: set TEXMF + TEXMFDIST env vars to point at the combined tree
before invoking the build (and per dvisvgm shim invocation). Now
dvisvgm's libkpathsea-based FileFinder searches the same TEXMF tree
that kpsewhich uses.
The actual path is obtained via 'kpsewhich -var-value TEXMFDIST'
from forest-texlive's bin/, with a fallback to /share
(texlive.combine symlinks /share to the combined-texmf store path).
If this works: cmcsc10/cmti10/lmroman10-* native font lookups should
finally land — operator-visible visible fix in next CDN refresh.
Refs #2
[AGENT: kael]
iter 97 + earlier confirmed dvisvgm's runtime kpathsea only sees PWD;
TEXMFAUXTREES and forest-texlive's TEXMF tree weren't being honored
for the .enc / OTF / .pfb chain we care about.
dvisvgm source (FontEncoding.cpp:43):
if (FileFinder::instance().lookup(encname + ".enc", false)) {...}
// no .enc file found => try to find a CMap
When this lookup fails, dvisvgm emits 'CMap file \'NAME\' not found'
(misleading — it's looking for .enc first). The operator wants the
CMap chain fixed even if it takes minimal patching; the minimum
necessary here is making the .enc files actually findable.
Iter 98: blanket-symlink every *.enc from FOREST_TEX_DIR into PWD.
We already stage lm, pxfonts, newpx, amsfonts which collectively
have all the lm-mathit.enc / lm-rm.enc / px-lms.enc / lm-rep-cm*.enc
files dvisvgm asks for. Mirroring the iter 87 cm*.pfb pattern.
Refs #2
[AGENT: kael]
iter 96 successfully built + uploaded forest-texlive (277MB NAR, in
manifest under .assets.texlive), but ag-0018 / e62c50c SVGs are
byte-identical to iter 94 (no texlive) — nf-fallbacks for cmcsc10/cmti10/
lmroman10-italic etc. unchanged. Warnings about lm-mathit/lm-rm CMaps
also unchanged. So either:
(a) The texmf-dist tree isn't where my hardcoded path expects;
(b) kpathsea isn't actually honoring TEXMFAUXTREES for OTF lookup.
iter 97 probes multiple candidate locations for the texmf-dist tree
under /share/ (and fallbacks), prints a diagnostic
when none match, sets OSFONTDIR explicitly, and runs kpsewhich
against forest-texlive's own bin to see if it can resolve
lmroman10-italic.otf via standard kpathsea lookup. This narrows
between (a) and (b).
Refs #2
[AGENT: kael]
iter 95 build-depot failed with 'error: attribute l3keys2e missing'.
l3keys2e is shipped inside l3packages (already in the combo), not as
a separate texlive attribute in nixpkgs. Also drop algpseudocodex
preemptively — it's a younger CTAN package that may not be in older
nixpkgs texlive sets either.
If iter 96 still fails on another missing attribute, the error log
will name it; iterate further from there.
Refs #2
[AGENT: kael]
After iter 87-94 couldn't get dvisvgm to load LM OTF native font
references that tectonic emits in XDV (lmroman10-italic, etc.) via
extension-less PWD symlinks + OPENTYPEFONTS env hacks, switch to a
real TEXMF tree via texlive.combine.
Changes:
flake.nix — forest-texlive RE-ENABLED in packages output;
texCombo extended with lm, lm-math, cm-super, pxfonts,
mathpazo, psnfss, algorithmicx, algpseudocodex, pseudo,
tcolorbox, todonotes, utfsym, l3keys2e, pgfplots, tipa,
physics — covering every font + macro pkg the render
has been staging piecemeal via stage_tl_pkg.
build-depot.yml — adds 'nix build .#forest-texlive', packages it as
forest-texlive-x86_64.nar.zst with full closure,
adds 'texlive' entry to manifest.json + the R2
upload list.
render.yml — manifest asset discovery picks up .assets.texlive (or
warns + falls back if missing — for backward compat with
iter 94's manifest until build-depot re-runs);
nix-store --imports the texlive closure;
adds /bin to PATH (mktexlsr, kpsewhich, etc.);
sets TEXLIVE_TEXMF=/share/texmf-dist;
dvisvgm shim now prepends TEXLIVE_TEXMF to TEXMFAUXTREES
so kpathsea searches the canonical fonts/<type>/public/<pkg>/
tree first.
Heavier rebuild: depot will rebuild from scratch with the texlive
combine (~5-10m on top of existing tectonic + forester builds).
Refs #2
[AGENT: kael]
iter 93's OPENTYPEFONTS env didn't move the needle. dvisvgm's
FontManager::registerFont (Font.cpp) calls FileSystem::exists(fname)
with the BARE name from tectonic's XDV cmdXFontDef (no extension —
'lmroman10-italic', not 'lmroman10-italic.otf'). Without an
extension, that literal-name check fails, registerFont returns -1,
and dvisvgm emits an nf-font.
iter 94 creates extensionless symlinks for the 6 LM native fonts:
lmroman10-bold -> lmroman10-bold.otf
lmroman10-italic -> lmroman10-italic.otf
lmroman10-regular -> lmroman10-regular.otf
lmromancaps10-regular -> lmromancaps10-regular.otf
lmromandemi10-regular -> lmromandemi10-regular.otf
lmroman9-regular -> lmroman9-regular.otf
If iter 94 lands, e62c50c (pseudo block) and ag-0018 should both
show the LM small caps + italic + regular + bold fonts instead of
nf2/3/5/7 — operator-visible.
Refs #2
[AGENT: kael]
iter 91 DVI capture revealed tectonic emits XDV native font refs as
bare filenames (lmroman10-italic, lmromancaps10-regular) rather than
through psfonts.map. For type1 fonts, psfonts.map has explicit
'<filename.pfb' directives that dvisvgm resolves via cwd. For OTF
native font lookups, dvisvgm goes through kpathsea's OPENTYPEFONTS
search path — which (per iter 86 kpsewhich evidence) is empty in
our setup.
Set:
OPENTYPEFONTS = .::/fonts/opentype//
TTFONTS = .::/fonts/truetype//
TYPE1FONTS = .::/fonts/type1//
Note iter 54 also tried setting TFMFONTS/TYPE1FONTS and regressed
(3 placeholders -> 7). The risk surface is the same; if iter 93 regresses
similarly, revert immediately.
Refs #2
[AGENT: kael]
iter 91 DVI capture (dvi-001-1cb0845c.bin) confirmed the pseudo block
(uts-001F's e62c50c.svg) has 7 font_def opcodes:
cmmi10 cmr10 cmsy10 ← working
lmroman10-bold lmroman10-italic lmroman10-regular lmromancaps10-regular ← nf-fallbacks
Tectonic emits these as native XDV font names (not as cmcsc10/cmti10
which is what gh-pages' lualatex+TL2025 emits). The lm package has all
4 OTFs at fonts/opentype/public/lm/ and they're in our targeted symlink
list since iter 59. But the SVG shows nf-fallbacks, so something about
the symlink chain is broken for OTFs.
iter 92 stashes per-file ls -la for these OTFs + a fallback note
'present in FOREST_TEX_DIR but NOT symlinked into PWD' / 'NOT staged'
so we know whether the issue is at the staging step or the symlink step.
Refs #2
[AGENT: kael]
iter 90 saved the first 20 DVIs but uts-001F's pseudo block (e62c50c.svg)
wasn't in that window — none of the 20 captured DVIs contained cmcsc10/
cmti10/cmbx10 native font references. Bump to 200 and add the first 8
chars of the DVI's MD5 to the filename so I can correlate the captured
DVI with the SVG output file.
Refs #2
[AGENT: kael]
iter 89's deploy is now live on CDN (49310 bytes) but cmcsc10/cmti10
still nf-fallback. Operator confirmed two pseudocode blocks in uts-001F:
first (algpseudocodex, 2330c77 svg) works, second (Frédéric Wang's
pseudo, e62c50c svg) is the visible-overlap one.
Direct font compare on e62c50c (pseudo block): tngl uses 7 fonts
including 4 nf-fallbacks, gh-pages uses 6 including the same cm
fonts (cmbx10, cmcsc10, cmti10) we know don't resolve in our setup.
Notably ag-0018 has cmbx10 WORKING but e62c50c has cmbx10 as nf-font.
Same .pfb file, two different DVIs, two different results — suggests
tectonic emits cmbx10 at different design sizes between them.
iter 90 modifies the dvisvgm shim to save the first 20 buffered DVI
byte streams to /tmp/dvi-samples/dvi-NNN.bin, then tars them into
output/forest/_debug-dvi-samples.tar.gz at deploy. I can then read
the cmdXFontDef opcodes locally and see exactly what native font
reference tectonic emits for each cmbx10 instance.
Refs #2
[AGENT: kael]
iter 88 confirmed cmcsc10.pfb/tfm + cmti10.pfb/tfm are symlinked into PWD
(146 symlinks total). But dvisvgm still emits nf-fonts for them. iter 87
fixed cmr10/cmbx10 etc. because they kept their LM-rep psfonts.map entry
(routed via lm-rep-cmrm.enc to lmr10.pfb) — and iter 87 also symlinked
that .enc into PWD (no wait, it didn't). Let me look at why cmr10 works.
Actually: iter 84 stripped ONLY cmcsc10 + cmti10 LM-rep entries; cmr10's
LM-rep kept. So cmr10 tries LM-rep first (lmr10.pfb via lm-rep-cmrm.enc),
which works because lmr10.pfb is symlinked in PWD. The .enc file may
or may not be needed — dvisvgm may inline-encode if it can't find one.
iter 89:
(a) revert iter 84 strip — restore LM-rep entry for cmcsc10/cmti10
(commented out, kept for archaeology).
(b) ALSO add lm-rep-cm{rm,sc,it,itt,tt,in}.enc files to the targeted
PWD symlink list so dvisvgm can find them at runtime via cwd
lookup (kpathsea-equivalent for our setup, per iter 86 evidence).
Refs #2
[AGENT: kael]
Iter 87 produced visible byte change (48088 -> 49307) and eliminated the
rm-* + 'can't embed' + 'cmr10.pfb not found' warnings, but ag-0018 still
renders cmcsc10/cmti10 as nf-fallbacks. Two possibilities:
(a) symlinks for cmcsc10 didn't actually land in PWD (e.g. file
missing from FOREST_TEX_DIR)
(b) symlinks landed but dvisvgm still can't load the PFBs
iter 88 stashes ls -la output for the key files in PWD to the
deployed _debug-staging.log. If MISSING, we look at staging; if
present, we look at dvisvgm.
(side note: tangled CDN at utensil.tngl.sh is still serving the
iter-86 SVG (48088) even though site branch has iter-87's (49307);
operator-facing latency unrelated to fixes themselves.)
Refs #2
[AGENT: kael]
iter 86 was the breakthrough: kpsewhich could NOT find ANY font in our setup
(including lmr10.pfb which renders fine in production). That means dvisvgm
isn't using kpathsea/TEXMF lookup at all at runtime — it reads from
PWD (cwd) where the iter-58/59 targeted symlink list places SPECIFIC font
files. Everything we added to FOREST_TEX_DIR / FOREST_TEXMF / psfonts.map /
ls-R was IRRELEVANT.
The targeted symlink list hand-curated which fonts get symlinked from
FOREST_TEX_DIR into forester's MAIN cwd $PWD. Missing from the list:
cmcsc10, cmti10, cmr10/7, cmbx10, cmmi10/7, cmsy10/7, plus the rm-lm* VF
shim TFMs that dvisvgm tries to load for the LM-rep fallback chain.
Adding all of them now. If iter 87 lands, ag-0018 should switch from
nf2/3/5/7 -> cmcsc10/cmti10 (and the operator sees the overlap go away
in their browser).
Refs #2
[AGENT: kael]
Iter 85 confirmed every relevant file is on disk in BOTH the flat
FOREST_TEX_DIR AND the canonical FOREST_TEXMF tree (rm-*.tfm at
fonts/tfm/public/lm/, cmcsc10.pfb mirrored to fonts/type1/public/cm/,
ls-R indexed, psfonts.map has the direct mapping). But the SVG bytes
DID NOT CHANGE (48088 across iters 80-85), and the warnings about
'can't find rm-lmb10.tfm' persist.
Hypothesis: kpathsea at iter staging context AND at dvisvgm runtime
search different paths — TEXMFAUXTREES is only exported inside the
dvisvgm shim (per-call), so a sanity kpsewhich here wouldn't pick it
up unless we set it manually.
iter 86 runs kpsewhich in 3 contexts:
(a) bare environment (what build sees by default)
(b) with TEXMFAUXTREES explicitly set (what shim sees)
(c) dvisvgm --version + --list-fonts (any config hints)
Refs #2
[AGENT: kael]
Iter 84's surgical psfonts.map strip worked at the file level
(verified: 1 cmcsc10 entry pointing direct to cmcsc10.pfb), but the
SVG bytes didn't change. Reread the warnings log — the loud failures
are NOT 'cmcsc10.pfb not found' (we have it staged) but:
WARNING: can't find rm-lmb10.tfm
WARNING: can't find rm-lmr{6,7,9,10}.tfm
WARNING: can't find rm-lmri{7,10}.tfm
These rm-*.tfm files DO exist in lm.tar.xz under fonts/tfm/public/lm/
and our stage_tl_pkg flat-copies all *.tfm — so they're on disk in
both FOREST_TEX_DIR AND FOREST_TEXMF tree, but kpathsea can't find
them at dvisvgm runtime.
Iter 85 verifies their presence in both stores and rebuilds ls-R
cache to make sure it's indexed before build runs (not after).
Refs #2
[AGENT: kael]
Iter 83 mirrored the cm PFBs into canonical fonts/type1/public/cm/
and rebuilt ls-R, but ag-0018 still shows nf2/3/5/7 — psfonts.map's
lm-rep redirect (first match wins) routes cmcsc10/cmti10 to
lmcsc10.pfb / lmri10.pfb via .enc files; lm-rep-cmsc.enc IS in the
lm package's fonts/enc/dvips/lm/ but the lookup still doesn't
succeed, and dvisvgm gives up rather than trying the second mapping.
iter 84 surgical: drop ONLY the lm-rep entries for cmcsc10 and
cmti10 (the two ag-0018 actually needs). Other CM fonts keep their
lm-rep redirect to lmr10.pfb / lmbx10.pfb (which work — those are
in lm package and resolve via FOREST_TEXMF), so iter 81's regression
on cmr10/cmbx10 doesn't recur. The surviving direct cmcsc10/cmti10
entries point at the PFBs iter 83 mirrored into canonical cm/ path.
Refs #2
[AGENT: kael]
Iter 79-82 dig narrowed it to: cmcsc10.pfb + cmti10.pfb DO land in
FOREST_TEXMF, but at fonts/type1/public/amsfonts/cm/ (amsfonts's own
sub-path), not at fonts/type1/public/cm/ which is where kpathsea's
standard CM lookup expects them. dvisvgm therefore can't resolve
cmcsc10 even though the file is on disk.
iter 83:
1. After all packages stage, cp amsfonts/cm/*.pfb into the
canonical cm/ path inside FOREST_TEXMF.
2. Rebuild the kpathsea ls-R cache so the newly-mirrored paths are
indexed (fall back to a manual 'ls -R > ls-R' if mktexlsr isn't
on PATH — texlive.bin.core may or may not surface it).
Iter 82's revert restored the lm-rep redirects in psfonts.map, so
other fonts (cmr10/cmbx10) keep working through their lmr10.pfb
route. cmcsc10/cmti10 fall through the redirect (lm-rep-cmsc.enc
missing), then dvisvgm gets a SECOND chance via the canonical cm/
path which iter 83 now populates.
Refs #2
[AGENT: kael]
Iter 81 stripped lm-rep-cm*.enc redirects from psfonts.map; the side
effect was MUCH WORSE — iter 81 _debug-warnings.log shows new failures:
WARNING: font file 'cmr10.pfb' not found
WARNING: font file 'cmbx10.pfb' not found
WARNING: font file 'cmr7.pfb' not found
WARNING: can't embed font 'cmr10'
The lm-rep redirect was the ONLY thing keeping cmr10 / cmbx10 / cmr7
working at runtime — it routed them to lmr10.pfb / lmbx10.pfb / lmr7.pfb
which kpathsea can find via the FOREST_TEXMF tree from the lm package.
The direct cm*.pfb mappings (from amsfonts) point at PFBs that ARE
present in FOREST_TEX_DIR — but evidently NOT on dvisvgm's runtime
search path for Type1 fonts. cmcsc10 + cmti10 fail for the same reason:
PFB present but unreachable.
Reverting. Real fix is to make FOREST_TEX_DIR (or just the cm*.pfb
subset) reachable to dvisvgm's t1 font lookup. Probably staging them
into the FOREST_TEXMF tree at canonical fonts/type1/public/amsfonts/cm/
paths (vs the current flat copy) — kpathsea's CM-by-name lookup goes
through that tree via TEXMFAUXTREES.
Refs #2
[AGENT: kael]
Iter 80's stashed psfonts.map check shows the root cause: aggregating
all staged .map files produces TWO entries per CM font — a Latin Modern
redirect (cmcsc10 -> lmcsc10.pfb via lm-rep-cmsc.enc, from the lm
package's cm-rep maps) AND a direct CM mapping (cmcsc10 -> cmcsc10.pfb,
from amsfonts). dvisvgm uses the FIRST entry, which is the lm-rep one,
then can't find lm-rep-cmsc.enc (not in our stage), so glyphs fall back
to nf-fonts (nf2/nf3/nf5/nf7).
Fix: sed-drop every line containing lm-rep-cm*.enc from psfonts.map.
The direct CM mappings stay (we have the PFBs from amsfonts), dvisvgm
finds them, no encoding-file dependency.
Refs #2
[AGENT: kael]
iter 79 confirmed cmcsc10.pfb + cmti10.pfb ARE staged in FOREST_TEX_DIR
(32KB + 38KB respectively) but dvisvgm still falls back to nf-fonts.
Two remaining hypotheses: aggregated psfonts.map has no mapping line
for these fonts (so dvisvgm doesn't know what file to load), or
psfonts.map has them but dvisvgm/kpathsea isn't honoring it at runtime.
Iter 80 appends psfonts.map grep results to the stashed _debug-
staging.log so we can tell which.
Refs #2
[AGENT: kael]
Per TeX Live tlpdb, cmcsc10.pfb + cmti10.pfb are bundled in the
'amsfonts' package (at fonts/type1/public/amsfonts/cm/), NOT in
cm-super (which uses sf* renamed PFBs and doesn't ship csc/ti
variants in t1.map). amsfonts is already staged via stage_tl_pkg
since iter 61.
Iter 79 swaps the verification echoes from sfcsc10.pfb/sfti10.pfb
(don't exist as files) to cmcsc10.pfb/cmti10.pfb directly, plus
total cm*.pfb counts in FOREST_TEX_DIR. If those are missing,
the amsfonts staging is the bug.
Refs #2
[AGENT: kael]
iter 77 verification echoes went to CI stdout only; spindle TUI doesn't
reliably surface them post-build. Tee the staging snapshot to
/tmp/iter78-staging.log then copy into output/forest/_debug-staging.log
alongside the existing _debug-{failures,warnings}.log. Lets me read the
actual state of cmcsc10.tfm / sfcsc10.pfb / cm-super-t1.map via a
plain clone of the site branch.
Refs #2
[AGENT: kael]
iter 75's cm/cm-super staging was a no-op, not a regression. Two changes:
1. stage_tl_pkg: bump curl --max-time from 60 to 240 (cm-super.tar.xz is
~64MB and was silently timing out — explained iter 75's null effect).
2. Add CI-log echoes verifying:
- cmcsc10.tfm + cmti10.tfm landed in FOREST_TEX_DIR
- cm-super sf*.pfb files are present (count)
- cm-super-t1.map is present
- cmcsc10 + cmti10 entries made it into the aggregated psfonts.map
If all checks fail to print real paths, the missing-fonts theory is
wrong and we look elsewhere. If they all succeed but dvisvgm still
nf-fallbacks them, the kpathsea lookup at runtime is the issue.
Refs #2
[AGENT: kael]
iter 75 added cm + cm-super hoping the missing cmcsc10/cmti10 TFMs
would resolve. Result was WORSE: ag-0018 went from 2 nf-fallbacks
(nf2/nf3) to 4 (nf2/nf3/nf5/nf7), with the same 'inconsistent
length values' warnings still present. cm's flat-copied bare TFMs
appear to have shadowed kpathsea-resolved versions used by other
glyphs.
Reverting back to iter 74's package set. Will revisit via the
FOREST_TEXMF tree (preserves canonical fonts/<type>/public/<pkg>/
structure) which avoids the flat-copy naming collision.
Refs #2
[AGENT: kael]
SVG diff of ag-0018 (tngl vs gh-pages) shows tngl rendering with nf2/nf3
(dvisvgm 'no font' fallback) where gh-pages uses cmcsc10 and cmti10. These
Computer Modern variants aren't in texlive.bin.core's kpathsea path at
runtime, so dvisvgm substitutes generic shapes with wrong metrics → text
glyphs overlap, especially in algorithm trees.
Stage the cm + cm-super packages via stage_tl_pkg so the matching TFM + PFB
land in FOREST_TEX_DIR. Render-only change, no depot rebuild (~7m total).
Refs #2
[AGENT: kael]
iter 73's fix (re-call lookup with *it) didn't actually fix the
'can't read font file otf' error — patch applied at source level
(verified) but the deployed binary still hits the bug. Likely a
subtle side-effect issue: _pathbuf is a static class member, so
re-calling lookup may interact with intermediate calls in unexpected
ways. Refactor: capture the path inside the find_if lambda via a
reference variable, so the path returned is exactly the one that
caused the lambda to return true.
[AGENT: kael]
iter 71's FontEngine fix landed (substr crash gone) but unmasked a
second upstream bug: FontManager.cpp's font-format fallback uses
algo::find_if over fontFormats[] {nullptr,"otf","ttf"} and
dereferences `*it` to get the path — which is the literal extension
string, not the lookup result. NativeFontImpl gets path="otf"; later
setFont("otf") returns 'can't read font file otf' and the figure
becomes a placeholder. Re-call FileFinder::lookup with the same args
to retrieve the actual resolved path. Patch file now contains both
hunks; verified to apply cleanly to a fresh 3.6 tarball.
[AGENT: kael]
Operator asked for a portable patch artifact. nix-patches/
dvisvgm-fontengine-substr-fix.patch is a clean diff (verified to
apply with patch -p1 on a fresh 3.6 tarball) that can be shipped
upstream and survives source-version bumps. Drops the iter 71 sed
shim from postPatch in favor of mkDerivation's standard 'patches'
attribute — Nix will fail loudly if it doesn't apply.
[AGENT: kael]
iter 70 saw a fresh tectonic NAR build but the failure log still
showed the same substr crash on the same 4 trees. Likely cause:
substituteInPlace's '--replace' silently no-matches and continues on
recent Nix versions. Switch to inline sed (# delimiter — pattern
contains || which breaks | delimiter), add before/after grep + a
hard exit 1 if the buggy line survives.
[AGENT: kael]
Root cause for ag-0018, uts-001F (x2), ca-000J tikz-cd placeholders:
dvisvgm 3.6's setFont() enters the sys:// branch on ANY name with
size <= 6 (because the bool is || not &&), then unconditionally calls
fname.substr(6) which throws basic_string::substr on names shorter
than 6 chars. Iter 67's stashed dvisvgm stderr showed __pos=6 > size=3
on a 3-char fname — exact match for this bug.
Fix: flip <= to > and || to && so only true sys://... paths enter the
branch. Triggers a build-depot rebuild (~14m) because dvisvgm is in
the depot NAR closure.
[AGENT: kael]
iter 69 broke yaml parsing because the multi-line python -c body had
0-space leading lines, which left the YAML block scalar mid-stream.
Indent each python line to match the heredoc's 6-space base (sed strip
in the shim writer leaves them at correct python indent at runtime).
[AGENT: kael]
iter 68 confirmed --bbox=min retry hits the same substr OOB. The crash
is in dvisvgm's preprocessing of the pdf:pagesize special (bbox-mode-
independent). In-place rename 'pdf:pagesize' -> 'xyz:pagesize' in the
buffered DVI/XDV stream (same byte length, so offsets stay intact);
SpecialManager then dispatches xyz: to no-op instead of crashing.
[AGENT: kael]
iter 67's stashed _debug-failures.log revealed all 4 placeholder trees
(ag-0018 / uts-001F x2 / ca-000J tikz-cd) fail with
'basic_string::substr: __pos (6) > size (3)' during DVI preprocessing,
NOT font resolution. The crash is in dvisvgm 3.6's pdf:pagesize special
handler. Switching --bbox=papersize to --bbox=min bypasses the special
parser entirely (uses content tight-fit bbox instead). Retry only fires
on the substr signature so well-rendered trees stay on papersize.
[AGENT: kael]
iter 65's adds regressed 4 trees (21bc037 ag-0018, 2330c77, 5d08d7c
ca-0001, e62c50c) back to 1x1 placeholders — same iter-58 substr
crash pattern, dvisvgm chokes on the version skew between staged and
bundled NewPX*/NewTX* font tables.
Revert. Back to 0 placeholders. Will need a different approach to
plug VF chain misses without breaking existing renders.
Operator pinged 'dig'. Comparing 064536 (ag-0017 sphere) tngl vs gh:
- tngl fonts: NewPXMI, pxmiaX, zplbmi
- gh fonts: NewPXBMI, NewPXMI, pxmiaX
tngl uses 'zplbmi' (broken — VF chain to NewPXBMI which we don't have).
gh resolves the VF and embeds NewPXBMI.pfb directly.
NewPXBMI.pfb / NewPXBMI_gnu.pfb / NewPXBBMI.pfb / NewPXMI{_gnu,}.tfm ARE
in newpx archive. Adding them + NewTXBMI/MI variants + ntx{bex,ex}*.pfb
+ ntxsy{b,r}alt.pfb to targeted symlinks.
iter 63 left 3 misses: ntxsyc, zplbmi, zplexx.
newpx's zplbmi.vf and zplexx.vf are virtual fonts whose internal
references chain to PFBs in the original pxfonts package (px/rpx*
glyph file naming). Stage pxfonts + symlink its PFBs into $PWD.
newtx's ntxsyc.vf likely also chains to pxbsyc.pfb (from pxfonts) so
same fix covers it.
iter 62 reduced misses but 6 remain: pxmiaX, txmiaSTbb, txsym, ntxsyc,
zplbmi, zplexx.
- pxmiaX.{tfm,pfb}: in newpx but not in my earlier list
- txmiaSTbb.{tfm,pfb}, txmiaX.{tfm,pfb}, txsym.{tfm,pfb}: in newtx
- ntxsyc.{tfm,pfb}: in newtx
For zplbmi/zplexx (VF chains in newpx with no direct .pfb), they
need a referenced font that the .vf file points at; without a way
to parse .vf inline these will persist.
iter 61's residual misses (6 fonts): stxscr, ntxsym, zplbmi, zplexx,
zplmia, zplsyc. Findings:
- stxscr.{tfm,pfb} + ntxsym lives in newtx.tar.xz (not stmaryrd/stix)
- zpl{bmi,exx,mia,syc} TFMs exist in newpx but they're VIRTUAL fonts
(.vf files) that chain to actual PFBs (fplmb.pfb etc. via mathpazo
or similar)
Iter 62:
1. Stage newtx package (provides stxscr.{tfm,pfb}, ntxsym).
2. Expand targeted symlinks with: stxscr/ntxsym files, fplm* (mathpazo
PFBs that newpx's VF chain references), zpl variants' VF files +
numbered variants (zplbmi{0,1}.vf etc.).
iter 60's warning log identified the silent font misses across all
figures (not just placeholders). Major categories:
Direct .pfb file misses:
- NewPXMI.pfb, NewPXMI_gnu.pfb (newpx)
- lmbx10.pfb, lmmi{7,9,10}.pfb, lmsy{7,10}.pfb (lm)
- lmmono8/lmroman10-{regular,bold,italic}/lmroman9 (lm OTF naming)
Font-name lookups (no file found):
- lmr{6,9}, lmri{7,10} (lm)
- msbm10 (amsfonts blackboard)
- stxscr (STIX script — not yet found in any TL package)
- zplbmi, zplexx, zplmia, zplsyc, zplsym (newpx variants)
Iter 61 changes:
1. Stage amsfonts package for msbm10.
2. Expand iter-59's targeted symlink list to cover all the LM/newpx
variants listed above. ~50 new symlinks.
Operator pointed at tt-0014 figures looking broken even though my
placeholder count says 0. Inspection: SVGs ARE produced (not 1×1) but
have fewer @font-face declarations than gh-pages' equivalents — some
fonts silently missing, glyphs render as blanks.
dvisvgm exits 0 even when it can't find some fonts; my shim only
captured the failure (non-zero exit) path. iter 60 adds a SUCCESS path
that also greps stderr for WARNING + font file lines and tees them to
/tmp/dvisvgm-warnings.log. The end-of-build dump shows unique font
misses sorted, so we can immediately see what to add to stage_tl_pkg.
iter 58 broke 4 working figures with 'basic_string::substr' crashes
because wholesale font symlinks conflicted with the bundle's font
parsing (the crash from iter 33-34 came back for specific .pfb files).
Narrow to ONLY the files mentioned in the 3 placeholders' dvisvgm
warnings: TeXGyrePagellaX-*.otf, pxsys.{pfb,tfm}, zplmi*, zplsy.*,
lmr7.{tfm,pfb}, lmr10.{tfm,pfb}, lmb10.{tfm,pfb}. Should fix the
3 originals without risking the other figures.
iter 57's TEXMFAUXTREES didn't take effect (maybe kpathsea version
in dvisvgm doesn't read TEXMFAUXTREES, or our texmf.cnf doesn't
declare it). Try a more direct route: symlink the staged font files
(.tfm/.otf/.pfb/.vf/.afm/.enc/.map/.ttf/.pfm) into $PWD which is
where forester invokes dvisvgm per its OCaml source.
cwd-relative kpathsea lookups will hit these symlinks. Restricted
to font files only — no .tex/.sty so we don't clobber project sources.
Belt+suspenders alongside iter 57.
stage_tl_pkg now ALSO unpacks into FOREST_TEXMF preserving the original
fonts/<type>/public/<pkg>/ + tex/latex/<pkg>/ structure. Then in the
dvisvgm shim --stdin path, export TEXMFAUXTREES=$FOREST_TEXMF// so
kpathsea adds the tree to its existing search cascade (additive — doesn't
override the bundle's path like iter 54's TFMFONTS/... did).
Hopefully resolves the 3 stubborn placeholders' font misses
(TeXGyrePagellaX-Regular.otf, pxsys.pfb, zplmi, lmr7, lmr10) since
all of those files ARE present in newpx.tar.xz + lm.tar.xz + others
that stage_tl_pkg has been processing — they just weren't reachable
because FOREST_TEX_DIR is flat and not a valid TEXMF tree.
iter 55 stable at 3 placeholders. Their dvisvgm warnings list:
- TeXGyrePagellaX-Regular.otf (in newpx; symlinks don't reach
forester's main cwd where dvisvgm runs)
- pxsys.pfb (not in any package staged so far)
- zplmi (in newpx; vf chain might need additional .pfb)
- lmr7/lmr10 (in lm)
Add three packages likely to fill the chain:
- mathpazo: Pazo/Pagella math, has fplmb.pfb chain
- psnfss: standard PostScript fonts (Symbol/psyr)
- lm-math: Latin Modern math (latinmodern-math.otf)
iter 54 set TFMFONTS/TYPE1FONTS/OPENTYPEFONTS/... to start with
FOREST_TEX_DIR, hoping to make dvisvgm find staged fonts at any cwd.
Result: placeholder count went from 3 to 7. Some other figures that
were rendering correctly (21bc037, 2330c77, 5d08d7, e62c50c — including
the ag-0018 algorithm figures) regressed.
Hypothesis: kpathsea's default cascade already searches well for the
figures that render; my explicit env vars overrode the working path
with one that's missing some fonts or directory structure.
Revert to iter 53 state (--fontmap=psfonts.map via shim, no font-search
env vars). Back to 3 placeholders.
iter 53's --fontmap fixed the 'default map files not found' miss but
the individual font lookups still fail because dvisvgm runs in
forester's MAIN cwd (per ocaml-forester LaTeX_pipeline.ml: cwd =
Eio.Stdenv.cwd env), NOT in the per-snippet tmp dir where the latex
shim ran. The font symlinks in tmp don't help — wrong cwd.
Fix: in the dvisvgm shim --stdin path, export TFMFONTS / TYPE1FONTS /
OPENTYPEFONTS / TRUETYPEFONTS / VFFONTS / AFMFONTS / ENCFONTS /
TEXFONTMAPS with FOREST_TEX_DIR prepended, so kpathsea finds the
staged fonts regardless of cwd.
iter 52b's psfonts.map staging didn't help — dvisvgm still warns
'none of the default map files could be found'. kpathsea's cascade
isn't finding the map via cwd or env vars for some reason.
dvisvgm has an explicit --fontmap=<filenames> flag (options.xml:fontmap)
that bypasses kpathsea entirely. Pass it from the shim if the staged
psfonts.map exists. Should resolve TeX font names (zplmi, rm-lmb10, etc.)
to the actual files in cwd.
iter 52 used 'cat $FOREST_TEX_DIR/*.map' which under set -e exits the
whole script on zero matches. Render bailed at 38s with a non-fatal-
looking command. Switch to find -exec which is safe under set -e.
iter 51's font symlinks DID land but the dvisvgm warnings persisted:
WARNING: none of the default map files could be found
WARNING: no font file found for 'zplmi'/'rm-lmb10'/...
The .tfm files in cwd aren't enough — kpathsea needs psfonts.map to
resolve TeX font names (zplmi, rm-lmb10, etc.) to actual files
(zplmi.tfm + .pfb, lmb10.tfm + .pfb). updmap normally does this
aggregation post-install; without it, dvisvgm gives up early.
Fix: after staging all the per-package map files (newpx.map, lm-*.map,
etc.), concatenate them into a single $FOREST_TEX_DIR/psfonts.map.
The latex shim's font symlink loop (iter 51) already covers .map, so
psfonts.map will be symlinked into per-snippet cwd alongside the tfms.
iter 50's dvisvgm failure dump revealed the actual bug:
WARNING: font file 'TeXGyrePagellaX-Regular.otf' not found
WARNING: no font file found for 'zplsy'/'zplmi'/'rm-lmr7'/'rm-lmr10'
DVI error: undefined font number 47
ALL of those font files ARE in newpx.tar.xz / lm.tar.xz that iter 48
staged into FOREST_TEX_DIR. But the latex shim's symlink loop only
covered '*.tex *.sty' — fonts (.tfm/.otf/.pfb/.vf/.afm/.enc/.map etc.)
never reached the per-snippet cwd where kpathsea actually searches.
Fix: extend the inner symlink loop to also symlink the font extensions.
Now the staged fonts (zplmi.tfm, zplsy.tfm, TeXGyrePagellaX-Regular.otf,
lmr10.tfm/.afm, etc.) reach cwd and dvisvgm finds them.
Operator visual verdict on iter 107: algpseudocodex blocks in ag-0018
and uts-001F (1st alg) render with 'chaos chars' on the right-hand-side
commands and comments. Root cause: tectonic's XDV emits cmdXGlyphArray
with FONT-SPECIFIC glyph indices into the LM font's glyph table;
redirecting font lookup to cmcsc10.pfb without remapping indices makes
dvisvgm pull wrong characters from CM's table. The pseudo block (2nd
alg) was unaffected by chance — its specific characters happen to map
to the same indices in both font families.
Iter 108: shrink rewrite_dvi.py to ONLY the pdf:pagesize -> xyz:pagesize
replacement (iter 69 baseline). Drop the LM->CM substitution entirely.
Residual difference: tngl uses LM glyph shapes via native XDV references,
gh-pages uses CM via Type1; both render correctly but with slightly
different appearances. The 'space after a/b' collapse in pseudo
block also accepted as residual — not fixable via DVI rewrite (would
require font-aware glyph table remapping for every cm/lm font pair).
Refs #2
[AGENT: kael]
iter 106 confirmed python failed on every shim invocation with
'IndentationError'. iter 107's first attempt used a quoted heredoc
('PYEOF') to write the python to a file, but the 0-indented python
lines exited the YAML block scalar and broke pipeline parsing
('could not find expected':'').
iter 107 (v2): use 'printf %s\n' with each python line as a
single-quoted argument. All lines stay within the YAML block scalar
(6-space leading); printf assembles them with newlines into
/tmp/bin/rewrite_dvi.py. Tested locally that the generated file
py_compiles cleanly.
The shim then invokes 'python3 /tmp/bin/rewrite_dvi.py $tmp' (file,
not -c). Same length-preserving LM->CM substitution.
[AGENT: kael]
iter 105 deployed byte-identical to iter 102 — strong signal the
LM-to-CM python substitution never actually ran. Add per-invocation
log lines:
1. on entry to the shim's XDV branch (echo to /tmp/iter106-shim.log)
2. inside python after the for-loop (size_d, size_d2, hits=[lm_names])
3. on python failure (separate echo)
Then stash the first 50 lines of /tmp/iter106-shim.log into
output/forest/_debug-shim.log so I can read it via 'git clone -b site'
without spindle TUI access.
[AGENT: kael]
iter 103/104 render workflows failed in 2-5 minutes with the same
narrowing logic that iter 102 succeeded with. Suspecting either
(a) f-string assertion behavior under set -e in the shim, or
(b) unicode chars in the python -c comments getting mangled at the
heredoc layer.
Replace assert + f-string with a silent 'if len(lm) == len(cm)'
length check. Rewrite comments to pure ASCII (no →, no LM->CM
arrow). ALSO fix a length bug discovered while typing:
lmromancaps10-regular is 21 bytes, not 22; iter 102 had
cmcsc10 + 15 NULs = 22 bytes which would have failed the
assertion silently (and may have left the DVI unmodified for
that pattern, explaining why iter 102 looked successful).
[AGENT: kael]
Operator visual verdict on iter 102: 2nd algorithm in uts-001F is
"almost all good" except space after 'a' / 'b' is collapsed in
'a is largest' / 'b is largest'. The collapse comes from substituting
cmr10 / cmbx10 for the body+bold text — tectonic computed
inter-glyph positions using lmroman10-regular widths, so swapping
to CM glyph metrics narrows the space character and inter-word
gaps. Revert those two substitutions; keep only the small-caps and
italic ones (lmromancaps10-regular → cmcsc10, lmroman10-italic →
cmti10) which the operator confirmed visually correct.
Refs #2
[AGENT: kael]
Operator wants gh-pages parity. Per iter 14 (nf<N> is NOT a fallback)
the actual residual difference is tectonic emitting Latin Modern
native font names in XDV (lmroman10-italic etc.) where gh-pages'
lualatex emits Computer Modern (cmti10 etc.). Both render, but with
slightly different glyph shapes.
Surgical fix in the dvisvgm shim, before piping the buffered DVI to
dvisvgm:
cm_subs = [
(b'lmromancaps10-regular', b'cmcsc10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0'), # 22 bytes
(b'lmroman10-italic', b'cmti10\0\0\0\0\0\0\0\0\0\0'), # 16 bytes
(b'lmroman10-bold', b'cmbx10\0\0\0\0\0\0\0\0'), # 14 bytes
(b'lmroman10-regular', b'cmr10\0\0\0\0\0\0\0\0\0\0\0\0'), # 17 bytes
]
Length-preserving rewrites so DVI offsets / length-prefix bytes stay
intact. dvisvgm reads psname_len bytes into a C++ string ('cmcsc10\0\0\0...');
FileSystem::exists(name.c_str()) sees only 'cmcsc10' (C-string null
terminator stops the read). Symlinks 'cmcsc10' → 'cmcsc10.pfb'
(extensionless, like iter 94 did for the LM OTFs) so the cwd lookup
resolves.
If this works the SVG will show font-family:cmcsc10 / cmti10 / cmbx10
/ cmr10 (gh-pages-equivalent) instead of nf-N. Visible byte/glyph
diff for ag-0018 and uts-001F's pseudo block.
Refs #2
[AGENT: kael]
iter 99 set TEXMF/TEXMFDIST env vars but dvisvgm STILL emits nf-fonts
for lmroman/cmcsc/cmti. Hypothesis: dvisvgm's libkpathsea (linked at
build time against texlive.bin.core) ignores TEXMF env var because it
was already initialized with a different config.
iter 100 bypasses kpathsea entirely for OTF/TTF: walk forest-texlive's
TEXMF (TEXLIVE_TEXMF/fonts/opentype/) and create symlinks in PWD for
every .otf / .ttf, both with extension AND without. dvisvgm's
FileFinder._additionalDirs lookup (which starts with cwd and walks
through stat()) will find them before falling back to its broken
libkpathsea — same approach that works for cm*.pfb / rm-*.tfm.
Refs #2
[AGENT: kael]
iter 98 evidence (the stashed kpsewhich verification ran inside the
build step) showed forest-texlive's bin/kpsewhich resolves
cmcsc10.pfb / lmr10.pfb / etc. through
/nix/store/<hash>-texlive-combined-2025-texmfdist/...
But dvisvgm uses its OWN libkpathsea (linked at build time against
texlive.bin.core) and doesn't pick that path up from PATH alone.
Fix: set TEXMF + TEXMFDIST env vars to point at the combined tree
before invoking the build (and per dvisvgm shim invocation). Now
dvisvgm's libkpathsea-based FileFinder searches the same TEXMF tree
that kpsewhich uses.
The actual path is obtained via 'kpsewhich -var-value TEXMFDIST'
from forest-texlive's bin/, with a fallback to /share
(texlive.combine symlinks /share to the combined-texmf store path).
If this works: cmcsc10/cmti10/lmroman10-* native font lookups should
finally land — operator-visible visible fix in next CDN refresh.
Refs #2
[AGENT: kael]
iter 97 + earlier confirmed dvisvgm's runtime kpathsea only sees PWD;
TEXMFAUXTREES and forest-texlive's TEXMF tree weren't being honored
for the .enc / OTF / .pfb chain we care about.
dvisvgm source (FontEncoding.cpp:43):
if (FileFinder::instance().lookup(encname + ".enc", false)) {...}
// no .enc file found => try to find a CMap
When this lookup fails, dvisvgm emits 'CMap file \'NAME\' not found'
(misleading — it's looking for .enc first). The operator wants the
CMap chain fixed even if it takes minimal patching; the minimum
necessary here is making the .enc files actually findable.
Iter 98: blanket-symlink every *.enc from FOREST_TEX_DIR into PWD.
We already stage lm, pxfonts, newpx, amsfonts which collectively
have all the lm-mathit.enc / lm-rm.enc / px-lms.enc / lm-rep-cm*.enc
files dvisvgm asks for. Mirroring the iter 87 cm*.pfb pattern.
Refs #2
[AGENT: kael]
iter 96 successfully built + uploaded forest-texlive (277MB NAR, in
manifest under .assets.texlive), but ag-0018 / e62c50c SVGs are
byte-identical to iter 94 (no texlive) — nf-fallbacks for cmcsc10/cmti10/
lmroman10-italic etc. unchanged. Warnings about lm-mathit/lm-rm CMaps
also unchanged. So either:
(a) The texmf-dist tree isn't where my hardcoded path expects;
(b) kpathsea isn't actually honoring TEXMFAUXTREES for OTF lookup.
iter 97 probes multiple candidate locations for the texmf-dist tree
under /share/ (and fallbacks), prints a diagnostic
when none match, sets OSFONTDIR explicitly, and runs kpsewhich
against forest-texlive's own bin to see if it can resolve
lmroman10-italic.otf via standard kpathsea lookup. This narrows
between (a) and (b).
Refs #2
[AGENT: kael]
iter 95 build-depot failed with 'error: attribute l3keys2e missing'.
l3keys2e is shipped inside l3packages (already in the combo), not as
a separate texlive attribute in nixpkgs. Also drop algpseudocodex
preemptively — it's a younger CTAN package that may not be in older
nixpkgs texlive sets either.
If iter 96 still fails on another missing attribute, the error log
will name it; iterate further from there.
Refs #2
[AGENT: kael]
After iter 87-94 couldn't get dvisvgm to load LM OTF native font
references that tectonic emits in XDV (lmroman10-italic, etc.) via
extension-less PWD symlinks + OPENTYPEFONTS env hacks, switch to a
real TEXMF tree via texlive.combine.
Changes:
flake.nix — forest-texlive RE-ENABLED in packages output;
texCombo extended with lm, lm-math, cm-super, pxfonts,
mathpazo, psnfss, algorithmicx, algpseudocodex, pseudo,
tcolorbox, todonotes, utfsym, l3keys2e, pgfplots, tipa,
physics — covering every font + macro pkg the render
has been staging piecemeal via stage_tl_pkg.
build-depot.yml — adds 'nix build .#forest-texlive', packages it as
forest-texlive-x86_64.nar.zst with full closure,
adds 'texlive' entry to manifest.json + the R2
upload list.
render.yml — manifest asset discovery picks up .assets.texlive (or
warns + falls back if missing — for backward compat with
iter 94's manifest until build-depot re-runs);
nix-store --imports the texlive closure;
adds /bin to PATH (mktexlsr, kpsewhich, etc.);
sets TEXLIVE_TEXMF=/share/texmf-dist;
dvisvgm shim now prepends TEXLIVE_TEXMF to TEXMFAUXTREES
so kpathsea searches the canonical fonts/<type>/public/<pkg>/
tree first.
Heavier rebuild: depot will rebuild from scratch with the texlive
combine (~5-10m on top of existing tectonic + forester builds).
Refs #2
[AGENT: kael]
iter 93's OPENTYPEFONTS env didn't move the needle. dvisvgm's
FontManager::registerFont (Font.cpp) calls FileSystem::exists(fname)
with the BARE name from tectonic's XDV cmdXFontDef (no extension —
'lmroman10-italic', not 'lmroman10-italic.otf'). Without an
extension, that literal-name check fails, registerFont returns -1,
and dvisvgm emits an nf-font.
iter 94 creates extensionless symlinks for the 6 LM native fonts:
lmroman10-bold -> lmroman10-bold.otf
lmroman10-italic -> lmroman10-italic.otf
lmroman10-regular -> lmroman10-regular.otf
lmromancaps10-regular -> lmromancaps10-regular.otf
lmromandemi10-regular -> lmromandemi10-regular.otf
lmroman9-regular -> lmroman9-regular.otf
If iter 94 lands, e62c50c (pseudo block) and ag-0018 should both
show the LM small caps + italic + regular + bold fonts instead of
nf2/3/5/7 — operator-visible.
Refs #2
[AGENT: kael]
iter 91 DVI capture revealed tectonic emits XDV native font refs as
bare filenames (lmroman10-italic, lmromancaps10-regular) rather than
through psfonts.map. For type1 fonts, psfonts.map has explicit
'<filename.pfb' directives that dvisvgm resolves via cwd. For OTF
native font lookups, dvisvgm goes through kpathsea's OPENTYPEFONTS
search path — which (per iter 86 kpsewhich evidence) is empty in
our setup.
Set:
OPENTYPEFONTS = .::/fonts/opentype//
TTFONTS = .::/fonts/truetype//
TYPE1FONTS = .::/fonts/type1//
Note iter 54 also tried setting TFMFONTS/TYPE1FONTS and regressed
(3 placeholders -> 7). The risk surface is the same; if iter 93 regresses
similarly, revert immediately.
Refs #2
[AGENT: kael]
iter 91 DVI capture (dvi-001-1cb0845c.bin) confirmed the pseudo block
(uts-001F's e62c50c.svg) has 7 font_def opcodes:
cmmi10 cmr10 cmsy10 ← working
lmroman10-bold lmroman10-italic lmroman10-regular lmromancaps10-regular ← nf-fallbacks
Tectonic emits these as native XDV font names (not as cmcsc10/cmti10
which is what gh-pages' lualatex+TL2025 emits). The lm package has all
4 OTFs at fonts/opentype/public/lm/ and they're in our targeted symlink
list since iter 59. But the SVG shows nf-fallbacks, so something about
the symlink chain is broken for OTFs.
iter 92 stashes per-file ls -la for these OTFs + a fallback note
'present in FOREST_TEX_DIR but NOT symlinked into PWD' / 'NOT staged'
so we know whether the issue is at the staging step or the symlink step.
Refs #2
[AGENT: kael]
iter 90 saved the first 20 DVIs but uts-001F's pseudo block (e62c50c.svg)
wasn't in that window — none of the 20 captured DVIs contained cmcsc10/
cmti10/cmbx10 native font references. Bump to 200 and add the first 8
chars of the DVI's MD5 to the filename so I can correlate the captured
DVI with the SVG output file.
Refs #2
[AGENT: kael]
iter 89's deploy is now live on CDN (49310 bytes) but cmcsc10/cmti10
still nf-fallback. Operator confirmed two pseudocode blocks in uts-001F:
first (algpseudocodex, 2330c77 svg) works, second (Frédéric Wang's
pseudo, e62c50c svg) is the visible-overlap one.
Direct font compare on e62c50c (pseudo block): tngl uses 7 fonts
including 4 nf-fallbacks, gh-pages uses 6 including the same cm
fonts (cmbx10, cmcsc10, cmti10) we know don't resolve in our setup.
Notably ag-0018 has cmbx10 WORKING but e62c50c has cmbx10 as nf-font.
Same .pfb file, two different DVIs, two different results — suggests
tectonic emits cmbx10 at different design sizes between them.
iter 90 modifies the dvisvgm shim to save the first 20 buffered DVI
byte streams to /tmp/dvi-samples/dvi-NNN.bin, then tars them into
output/forest/_debug-dvi-samples.tar.gz at deploy. I can then read
the cmdXFontDef opcodes locally and see exactly what native font
reference tectonic emits for each cmbx10 instance.
Refs #2
[AGENT: kael]
iter 88 confirmed cmcsc10.pfb/tfm + cmti10.pfb/tfm are symlinked into PWD
(146 symlinks total). But dvisvgm still emits nf-fonts for them. iter 87
fixed cmr10/cmbx10 etc. because they kept their LM-rep psfonts.map entry
(routed via lm-rep-cmrm.enc to lmr10.pfb) — and iter 87 also symlinked
that .enc into PWD (no wait, it didn't). Let me look at why cmr10 works.
Actually: iter 84 stripped ONLY cmcsc10 + cmti10 LM-rep entries; cmr10's
LM-rep kept. So cmr10 tries LM-rep first (lmr10.pfb via lm-rep-cmrm.enc),
which works because lmr10.pfb is symlinked in PWD. The .enc file may
or may not be needed — dvisvgm may inline-encode if it can't find one.
iter 89:
(a) revert iter 84 strip — restore LM-rep entry for cmcsc10/cmti10
(commented out, kept for archaeology).
(b) ALSO add lm-rep-cm{rm,sc,it,itt,tt,in}.enc files to the targeted
PWD symlink list so dvisvgm can find them at runtime via cwd
lookup (kpathsea-equivalent for our setup, per iter 86 evidence).
Refs #2
[AGENT: kael]
Iter 87 produced visible byte change (48088 -> 49307) and eliminated the
rm-* + 'can't embed' + 'cmr10.pfb not found' warnings, but ag-0018 still
renders cmcsc10/cmti10 as nf-fallbacks. Two possibilities:
(a) symlinks for cmcsc10 didn't actually land in PWD (e.g. file
missing from FOREST_TEX_DIR)
(b) symlinks landed but dvisvgm still can't load the PFBs
iter 88 stashes ls -la output for the key files in PWD to the
deployed _debug-staging.log. If MISSING, we look at staging; if
present, we look at dvisvgm.
(side note: tangled CDN at utensil.tngl.sh is still serving the
iter-86 SVG (48088) even though site branch has iter-87's (49307);
operator-facing latency unrelated to fixes themselves.)
Refs #2
[AGENT: kael]
iter 86 was the breakthrough: kpsewhich could NOT find ANY font in our setup
(including lmr10.pfb which renders fine in production). That means dvisvgm
isn't using kpathsea/TEXMF lookup at all at runtime — it reads from
PWD (cwd) where the iter-58/59 targeted symlink list places SPECIFIC font
files. Everything we added to FOREST_TEX_DIR / FOREST_TEXMF / psfonts.map /
ls-R was IRRELEVANT.
The targeted symlink list hand-curated which fonts get symlinked from
FOREST_TEX_DIR into forester's MAIN cwd $PWD. Missing from the list:
cmcsc10, cmti10, cmr10/7, cmbx10, cmmi10/7, cmsy10/7, plus the rm-lm* VF
shim TFMs that dvisvgm tries to load for the LM-rep fallback chain.
Adding all of them now. If iter 87 lands, ag-0018 should switch from
nf2/3/5/7 -> cmcsc10/cmti10 (and the operator sees the overlap go away
in their browser).
Refs #2
[AGENT: kael]
Iter 85 confirmed every relevant file is on disk in BOTH the flat
FOREST_TEX_DIR AND the canonical FOREST_TEXMF tree (rm-*.tfm at
fonts/tfm/public/lm/, cmcsc10.pfb mirrored to fonts/type1/public/cm/,
ls-R indexed, psfonts.map has the direct mapping). But the SVG bytes
DID NOT CHANGE (48088 across iters 80-85), and the warnings about
'can't find rm-lmb10.tfm' persist.
Hypothesis: kpathsea at iter staging context AND at dvisvgm runtime
search different paths — TEXMFAUXTREES is only exported inside the
dvisvgm shim (per-call), so a sanity kpsewhich here wouldn't pick it
up unless we set it manually.
iter 86 runs kpsewhich in 3 contexts:
(a) bare environment (what build sees by default)
(b) with TEXMFAUXTREES explicitly set (what shim sees)
(c) dvisvgm --version + --list-fonts (any config hints)
Refs #2
[AGENT: kael]
Iter 84's surgical psfonts.map strip worked at the file level
(verified: 1 cmcsc10 entry pointing direct to cmcsc10.pfb), but the
SVG bytes didn't change. Reread the warnings log — the loud failures
are NOT 'cmcsc10.pfb not found' (we have it staged) but:
WARNING: can't find rm-lmb10.tfm
WARNING: can't find rm-lmr{6,7,9,10}.tfm
WARNING: can't find rm-lmri{7,10}.tfm
These rm-*.tfm files DO exist in lm.tar.xz under fonts/tfm/public/lm/
and our stage_tl_pkg flat-copies all *.tfm — so they're on disk in
both FOREST_TEX_DIR AND FOREST_TEXMF tree, but kpathsea can't find
them at dvisvgm runtime.
Iter 85 verifies their presence in both stores and rebuilds ls-R
cache to make sure it's indexed before build runs (not after).
Refs #2
[AGENT: kael]
Iter 83 mirrored the cm PFBs into canonical fonts/type1/public/cm/
and rebuilt ls-R, but ag-0018 still shows nf2/3/5/7 — psfonts.map's
lm-rep redirect (first match wins) routes cmcsc10/cmti10 to
lmcsc10.pfb / lmri10.pfb via .enc files; lm-rep-cmsc.enc IS in the
lm package's fonts/enc/dvips/lm/ but the lookup still doesn't
succeed, and dvisvgm gives up rather than trying the second mapping.
iter 84 surgical: drop ONLY the lm-rep entries for cmcsc10 and
cmti10 (the two ag-0018 actually needs). Other CM fonts keep their
lm-rep redirect to lmr10.pfb / lmbx10.pfb (which work — those are
in lm package and resolve via FOREST_TEXMF), so iter 81's regression
on cmr10/cmbx10 doesn't recur. The surviving direct cmcsc10/cmti10
entries point at the PFBs iter 83 mirrored into canonical cm/ path.
Refs #2
[AGENT: kael]
Iter 79-82 dig narrowed it to: cmcsc10.pfb + cmti10.pfb DO land in
FOREST_TEXMF, but at fonts/type1/public/amsfonts/cm/ (amsfonts's own
sub-path), not at fonts/type1/public/cm/ which is where kpathsea's
standard CM lookup expects them. dvisvgm therefore can't resolve
cmcsc10 even though the file is on disk.
iter 83:
1. After all packages stage, cp amsfonts/cm/*.pfb into the
canonical cm/ path inside FOREST_TEXMF.
2. Rebuild the kpathsea ls-R cache so the newly-mirrored paths are
indexed (fall back to a manual 'ls -R > ls-R' if mktexlsr isn't
on PATH — texlive.bin.core may or may not surface it).
Iter 82's revert restored the lm-rep redirects in psfonts.map, so
other fonts (cmr10/cmbx10) keep working through their lmr10.pfb
route. cmcsc10/cmti10 fall through the redirect (lm-rep-cmsc.enc
missing), then dvisvgm gets a SECOND chance via the canonical cm/
path which iter 83 now populates.
Refs #2
[AGENT: kael]
Iter 81 stripped lm-rep-cm*.enc redirects from psfonts.map; the side
effect was MUCH WORSE — iter 81 _debug-warnings.log shows new failures:
WARNING: font file 'cmr10.pfb' not found
WARNING: font file 'cmbx10.pfb' not found
WARNING: font file 'cmr7.pfb' not found
WARNING: can't embed font 'cmr10'
The lm-rep redirect was the ONLY thing keeping cmr10 / cmbx10 / cmr7
working at runtime — it routed them to lmr10.pfb / lmbx10.pfb / lmr7.pfb
which kpathsea can find via the FOREST_TEXMF tree from the lm package.
The direct cm*.pfb mappings (from amsfonts) point at PFBs that ARE
present in FOREST_TEX_DIR — but evidently NOT on dvisvgm's runtime
search path for Type1 fonts. cmcsc10 + cmti10 fail for the same reason:
PFB present but unreachable.
Reverting. Real fix is to make FOREST_TEX_DIR (or just the cm*.pfb
subset) reachable to dvisvgm's t1 font lookup. Probably staging them
into the FOREST_TEXMF tree at canonical fonts/type1/public/amsfonts/cm/
paths (vs the current flat copy) — kpathsea's CM-by-name lookup goes
through that tree via TEXMFAUXTREES.
Refs #2
[AGENT: kael]
Iter 80's stashed psfonts.map check shows the root cause: aggregating
all staged .map files produces TWO entries per CM font — a Latin Modern
redirect (cmcsc10 -> lmcsc10.pfb via lm-rep-cmsc.enc, from the lm
package's cm-rep maps) AND a direct CM mapping (cmcsc10 -> cmcsc10.pfb,
from amsfonts). dvisvgm uses the FIRST entry, which is the lm-rep one,
then can't find lm-rep-cmsc.enc (not in our stage), so glyphs fall back
to nf-fonts (nf2/nf3/nf5/nf7).
Fix: sed-drop every line containing lm-rep-cm*.enc from psfonts.map.
The direct CM mappings stay (we have the PFBs from amsfonts), dvisvgm
finds them, no encoding-file dependency.
Refs #2
[AGENT: kael]
iter 79 confirmed cmcsc10.pfb + cmti10.pfb ARE staged in FOREST_TEX_DIR
(32KB + 38KB respectively) but dvisvgm still falls back to nf-fonts.
Two remaining hypotheses: aggregated psfonts.map has no mapping line
for these fonts (so dvisvgm doesn't know what file to load), or
psfonts.map has them but dvisvgm/kpathsea isn't honoring it at runtime.
Iter 80 appends psfonts.map grep results to the stashed _debug-
staging.log so we can tell which.
Refs #2
[AGENT: kael]
Per TeX Live tlpdb, cmcsc10.pfb + cmti10.pfb are bundled in the
'amsfonts' package (at fonts/type1/public/amsfonts/cm/), NOT in
cm-super (which uses sf* renamed PFBs and doesn't ship csc/ti
variants in t1.map). amsfonts is already staged via stage_tl_pkg
since iter 61.
Iter 79 swaps the verification echoes from sfcsc10.pfb/sfti10.pfb
(don't exist as files) to cmcsc10.pfb/cmti10.pfb directly, plus
total cm*.pfb counts in FOREST_TEX_DIR. If those are missing,
the amsfonts staging is the bug.
Refs #2
[AGENT: kael]
iter 77 verification echoes went to CI stdout only; spindle TUI doesn't
reliably surface them post-build. Tee the staging snapshot to
/tmp/iter78-staging.log then copy into output/forest/_debug-staging.log
alongside the existing _debug-{failures,warnings}.log. Lets me read the
actual state of cmcsc10.tfm / sfcsc10.pfb / cm-super-t1.map via a
plain clone of the site branch.
Refs #2
[AGENT: kael]
iter 75's cm/cm-super staging was a no-op, not a regression. Two changes:
1. stage_tl_pkg: bump curl --max-time from 60 to 240 (cm-super.tar.xz is
~64MB and was silently timing out — explained iter 75's null effect).
2. Add CI-log echoes verifying:
- cmcsc10.tfm + cmti10.tfm landed in FOREST_TEX_DIR
- cm-super sf*.pfb files are present (count)
- cm-super-t1.map is present
- cmcsc10 + cmti10 entries made it into the aggregated psfonts.map
If all checks fail to print real paths, the missing-fonts theory is
wrong and we look elsewhere. If they all succeed but dvisvgm still
nf-fallbacks them, the kpathsea lookup at runtime is the issue.
Refs #2
[AGENT: kael]
iter 75 added cm + cm-super hoping the missing cmcsc10/cmti10 TFMs
would resolve. Result was WORSE: ag-0018 went from 2 nf-fallbacks
(nf2/nf3) to 4 (nf2/nf3/nf5/nf7), with the same 'inconsistent
length values' warnings still present. cm's flat-copied bare TFMs
appear to have shadowed kpathsea-resolved versions used by other
glyphs.
Reverting back to iter 74's package set. Will revisit via the
FOREST_TEXMF tree (preserves canonical fonts/<type>/public/<pkg>/
structure) which avoids the flat-copy naming collision.
Refs #2
[AGENT: kael]
SVG diff of ag-0018 (tngl vs gh-pages) shows tngl rendering with nf2/nf3
(dvisvgm 'no font' fallback) where gh-pages uses cmcsc10 and cmti10. These
Computer Modern variants aren't in texlive.bin.core's kpathsea path at
runtime, so dvisvgm substitutes generic shapes with wrong metrics → text
glyphs overlap, especially in algorithm trees.
Stage the cm + cm-super packages via stage_tl_pkg so the matching TFM + PFB
land in FOREST_TEX_DIR. Render-only change, no depot rebuild (~7m total).
Refs #2
[AGENT: kael]
iter 73's fix (re-call lookup with *it) didn't actually fix the
'can't read font file otf' error — patch applied at source level
(verified) but the deployed binary still hits the bug. Likely a
subtle side-effect issue: _pathbuf is a static class member, so
re-calling lookup may interact with intermediate calls in unexpected
ways. Refactor: capture the path inside the find_if lambda via a
reference variable, so the path returned is exactly the one that
caused the lambda to return true.
[AGENT: kael]
iter 71's FontEngine fix landed (substr crash gone) but unmasked a
second upstream bug: FontManager.cpp's font-format fallback uses
algo::find_if over fontFormats[] {nullptr,"otf","ttf"} and
dereferences `*it` to get the path — which is the literal extension
string, not the lookup result. NativeFontImpl gets path="otf"; later
setFont("otf") returns 'can't read font file otf' and the figure
becomes a placeholder. Re-call FileFinder::lookup with the same args
to retrieve the actual resolved path. Patch file now contains both
hunks; verified to apply cleanly to a fresh 3.6 tarball.
[AGENT: kael]
Operator asked for a portable patch artifact. nix-patches/
dvisvgm-fontengine-substr-fix.patch is a clean diff (verified to
apply with patch -p1 on a fresh 3.6 tarball) that can be shipped
upstream and survives source-version bumps. Drops the iter 71 sed
shim from postPatch in favor of mkDerivation's standard 'patches'
attribute — Nix will fail loudly if it doesn't apply.
[AGENT: kael]
iter 70 saw a fresh tectonic NAR build but the failure log still
showed the same substr crash on the same 4 trees. Likely cause:
substituteInPlace's '--replace' silently no-matches and continues on
recent Nix versions. Switch to inline sed (# delimiter — pattern
contains || which breaks | delimiter), add before/after grep + a
hard exit 1 if the buggy line survives.
[AGENT: kael]
Root cause for ag-0018, uts-001F (x2), ca-000J tikz-cd placeholders:
dvisvgm 3.6's setFont() enters the sys:// branch on ANY name with
size <= 6 (because the bool is || not &&), then unconditionally calls
fname.substr(6) which throws basic_string::substr on names shorter
than 6 chars. Iter 67's stashed dvisvgm stderr showed __pos=6 > size=3
on a 3-char fname — exact match for this bug.
Fix: flip <= to > and || to && so only true sys://... paths enter the
branch. Triggers a build-depot rebuild (~14m) because dvisvgm is in
the depot NAR closure.
[AGENT: kael]
iter 68 confirmed --bbox=min retry hits the same substr OOB. The crash
is in dvisvgm's preprocessing of the pdf:pagesize special (bbox-mode-
independent). In-place rename 'pdf:pagesize' -> 'xyz:pagesize' in the
buffered DVI/XDV stream (same byte length, so offsets stay intact);
SpecialManager then dispatches xyz: to no-op instead of crashing.
[AGENT: kael]
iter 67's stashed _debug-failures.log revealed all 4 placeholder trees
(ag-0018 / uts-001F x2 / ca-000J tikz-cd) fail with
'basic_string::substr: __pos (6) > size (3)' during DVI preprocessing,
NOT font resolution. The crash is in dvisvgm 3.6's pdf:pagesize special
handler. Switching --bbox=papersize to --bbox=min bypasses the special
parser entirely (uses content tight-fit bbox instead). Retry only fires
on the substr signature so well-rendered trees stay on papersize.
[AGENT: kael]
iter 65's adds regressed 4 trees (21bc037 ag-0018, 2330c77, 5d08d7c
ca-0001, e62c50c) back to 1x1 placeholders — same iter-58 substr
crash pattern, dvisvgm chokes on the version skew between staged and
bundled NewPX*/NewTX* font tables.
Revert. Back to 0 placeholders. Will need a different approach to
plug VF chain misses without breaking existing renders.
Operator pinged 'dig'. Comparing 064536 (ag-0017 sphere) tngl vs gh:
- tngl fonts: NewPXMI, pxmiaX, zplbmi
- gh fonts: NewPXBMI, NewPXMI, pxmiaX
tngl uses 'zplbmi' (broken — VF chain to NewPXBMI which we don't have).
gh resolves the VF and embeds NewPXBMI.pfb directly.
NewPXBMI.pfb / NewPXBMI_gnu.pfb / NewPXBBMI.pfb / NewPXMI{_gnu,}.tfm ARE
in newpx archive. Adding them + NewTXBMI/MI variants + ntx{bex,ex}*.pfb
+ ntxsy{b,r}alt.pfb to targeted symlinks.
iter 63 left 3 misses: ntxsyc, zplbmi, zplexx.
newpx's zplbmi.vf and zplexx.vf are virtual fonts whose internal
references chain to PFBs in the original pxfonts package (px/rpx*
glyph file naming). Stage pxfonts + symlink its PFBs into $PWD.
newtx's ntxsyc.vf likely also chains to pxbsyc.pfb (from pxfonts) so
same fix covers it.
iter 62 reduced misses but 6 remain: pxmiaX, txmiaSTbb, txsym, ntxsyc,
zplbmi, zplexx.
- pxmiaX.{tfm,pfb}: in newpx but not in my earlier list
- txmiaSTbb.{tfm,pfb}, txmiaX.{tfm,pfb}, txsym.{tfm,pfb}: in newtx
- ntxsyc.{tfm,pfb}: in newtx
For zplbmi/zplexx (VF chains in newpx with no direct .pfb), they
need a referenced font that the .vf file points at; without a way
to parse .vf inline these will persist.
iter 61's residual misses (6 fonts): stxscr, ntxsym, zplbmi, zplexx,
zplmia, zplsyc. Findings:
- stxscr.{tfm,pfb} + ntxsym lives in newtx.tar.xz (not stmaryrd/stix)
- zpl{bmi,exx,mia,syc} TFMs exist in newpx but they're VIRTUAL fonts
(.vf files) that chain to actual PFBs (fplmb.pfb etc. via mathpazo
or similar)
Iter 62:
1. Stage newtx package (provides stxscr.{tfm,pfb}, ntxsym).
2. Expand targeted symlinks with: stxscr/ntxsym files, fplm* (mathpazo
PFBs that newpx's VF chain references), zpl variants' VF files +
numbered variants (zplbmi{0,1}.vf etc.).
iter 60's warning log identified the silent font misses across all
figures (not just placeholders). Major categories:
Direct .pfb file misses:
- NewPXMI.pfb, NewPXMI_gnu.pfb (newpx)
- lmbx10.pfb, lmmi{7,9,10}.pfb, lmsy{7,10}.pfb (lm)
- lmmono8/lmroman10-{regular,bold,italic}/lmroman9 (lm OTF naming)
Font-name lookups (no file found):
- lmr{6,9}, lmri{7,10} (lm)
- msbm10 (amsfonts blackboard)
- stxscr (STIX script — not yet found in any TL package)
- zplbmi, zplexx, zplmia, zplsyc, zplsym (newpx variants)
Iter 61 changes:
1. Stage amsfonts package for msbm10.
2. Expand iter-59's targeted symlink list to cover all the LM/newpx
variants listed above. ~50 new symlinks.
Operator pointed at tt-0014 figures looking broken even though my
placeholder count says 0. Inspection: SVGs ARE produced (not 1×1) but
have fewer @font-face declarations than gh-pages' equivalents — some
fonts silently missing, glyphs render as blanks.
dvisvgm exits 0 even when it can't find some fonts; my shim only
captured the failure (non-zero exit) path. iter 60 adds a SUCCESS path
that also greps stderr for WARNING + font file lines and tees them to
/tmp/dvisvgm-warnings.log. The end-of-build dump shows unique font
misses sorted, so we can immediately see what to add to stage_tl_pkg.
iter 58 broke 4 working figures with 'basic_string::substr' crashes
because wholesale font symlinks conflicted with the bundle's font
parsing (the crash from iter 33-34 came back for specific .pfb files).
Narrow to ONLY the files mentioned in the 3 placeholders' dvisvgm
warnings: TeXGyrePagellaX-*.otf, pxsys.{pfb,tfm}, zplmi*, zplsy.*,
lmr7.{tfm,pfb}, lmr10.{tfm,pfb}, lmb10.{tfm,pfb}. Should fix the
3 originals without risking the other figures.
iter 57's TEXMFAUXTREES didn't take effect (maybe kpathsea version
in dvisvgm doesn't read TEXMFAUXTREES, or our texmf.cnf doesn't
declare it). Try a more direct route: symlink the staged font files
(.tfm/.otf/.pfb/.vf/.afm/.enc/.map/.ttf/.pfm) into $PWD which is
where forester invokes dvisvgm per its OCaml source.
cwd-relative kpathsea lookups will hit these symlinks. Restricted
to font files only — no .tex/.sty so we don't clobber project sources.
Belt+suspenders alongside iter 57.
stage_tl_pkg now ALSO unpacks into FOREST_TEXMF preserving the original
fonts/<type>/public/<pkg>/ + tex/latex/<pkg>/ structure. Then in the
dvisvgm shim --stdin path, export TEXMFAUXTREES=$FOREST_TEXMF// so
kpathsea adds the tree to its existing search cascade (additive — doesn't
override the bundle's path like iter 54's TFMFONTS/... did).
Hopefully resolves the 3 stubborn placeholders' font misses
(TeXGyrePagellaX-Regular.otf, pxsys.pfb, zplmi, lmr7, lmr10) since
all of those files ARE present in newpx.tar.xz + lm.tar.xz + others
that stage_tl_pkg has been processing — they just weren't reachable
because FOREST_TEX_DIR is flat and not a valid TEXMF tree.
iter 55 stable at 3 placeholders. Their dvisvgm warnings list:
- TeXGyrePagellaX-Regular.otf (in newpx; symlinks don't reach
forester's main cwd where dvisvgm runs)
- pxsys.pfb (not in any package staged so far)
- zplmi (in newpx; vf chain might need additional .pfb)
- lmr7/lmr10 (in lm)
Add three packages likely to fill the chain:
- mathpazo: Pazo/Pagella math, has fplmb.pfb chain
- psnfss: standard PostScript fonts (Symbol/psyr)
- lm-math: Latin Modern math (latinmodern-math.otf)
iter 54 set TFMFONTS/TYPE1FONTS/OPENTYPEFONTS/... to start with
FOREST_TEX_DIR, hoping to make dvisvgm find staged fonts at any cwd.
Result: placeholder count went from 3 to 7. Some other figures that
were rendering correctly (21bc037, 2330c77, 5d08d7, e62c50c — including
the ag-0018 algorithm figures) regressed.
Hypothesis: kpathsea's default cascade already searches well for the
figures that render; my explicit env vars overrode the working path
with one that's missing some fonts or directory structure.
Revert to iter 53 state (--fontmap=psfonts.map via shim, no font-search
env vars). Back to 3 placeholders.
iter 53's --fontmap fixed the 'default map files not found' miss but
the individual font lookups still fail because dvisvgm runs in
forester's MAIN cwd (per ocaml-forester LaTeX_pipeline.ml: cwd =
Eio.Stdenv.cwd env), NOT in the per-snippet tmp dir where the latex
shim ran. The font symlinks in tmp don't help — wrong cwd.
Fix: in the dvisvgm shim --stdin path, export TFMFONTS / TYPE1FONTS /
OPENTYPEFONTS / TRUETYPEFONTS / VFFONTS / AFMFONTS / ENCFONTS /
TEXFONTMAPS with FOREST_TEX_DIR prepended, so kpathsea finds the
staged fonts regardless of cwd.
iter 52b's psfonts.map staging didn't help — dvisvgm still warns
'none of the default map files could be found'. kpathsea's cascade
isn't finding the map via cwd or env vars for some reason.
dvisvgm has an explicit --fontmap=<filenames> flag (options.xml:fontmap)
that bypasses kpathsea entirely. Pass it from the shim if the staged
psfonts.map exists. Should resolve TeX font names (zplmi, rm-lmb10, etc.)
to the actual files in cwd.
iter 51's font symlinks DID land but the dvisvgm warnings persisted:
WARNING: none of the default map files could be found
WARNING: no font file found for 'zplmi'/'rm-lmb10'/...
The .tfm files in cwd aren't enough — kpathsea needs psfonts.map to
resolve TeX font names (zplmi, rm-lmb10, etc.) to actual files
(zplmi.tfm + .pfb, lmb10.tfm + .pfb). updmap normally does this
aggregation post-install; without it, dvisvgm gives up early.
Fix: after staging all the per-package map files (newpx.map, lm-*.map,
etc.), concatenate them into a single $FOREST_TEX_DIR/psfonts.map.
The latex shim's font symlink loop (iter 51) already covers .map, so
psfonts.map will be symlinked into per-snippet cwd alongside the tfms.
iter 50's dvisvgm failure dump revealed the actual bug:
WARNING: font file 'TeXGyrePagellaX-Regular.otf' not found
WARNING: no font file found for 'zplsy'/'zplmi'/'rm-lmr7'/'rm-lmr10'
DVI error: undefined font number 47
ALL of those font files ARE in newpx.tar.xz / lm.tar.xz that iter 48
staged into FOREST_TEX_DIR. But the latex shim's symlink loop only
covered '*.tex *.sty' — fonts (.tfm/.otf/.pfb/.vf/.afm/.enc/.map etc.)
never reached the per-snippet cwd where kpathsea actually searches.
Fix: extend the inner symlink loop to also symlink the font extensions.
Now the staged fonts (zplmi.tfm, zplsy.tfm, TeXGyrePagellaX-Regular.otf,
lmr10.tfm/.afm, etc.) reach cwd and dvisvgm finds them.