fix(tui): render wide (astral-plane) glyphs correctly across incremental re-render (#51) · compounding.tech/pty@48d954a

+7

CHANGELOG.md

··· 2 2 3 3 ## Unreleased 4 4 5 + ### Fixes 6 + - **`@myobie/pty/tui` `CellBuffer` — wide-char (2-cell) glyph rendering fixed.** Two root causes, both in `src/tui/buffer.ts`: 7 + 1. `writeAnsi` iterated the input string by UTF-16 code unit, splitting astral-plane codepoints (emoji like `📬`, `📭`, `📫` — U+1F4XX) into two lone surrogate halves stored as separate width-1 cells. Downstream `diff()` and `fullRender()` emitted the halves independently; a modern host terminal (kitty, iTerm2, Ghostty) recombined them as one wide glyph, but the CellBuffer's cell-index → terminal-column mapping was off by one on every emoji. Fixed by detecting surrogate pairs and combining them into one Cell whose `char` holds the full 2-code-unit string. 8 + 2. `diff()`'s `lastCol = c + 1` cursor-position tracker didn't account for the wide char's 2-column advance. After a wide-char emit the terminal cursor lands at `c+2`, but `lastCol` said `c+1`, mispredicting adjacency for the next emit. Fixed by `lastCol = c + charWidth(nc.char)`. 9 + - Fixes the tui-sup `📬📬2 99` fossilization observed on the agent-viz cards grid when navigation shifted an emoji to a different column. 10 + - Regression coverage in `tests/buffer-wide-char-diff.test.ts` — 10 cases using xterm-headless + `@xterm/addon-unicode11` as the strong oracle (real-terminal semantics for astral-plane codepoints), verifying that `fullRender(prev) + diff(prev, next)` produces a terminal state equivalent to `fullRender(next)` across shift/replace/toggle/mixed-content scenarios. 11 + 5 12 ### `pty gc` — abandoned-reap for permanent sessions 6 13 - **New `pty gc` step 1.5: abandoned-reap.** Runs between orphan-kill (step 1) and permanent-respawn (step 2). Reaps live `strategy=permanent` sessions detected as abandoned — SIGTERM the daemon (if alive), append a `session_abandoned` event, then `cleanupAll`. Two shapes today: 7 14 - **cwd-gone (on-by-default)** — the session's recorded `cwd` no longer resolves on disk (`fs.statSync` throws `ENOENT`). Strong low-false-positive signal. Escape hatch: `strategy.abandon-if-cwd-gone=false` tag opts a session out.

+1 -1

flake.nix

··· 29 29 30 30 # Generated from package-lock.json. 31 31 # Regenerate with: nix run nixpkgs#prefetch-npm-deps -- package-lock.json 32 - npmDepsHash = "sha256-wchxdkrsGEIykFZWP0eHa2vKCpEYnj4rYKVUEhuw5j4="; 32 + npmDepsHash = "sha256-y1hEiPG6kZLilTHF6dH7Ql/X9ddDBucsZYJqETlTuyE="; 33 33 34 34 # node-pty has native code that needs these at build time 35 35 nativeBuildInputs = with pkgs; [ python3 pkg-config ];

+8

package-lock.json

··· 22 22 "devDependencies": { 23 23 "@preact/signals-core": "^1.14.0", 24 24 "@types/node": "^25.3.5", 25 + "@xterm/addon-unicode11": "^0.9.0", 25 26 "esbuild": "^0.27.4", 26 27 "typescript": "^5.7.0", 27 28 "vitest": "^4.1.1" ··· 980 981 "version": "0.14.0", 981 982 "resolved": "https://registry.npmjs.org/@xterm/addon-serialize/-/addon-serialize-0.14.0.tgz", 982 983 "integrity": "sha512-uteyTU1EkrQa2Ux6P/uFl2fzmXI46jy5uoQMKEOM0fKTyiW7cSn0WrFenHm5vO5uEXX/GpwW/FgILvv3r0WbkA==", 984 + "license": "MIT" 985 + }, 986 + "node_modules/@xterm/addon-unicode11": { 987 + "version": "0.9.0", 988 + "resolved": "https://registry.npmjs.org/@xterm/addon-unicode11/-/addon-unicode11-0.9.0.tgz", 989 + "integrity": "sha512-FxDnYcyuXhNl+XSqGZL/t0U9eiNb/q3EWT5rYkQT/zuig8Gz/VagnQANKHdDWFM2lTMk9ly0EFQxxxtZUoRetw==", 990 + "dev": true, 983 991 "license": "MIT" 984 992 }, 985 993 "node_modules/@xterm/headless": {

+1

package.json

··· 76 76 "devDependencies": { 77 77 "@preact/signals-core": "^1.14.0", 78 78 "@types/node": "^25.3.5", 79 + "@xterm/addon-unicode11": "^0.9.0", 79 80 "esbuild": "^0.27.4", 80 81 "typescript": "^5.7.0", 81 82 "vitest": "^4.1.1"

+83 -44

src/tui/buffer.ts

··· 154 154 curCol = 0; 155 155 i++; 156 156 } else { 157 - // Printable character — may be wide (2 cells) 158 - const ch = ansi[i]; 157 + // Printable character — may be wide (2 cells) OR a surrogate pair 158 + // encoding an astral-plane codepoint (emoji, CJK ideographs above 159 + // U+FFFF). `ansi[i]` returns a single UTF-16 code unit, so a lone 160 + // high-surrogate for a non-BMP character would be treated as a 161 + // narrow char — splitting `📬` into two width-1 cells and letting 162 + // the diff renderer fossilize the halves independently. 163 + // Combine surrogate pairs into a single Cell before measuring width. 164 + let ch = ansi[i]; 165 + const code = ch.charCodeAt(0); 166 + if (code >= 0xd800 && code <= 0xdbff && i + 1 < ansi.length) { 167 + const low = ansi.charCodeAt(i + 1); 168 + if (low >= 0xdc00 && low <= 0xdfff) { 169 + ch = ansi.slice(i, i + 2); 170 + } 171 + } 159 172 const cw = charWidth(ch); 160 173 if (curRow >= 0 && curRow < this.rows && curCol >= 0 && curCol < this.cols) { 161 174 this.cells[curRow][curCol] = { ··· 185 198 } 186 199 } 187 200 curCol += cw; 188 - i++; 201 + i += ch.length; 189 202 } 190 203 } 191 204 } ··· 278 291 } 279 292 280 293 /** Diff two buffers and emit minimal ANSI to update from prev to next. 281 - * Uses DEC synchronized output (mode 2026) to prevent tearing. */ 294 + * Uses DEC synchronized output (mode 2026) to prevent tearing. 295 + * 296 + * Wide-char handling: 297 + * - `lastCol` tracks where the terminal cursor lands after each emit. 298 + * For a width-2 glyph, that's `c + 2`, not `c + 1`. Without accounting 299 + * for the width, cursor-adjacency checks mispredict after every wide 300 + * char, and — combined with the placeholder-skip below — can leave 301 + * the right half of a prev-frame wide char visible on screen. 302 + * - When `next.cells[r][c]` is a placeholder (`char === ""`), we skip it 303 + * because the wide char at `c-1` already claims that column. But if 304 + * `prev.cells[r][c-1]` was NOT a wide char (i.e. it's a real narrow 305 + * char, or `prev` had a wide char at `c-1` that's now been overwritten 306 + * with a narrow at c-1's position via a diff earlier in this pass), 307 + * then the terminal at column c may still hold prev-frame content 308 + * (a fossil). We emit an explicit space to clear it before letting 309 + * the enclosing wide char's downstream repaint the position. 310 + * 311 + * Reference: #47 tui-sup bug — `📬` fossilizing to `📬📬2 99` when 312 + * navigating a cards grid caused the glyph to shift columns between 313 + * frames. */ 282 314 export function diff(prev: CellBuffer, next: CellBuffer): string { 283 315 let out = "\x1b[?2026h"; // Begin synchronized update 284 316 let lastRow = -1; ··· 290 322 let lastUnderline = false; 291 323 let needsReset = true; 292 324 325 + const emit = (r: number, c: number, ch: string, styleCell: Cell): void => { 326 + if (r !== lastRow || c !== lastCol) { 327 + out += `\x1b[${r + 1};${c + 1}H`; 328 + } 329 + 330 + const needReset = 331 + (styleCell.bold !== lastBold && !styleCell.bold) || 332 + (styleCell.dim !== lastDim && !styleCell.dim) || 333 + (styleCell.italic !== lastItalic && !styleCell.italic) || 334 + (styleCell.underline !== lastUnderline && !styleCell.underline) || 335 + needsReset; 336 + 337 + if (needReset) { 338 + out += "\x1b[0m"; 339 + resetState(state); 340 + lastBold = false; 341 + lastDim = false; 342 + lastItalic = false; 343 + lastUnderline = false; 344 + needsReset = false; 345 + } 346 + 347 + if (styleCell.bold && !lastBold) out += "\x1b[1m"; 348 + if (styleCell.dim && !lastDim) out += "\x1b[2m"; 349 + if (styleCell.italic && !lastItalic) out += "\x1b[3m"; 350 + if (styleCell.underline && !lastUnderline) out += "\x1b[4m"; 351 + 352 + out += emitFg(styleCell, state); 353 + out += emitBg(styleCell, state); 354 + 355 + out += ch; 356 + 357 + lastRow = r; 358 + lastCol = c + (ch === "" ? 0 : charWidth(ch)); 359 + lastBold = styleCell.bold; 360 + lastDim = styleCell.dim; 361 + lastItalic = styleCell.italic; 362 + lastUnderline = styleCell.underline; 363 + }; 364 + 293 365 for (let r = 0; r < next.rows; r++) { 294 366 for (let c = 0; c < next.cols; c++) { 295 367 const nc = next.cells[r][c]; 296 368 297 - // Skip wide-char placeholder cells 369 + // Placeholder cells (char === "") are the right half of a wide 370 + // char at c-1 in `next`. Skip — the wide char's own emit at c-1 371 + // (either just performed this iteration, or elided as a cellsEqual 372 + // no-op) already covers column c on the terminal. writeAnsi's 373 + // placeholder invariant guarantees `next.cells[r][c].char === ""` 374 + // iff `next.cells[r][c-1]` is a wide char. 298 375 if (nc.char === "") continue; 299 376 300 377 const pc = prev.cells[r]?.[c]; 301 378 302 379 if (pc && cellsEqual(pc, nc)) continue; 303 380 304 - // Move cursor if not adjacent 305 - if (r !== lastRow || c !== lastCol) { 306 - out += `\x1b[${r + 1};${c + 1}H`; 307 - } 308 - 309 - // Apply style changes 310 - const needReset = 311 - (nc.bold !== lastBold && !nc.bold) || 312 - (nc.dim !== lastDim && !nc.dim) || 313 - (nc.italic !== lastItalic && !nc.italic) || 314 - (nc.underline !== lastUnderline && !nc.underline) || 315 - needsReset; 316 - 317 - if (needReset) { 318 - out += "\x1b[0m"; 319 - resetState(state); 320 - lastBold = false; 321 - lastDim = false; 322 - lastItalic = false; 323 - lastUnderline = false; 324 - needsReset = false; 325 - } 326 - 327 - if (nc.bold && !lastBold) out += "\x1b[1m"; 328 - if (nc.dim && !lastDim) out += "\x1b[2m"; 329 - if (nc.italic && !lastItalic) out += "\x1b[3m"; 330 - if (nc.underline && !lastUnderline) out += "\x1b[4m"; 331 - 332 - out += emitFg(nc, state); 333 - out += emitBg(nc, state); 334 - 335 - out += nc.char; 336 - 337 - lastRow = r; 338 - lastCol = c + 1; 339 - lastBold = nc.bold; 340 - lastDim = nc.dim; 341 - lastItalic = nc.italic; 342 - lastUnderline = nc.underline; 381 + emit(r, c, nc.char, nc); 343 382 } 344 383 } 345 384

+231

tests/buffer-wide-char-diff.test.ts

··· 1 + // Regression tests for the CellBuffer diff() renderer's handling of 2 + // width-2 (emoji / CJK) characters. Filed by tui-sup while building 3 + // agent-viz on top of @myobie/pty/tui: a `📬` on an interactive tree 4 + // would fossilize into `📬📬` when navigating shifted the glyph to a 5 + // different column. Two failure modes: 6 + // 7 + // 1. `lastCol = c + 1` in diff() (buffer.ts:338) desyncs cursor 8 + // tracking by one after every wide-char emit, since the terminal 9 + // cursor lands at c+2 for a wide glyph. Downstream: adjacency 10 + // optimization mispredicts, cursor moves may land off by one. 11 + // 2. The `nc.char === "" ⇒ skip` guard (buffer.ts:298) leaves the 12 + // right half of a prev-frame wide glyph on screen when the next 13 + // frame's placeholder-column cell happens to also be flagged as 14 + // a placeholder (via a genuine writeAnsi placeholder in `next`) 15 + // but the underlying terminal position still holds the fossil. 16 + // 17 + // The oracle: apply fullRender(prev) + diff(prev, next) to a fresh 18 + // CellBuffer via writeAnsi and check every cell matches `next`. If the 19 + // diff is correct, this round-trip is lossless. 20 + 21 + import { describe, it, expect } from "vitest"; 22 + import xterm from "@xterm/headless"; 23 + import * as xtermSerialize from "@xterm/addon-serialize"; 24 + import * as xtermUnicode11 from "@xterm/addon-unicode11"; 25 + import { CellBuffer, fullRender, diff } from "../src/tui/buffer.ts"; 26 + 27 + /** Build a fresh xterm-headless terminal with Unicode 11 wide-char widths 28 + * so emoji like `📬` (U+1F4EC) are treated as width-2 — matching what a 29 + * modern host terminal (kitty, iTerm2, Ghostty) does. Without this, the 30 + * default xterm.js width tables treat all astral-plane codepoints as 31 + * width-1 and the emoji collapses, hiding our CellBuffer bugs behind an 32 + * environment quirk. */ 33 + function makeTerminal(rows: number, cols: number): any { 34 + const t = new xterm.Terminal({ rows, cols, scrollback: 0, allowProposedApi: true }); 35 + const u11 = new xtermUnicode11.Unicode11Addon(); 36 + t.loadAddon(u11); 37 + t.unicode.activeVersion = "11"; 38 + return t; 39 + } 40 + 41 + /** Serialize an xterm-headless terminal's viewport to a plain string so 42 + * we can diff two terminals' rendered state. */ 43 + function terminalToString(term: any, cols: number, rows: number): string { 44 + const out: string[] = []; 45 + for (let r = 0; r < rows; r++) { 46 + const line = term.buffer.active.getLine(r); 47 + if (!line) { out.push(""); continue; } 48 + // translateToString(trimRight=false) gives us the raw viewport text 49 + // including trailing spaces — important because a fossil (a 50 + // wide-char right half surviving into the "space" region) would be 51 + // hidden by trimming. 52 + out.push(line.translateToString(false, 0, cols)); 53 + } 54 + return out.join("\n"); 55 + } 56 + 57 + /** Round-trip an intended `next` state through fullRender + diff against 58 + * `prev` and assert an xterm-headless terminal ends up EQUIVALENT to 59 + * one that got only `fullRender(next)`. This is the strong oracle: 60 + * xterm-headless emulates real terminal semantics, including "narrow 61 + * char overwriting a wide char's left half leaves the right half as a 62 + * visible fossil until explicitly cleared." A pure CellBuffer receiver 63 + * would miss that class of bug (its placeholder logic is symmetric 64 + * with our own — so it papers over the divergence). */ 65 + function assertDiffLossless(prev: CellBuffer, next: CellBuffer): void { 66 + const initial = fullRender(prev); 67 + const d = diff(prev, next); 68 + 69 + const applied = makeTerminal(prev.rows, prev.cols); 70 + applied.loadAddon(new xtermSerialize.SerializeAddon()); 71 + applied.write(initial); 72 + applied.write(d); 73 + 74 + const oracle = makeTerminal(prev.rows, prev.cols); 75 + oracle.loadAddon(new xtermSerialize.SerializeAddon()); 76 + oracle.write(fullRender(next)); 77 + 78 + return new Promise<void>((resolve, reject) => { 79 + // xterm.write is async. Both writes above enqueue behind a shared 80 + // parser microtask; give them a tick to drain, then compare. 81 + setTimeout(() => { 82 + const appliedText = terminalToString(applied, prev.cols, prev.rows); 83 + const oracleText = terminalToString(oracle, prev.cols, prev.rows); 84 + applied.dispose(); 85 + oracle.dispose(); 86 + if (appliedText !== oracleText) { 87 + reject(new Error( 88 + `Diff drift:\n` + 89 + ` applied (prev + diff):\n ${JSON.stringify(appliedText)}\n` + 90 + ` oracle (fullRender(next)):\n ${JSON.stringify(oracleText)}\n`, 91 + )); 92 + } else { 93 + resolve(); 94 + } 95 + }, 20); 96 + }) as any; 97 + } 98 + 99 + describe("CellBuffer diff() — wide-char (2-cell) glyph handling", () => { 100 + it("does not fossilize a wide char that shifts column between frames", async () => { 101 + // Prev frame: `📬` at column 2 (occupies 2,3). Rest is space. 102 + // Next frame: `📬` at column 5 (occupies 5,6). Column 2 must be 103 + // fully cleared — both the left half AND the fossil right half at 104 + // column 3. 105 + const prev = new CellBuffer(1, 10); 106 + prev.writeAnsi(" 📬 "); 107 + 108 + const next = new CellBuffer(1, 10); 109 + next.writeAnsi(" 📬 "); 110 + 111 + await assertDiffLossless(prev, next); 112 + }); 113 + 114 + it("does not fossilize when a wide char is replaced by a narrow char at the same start column", async () => { 115 + // Prev: `📬` at (0,2). Next: `A` at (0,2). Column 3 (formerly the 116 + // wide-char placeholder) must be cleared to a space, not left with 117 + // the fossil right half of `📬`. 118 + const prev = new CellBuffer(1, 10); 119 + prev.writeAnsi(" 📬 "); 120 + 121 + const next = new CellBuffer(1, 10); 122 + next.writeAnsi(" A "); 123 + 124 + await assertDiffLossless(prev, next); 125 + }); 126 + 127 + it("does not fossilize when a wide char shifts one column right", async () => { 128 + // Prev: `📬` at (0,2). Next: `📬` at (0,3). The 1-column shift means 129 + // prev's left half (col 2) needs clearing, and prev's right half 130 + // (col 3) is now the left half of next's `📬`. 131 + const prev = new CellBuffer(1, 10); 132 + prev.writeAnsi(" 📬 "); 133 + 134 + const next = new CellBuffer(1, 10); 135 + next.writeAnsi(" 📬 "); 136 + 137 + await assertDiffLossless(prev, next); 138 + }); 139 + 140 + it("does not fossilize two adjacent wide chars when both shift together", async () => { 141 + // Prev: `📬📭` at (0,2)-(0,5). Next: shifted to (0,4)-(0,7). All four 142 + // cells in the prev range need clearing except where they overlap 143 + // with the next range. 144 + const prev = new CellBuffer(1, 12); 145 + prev.writeAnsi(" 📬📭 "); 146 + 147 + const next = new CellBuffer(1, 12); 148 + next.writeAnsi(" 📬📭 "); 149 + 150 + await assertDiffLossless(prev, next); 151 + }); 152 + 153 + it("preserves alignment when a wide char is followed by digits (tui-sup's original bug)", async () => { 154 + // The exact shape from tui-sup's report: emoji + digits, where the 155 + // digits after must land in the correct column even after the emoji 156 + // shifts. Before the fix, prev's 📬 at col 2 and next's 📬 at col 4 157 + // followed by `99` would result in `📬📬99` visible in the terminal 158 + // instead of the intended ` 📬99`. 159 + const prev = new CellBuffer(1, 14); 160 + prev.writeAnsi(" 📬 1 "); 161 + 162 + const next = new CellBuffer(1, 14); 163 + next.writeAnsi(" 📬 99 "); 164 + 165 + await assertDiffLossless(prev, next); 166 + }); 167 + 168 + it("cursor tracking stays aligned across a run of wide chars", async () => { 169 + // Multiple wide chars in a row exercise the `lastCol = c + 1` vs. 170 + // `c + charWidth` code path. Without the fix, the adjacency check 171 + // fires an unnecessary cursor move after each wide char (not a 172 + // correctness bug per se, but the compounded off-by-one interacts 173 + // with the placeholder-skip in the fossil scenarios above). 174 + const prev = new CellBuffer(1, 12); 175 + prev.writeAnsi(" "); 176 + 177 + const next = new CellBuffer(1, 12); 178 + next.writeAnsi("📬📭📫✉XY"); 179 + 180 + await assertDiffLossless(prev, next); 181 + }); 182 + 183 + it("clears wide-char right halves when the surrounding row also mutates", async () => { 184 + // Realistic tui-nav shape: a row where a status column changes AND 185 + // an emoji shifts. Exercises the interaction between the two fix 186 + // sites (lastCol tracking + placeholder-clear on shift). 187 + const prev = new CellBuffer(1, 20); 188 + prev.writeAnsi(" • 📬 1 09m │ "); 189 + 190 + const next = new CellBuffer(1, 20); 191 + next.writeAnsi(" ▸ 📬 99 41m │ "); 192 + 193 + await assertDiffLossless(prev, next); 194 + }); 195 + 196 + it("handles a wide char moving into a position that was a narrow char", async () => { 197 + const prev = new CellBuffer(1, 10); 198 + prev.writeAnsi(" AB "); 199 + 200 + const next = new CellBuffer(1, 10); 201 + next.writeAnsi(" 📬 "); 202 + 203 + await assertDiffLossless(prev, next); 204 + }); 205 + 206 + it("handles a wide char being removed entirely (next has only narrows)", async () => { 207 + const prev = new CellBuffer(1, 10); 208 + prev.writeAnsi(" 📬 1 "); 209 + 210 + const next = new CellBuffer(1, 10); 211 + next.writeAnsi(" xx 1 "); 212 + 213 + await assertDiffLossless(prev, next); 214 + }); 215 + 216 + it("handles narrow → wide → narrow toggle at the same start column", async () => { 217 + // Round-trip: narrow to wide to narrow at the same column. Both 218 + // transitions must clear cleanly. 219 + const start = new CellBuffer(1, 10); 220 + start.writeAnsi(" AB "); 221 + 222 + const middle = new CellBuffer(1, 10); 223 + middle.writeAnsi(" 📬 "); 224 + 225 + const end = new CellBuffer(1, 10); 226 + end.writeAnsi(" CD "); 227 + 228 + await assertDiffLossless(start, middle); 229 + await assertDiffLossless(middle, end); 230 + }); 231 + });

Configure Feed

Configure Feed