fix(gc): fast-fail respawn cap on permanent sessions (#54) (#56) · compounding.tech/pty@a9850bc

+13

CHANGELOG.md

··· 2 2 3 3 ## Unreleased 4 4 5 + ### `pty gc` fast-fail respawn cap (fixes #54) 6 + 7 + - **New:** `pty gc` STEP-2 now detects a crash-looping permanent session and stops respawning it before it churns forever. A respawn whose leaf exits within `strategy.fast-fail-window` seconds (default 60) of the previous respawn counts as a fast fail; after `strategy.fast-fail-limit` consecutive fast fails (default 3) the session gets `strategy.status=flapping` written to its tags and a `session_flapping` event, and subsequent gc ticks silently skip it. This is the crash-loop-with-live-cwd case that the earlier `cwd-gone`/`idle` reap (#47, #50) didn't cover. 8 + - **New event `session_flapping`.** Payload `{ session, type: "session_flapping", ts, counter, limit, window }`. `counter` is the fast-fail streak at the moment of flip (>= `limit`); `window` is the resolved window in seconds. Documented in `docs/disk-layout.md`. 9 + - **New bookkeeping tags on permanent sessions.** Written by `pty gc` on every respawn: `strategy.last-respawn-at` (ISO timestamp), `strategy.consecutive-fast-fails` (running counter), `strategy.command-hash` (16-char SHA-256 prefix of the respawn command line — used to auto-reset the counter and clear a stale flapping mark when the operator edits the stored command). At the flip: `strategy.status=flapping` is added. 10 + - **Manual reset:** `pty tag <name> --rm strategy.status` (or `pty tag ... --rm strategy.consecutive-fast-fails --rm strategy.last-respawn-at` for a full clean-slate). The next gc tick retries. 11 + - **Auto-reset on command change:** the classifier compares the new command's fingerprint against `strategy.command-hash`. If they differ, the counter resets to zero and `strategy.status=flapping` clears — the operator has already reshaped the problem, no manual step needed. 12 + - **Per-session overrides** — `strategy.fast-fail-window=<sec>` and `strategy.fast-fail-limit=<int>` tags override the defaults for one session; higher precedence than the CLI globals. 13 + - **New CLI globals** — `pty gc --fast-fail-window=<sec>` and `pty gc --fast-fail-limit=<int>` mirror the per-session tags for anyone who wants to override the defaults for a whole gc run. 14 + - **`GcResult` gains two fields.** `flapped: { name, counter, limit, window }[]` — sessions the classifier flipped this tick. `flappingSkipped: string[]` — sessions the classifier silently skipped because they were already flagged. Both are surfaced in `cmdGc` output. Additive; existing consumers keep working. 15 + - **CLI output** — new lines: `Flapping: <name> (<N> fast-fails in <W>s, limit <L>)` on the flip tick and `Skipped (flapping): <name> — remove strategy.status tag to retry` on subsequent ticks. Summary bar adds `N flapping` / `N skipped-flapping` when non-empty. `--dry-run` mirrors with `Would flap: ...`. 16 + - Tests in `tests/gc-flapping.test.ts` (10 new) cover: dry-run previews, at-limit persistence + event emission, already-flapping silent skip, slow-fail counter reset, command-hash auto-reset (clears flapping mark), per-session and CLI-global window/limit overrides, and the first-respawn (no prior last-respawn-at) case. 17 + 5 18 ### Per-namespace isolation — `PTY_ROOT` + `pty --root <path>` 6 19 7 20 - **New canonical env var `PTY_ROOT`** for the state registry directory (default `~/.local/state/pty`). The pre-existing `PTY_SESSION_DIR` still works and continues to point at the same registry; when only the legacy name is set, `pty` emits a one-time deprecation notice to stderr per process. Precedence: `PTY_ROOT > PTY_SESSION_DIR > default`.

+16

README.md

··· 266 266 267 267 Restart is stateless — every `pty gc` invocation re-derives intent from on-disk metadata. There's no in-memory restart counter, no `[failed]` state, no persisted bookkeeping. If a session's binary isn't reachable (volume not mounted, broken symlink), `pty gc` reports `Respawn failed:` and the next tick tries again. 268 268 269 + **Fast-fail cap** — a permanent session whose leaf exits within `strategy.fast-fail-window` seconds of its previous `pty gc` respawn counts as a fast fail. After `strategy.fast-fail-limit` consecutive fast fails, `pty gc` writes `strategy.status=flapping` on the session, emits a `session_flapping` event, and stops respawning it. Subsequent gc ticks print `Skipped (flapping): <name>` and take no action. Defaults: 60 s window, 3 consecutive fast fails. 270 + 271 + Reset a flagged session with one of: 272 + 273 + - `pty tag <name> --rm strategy.status` — clear the mark, `pty gc` retries on the next tick. 274 + - Edit the session's `pty.toml` command — the classifier notices the SHA-256 fingerprint change and auto-resets the counter and mark. 275 + 276 + Per-session overrides tune the cap without editing gc's globals: 277 + 278 + ```sh 279 + pty tag myserver strategy.fast-fail-window=120 # allow 2min of runtime before "fast" 280 + pty tag myserver strategy.fast-fail-limit=5 # tolerate 5 fast fails before flapping 281 + ``` 282 + 283 + CLI globals mirror the per-session tags (`--fast-fail-window=N`, `--fast-fail-limit=N`); the per-session tag wins when both are set. 284 + 269 285 ### Parent-child sessions 270 286 271 287 Tag a session with `parent=<name>` and `pty gc` will SIGTERM it (and clean up its metadata) when the referenced parent's daemon is no longer alive — useful for sidecar workers that shouldn't outlive their primary:

+1

docs/disk-layout.md

+51 -29

src/cli.ts

··· 129 129 pty tag-multi <selector> [ops...] Read/write tags across multiple sessions (--all / --filter-tag k=v / <name>...) 130 130 pty gc --print-launchd-plist [--interval=N] Print a launchd plist that runs 'pty gc' every N seconds (default 30) 131 131 pty gc [--dry-run] [--idle-days N] Reconciliation pass; --idle-days N reaps permanent sessions with no attach in N days 132 + pty gc --fast-fail-window=N Fast-fail window (seconds) for the respawn cap (default 60; per-session tag wins) 133 + pty gc --fast-fail-limit=N Consecutive fast fails before a permanent session is flagged flapping (default 3) 132 134 pty wrap <command> Auto-wrap a command in pty sessions 133 135 pty unwrap <command> Remove a wrap 134 136 pty wrap --list List wrapped commands ··· 758 760 const printPlist = gcArgs.includes("--print-launchd-plist"); 759 761 let interval = 30; 760 762 let idleDays: number | undefined; 763 + let fastFailWindowSec: number | undefined; 764 + let fastFailLimit: number | undefined; 765 + const parsePositive = (flag: string, raw: string): number => { 766 + const v = parseInt(raw, 10); 767 + if (!Number.isFinite(v) || v <= 0) { 768 + console.error(`pty gc: ${flag} expects a positive integer (got "${raw}")`); 769 + process.exit(1); 770 + } 771 + return v; 772 + }; 761 773 for (let i = 0; i < gcArgs.length; i++) { 762 774 const a = gcArgs[i]; 763 775 if (a === "--interval" && i + 1 < gcArgs.length) { 764 - const v = parseInt(gcArgs[i + 1], 10); 765 - if (!Number.isFinite(v) || v <= 0) { 766 - console.error(`pty gc: --interval expects a positive integer (got "${gcArgs[i + 1]}")`); 767 - process.exit(1); 768 - } 769 - interval = v; 770 - i++; 776 + interval = parsePositive("--interval", gcArgs[++i]); 771 777 } else if (a.startsWith("--interval=")) { 772 - const v = parseInt(a.slice("--interval=".length), 10); 773 - if (!Number.isFinite(v) || v <= 0) { 774 - console.error(`pty gc: --interval expects a positive integer (got "${a.slice("--interval=".length)}")`); 775 - process.exit(1); 776 - } 777 - interval = v; 778 + interval = parsePositive("--interval", a.slice("--interval=".length)); 778 779 } else if (a === "--idle-days" && i + 1 < gcArgs.length) { 779 - const v = parseInt(gcArgs[i + 1], 10); 780 - if (!Number.isFinite(v) || v <= 0) { 781 - console.error(`pty gc: --idle-days expects a positive integer (got "${gcArgs[i + 1]}")`); 782 - process.exit(1); 783 - } 784 - idleDays = v; 785 - i++; 780 + idleDays = parsePositive("--idle-days", gcArgs[++i]); 786 781 } else if (a.startsWith("--idle-days=")) { 787 - const v = parseInt(a.slice("--idle-days=".length), 10); 788 - if (!Number.isFinite(v) || v <= 0) { 789 - console.error(`pty gc: --idle-days expects a positive integer (got "${a.slice("--idle-days=".length)}")`); 790 - process.exit(1); 791 - } 792 - idleDays = v; 782 + idleDays = parsePositive("--idle-days", a.slice("--idle-days=".length)); 783 + } else if (a === "--fast-fail-window" && i + 1 < gcArgs.length) { 784 + fastFailWindowSec = parsePositive("--fast-fail-window", gcArgs[++i]); 785 + } else if (a.startsWith("--fast-fail-window=")) { 786 + fastFailWindowSec = parsePositive("--fast-fail-window", a.slice("--fast-fail-window=".length)); 787 + } else if (a === "--fast-fail-limit" && i + 1 < gcArgs.length) { 788 + fastFailLimit = parsePositive("--fast-fail-limit", gcArgs[++i]); 789 + } else if (a.startsWith("--fast-fail-limit=")) { 790 + fastFailLimit = parsePositive("--fast-fail-limit", a.slice("--fast-fail-limit=".length)); 793 791 } 794 792 } 795 793 if (printPlist) { 796 794 printLaunchdPlist(interval); 797 795 break; 798 796 } 799 - await cmdGc(dryRun, idleDays); 797 + await cmdGc(dryRun, idleDays, fastFailWindowSec, fastFailLimit); 800 798 break; 801 799 } 802 800 ··· 1930 1928 console.log(`Session "${name}" removed.`); 1931 1929 } 1932 1930 1933 - async function cmdGc(dryRun: boolean, idleDays?: number): Promise<void> { 1934 - const result = await gc({ dryRun, idleDays }); 1931 + async function cmdGc( 1932 + dryRun: boolean, 1933 + idleDays?: number, 1934 + fastFailWindowSec?: number, 1935 + fastFailLimit?: number, 1936 + ): Promise<void> { 1937 + const result = await gc({ dryRun, idleDays, fastFailWindowSec, fastFailLimit }); 1935 1938 const prunedTags = await pruneOrphanLayoutTags({ dryRun }); 1936 1939 1937 1940 const killedVerb = dryRun ? "Would kill orphan child" : "Killed orphan child"; 1938 1941 const abandonVerb = dryRun ? "Would abandon" : "Abandoned"; 1939 1942 const respawnVerb = dryRun ? "Would respawn" : "Respawned"; 1943 + const flapVerb = dryRun ? "Would flap" : "Flapping"; 1940 1944 const removeVerb = dryRun ? "Would remove" : "Removed"; 1941 1945 const prunedVerb = dryRun ? "Would prune" : "Pruned"; 1942 1946 ··· 1956 1960 for (const f of result.respawnFailed) { 1957 1961 console.log(`Respawn failed: ${f.name} — ${f.error}`); 1958 1962 } 1963 + for (const fl of result.flapped) { 1964 + console.log( 1965 + `${flapVerb}: ${fl.name} (${fl.counter} fast-fails in ${fl.window}s, limit ${fl.limit})`, 1966 + ); 1967 + } 1968 + for (const name of result.flappingSkipped) { 1969 + console.log( 1970 + `Skipped (flapping): ${name} — remove strategy.status tag to retry`, 1971 + ); 1972 + } 1959 1973 for (const name of result.removed) { 1960 1974 console.log(`${removeVerb}: ${name}`); 1961 1975 } ··· 1971 1985 result.abandoned.length + 1972 1986 result.respawned.length + 1973 1987 result.respawnFailed.length + 1988 + result.flapped.length + 1989 + result.flappingSkipped.length + 1974 1990 result.removed.length + 1975 1991 totalTags; 1976 1992 ··· 1991 2007 } 1992 2008 if (result.respawnFailed.length > 0) { 1993 2009 parts.push(`${result.respawnFailed.length} respawn failure${result.respawnFailed.length === 1 ? "" : "s"}`); 2010 + } 2011 + if (result.flapped.length > 0) { 2012 + parts.push(`${result.flapped.length} flapping`); 2013 + } 2014 + if (result.flappingSkipped.length > 0) { 2015 + parts.push(`${result.flappingSkipped.length} skipped-flapping`); 1994 2016 } 1995 2017 if (result.removed.length > 0) { 1996 2018 parts.push(`${result.removed.length} stale session${result.removed.length === 1 ? "" : "s"}`);

+19

src/events.ts

··· 16 16 SESSION_EXEC: "session_exec", 17 17 SESSION_RESPAWN: "session_respawn", 18 18 SESSION_ABANDONED: "session_abandoned", 19 + SESSION_FLAPPING: "session_flapping", 19 20 } as const; 20 21 21 22 export type EventType = (typeof EventType)[keyof typeof EventType]; ··· 107 108 idleDays?: number; 108 109 } 109 110 111 + /** Emitted by `pty gc` when a `strategy=permanent` session has fast-failed 112 + * its respawn `limit` consecutive times within `window` seconds. Marks the 113 + * session `strategy.status=flapping` and stops respawning until the operator 114 + * removes the tag (or the stored command changes — auto-reset on hash 115 + * divergence). Emitted at the moment the threshold is crossed; subsequent 116 + * gc ticks silently skip the flapping session. */ 117 + export interface SessionFlappingEvent extends EventBase { 118 + type: "session_flapping"; 119 + /** Fast-fails counted, at or above `limit` when this event fired. */ 120 + counter: number; 121 + /** Threshold that was crossed. */ 122 + limit: number; 123 + /** Fast-fail window in seconds. A respawn that exits within this many 124 + * seconds of its `strategy.last-respawn-at` stamp counts as fast. */ 125 + window: number; 126 + } 127 + 110 128 /** User-published event. `type` must begin with `user.` — the CLI 111 129 * (`pty emit`) rejects anything else, and the client-API `emitEvent` 112 130 * helper throws on bad types. Payload is free-form JSON. */ ··· 159 177 | SessionExecEvent 160 178 | SessionRespawnEvent 161 179 | SessionAbandonedEvent 180 + | SessionFlappingEvent 162 181 | UserEvent 163 182 | StateSetEvent 164 183 | StateDeleteEvent

+278 -4

src/sessions.ts

··· 3 3 import * as path from "node:path"; 4 4 import * as os from "node:os"; 5 5 import * as net from "node:net"; 6 + import { createHash } from "node:crypto"; 6 7 // Circular import: events.ts imports getEventsPath/ensureSessionDir from 7 8 // this file. Cycle is safe — `appendEventSync` is only called at runtime 8 9 // from inside functions, never at module-init time. ··· 510 511 * binary is on an unmounted volume). Cron interval is the rate limit; 511 512 * next tick tries again. */ 512 513 respawnFailed: { name: string; error: string }[]; 514 + /** Permanent sessions the fast-fail cap flipped to `flapping` on this 515 + * tick. Each entry records the counter at the moment of flip plus the 516 + * effective `limit`/`window` in play. Sessions already flagged before 517 + * this tick are silently skipped from the respawn loop and do NOT 518 + * appear here — this bucket is transitions only. */ 519 + flapped: { name: string; counter: number; limit: number; window: number }[]; 520 + /** Permanent sessions skipped this tick because they are already 521 + * `strategy.status=flapping`. Distinct from `flapped` (transitions), 522 + * `respawnFailed` (attempted + failed), and `respawned` (attempted + 523 + * succeeded). Consumers can render "N flapping" without having to 524 + * read tags themselves. */ 525 + flappingSkipped: string[]; 526 + } 527 + 528 + /** Default fast-fail respawn cap window (seconds). A permanent session 529 + * that exits within `DEFAULT_FAST_FAIL_WINDOW_SEC` of its previous gc 530 + * respawn counts as a fast fail. Overridden by `opts.fastFailWindowSec` 531 + * or the per-session `strategy.fast-fail-window` tag. */ 532 + export const DEFAULT_FAST_FAIL_WINDOW_SEC = 60; 533 + 534 + /** Default fast-fail limit. `DEFAULT_FAST_FAIL_LIMIT` consecutive fast 535 + * fails flip the session to `strategy.status=flapping` and stop future 536 + * respawns until the operator intervenes (or the stored command changes, 537 + * which auto-resets). Overridden by `opts.fastFailLimit` or the 538 + * per-session `strategy.fast-fail-limit` tag. */ 539 + export const DEFAULT_FAST_FAIL_LIMIT = 3; 540 + 541 + /** SHA-256 of a session's respawn command line, used to auto-reset the 542 + * fast-fail counter when the operator edits the pty.toml (or otherwise 543 + * changes the stored command). Kept short — the tag surface is user- 544 + * facing, not a cryptographic identifier. */ 545 + function commandFingerprint(command: string, args: string[]): string { 546 + const h = createHash("sha256"); 547 + h.update(command); 548 + h.update("\0"); 549 + h.update(args.join("\0")); 550 + return h.digest("hex").slice(0, 16); 513 551 } 514 552 515 553 /** Reconciliation pass driven by `pty gc`. Stateless: every invocation ··· 530 568 * exited/vanished is respawned via `spawnDaemon` (lazy-imported to 531 569 * avoid the `sessions ↔ spawn` cycle). Sessions with `ptyfile` + 532 570 * `ptyfile.session` tags re-read the toml to pick up any edits. 571 + * A fast-fail cap prevents a crash-looping leaf from being 572 + * respawned forever: `strategy.fast-fail-limit` consecutive 573 + * respawns whose leaf exited within `strategy.fast-fail-window` 574 + * seconds flip the session to `strategy.status=flapping` and 575 + * skip it on subsequent ticks. Auto-reset when the stored command 576 + * changes; manual reset via `pty tag <name> --rm strategy.status`. 533 577 * 3. Existing sweep: the historic behavior — exited/vanished sessions 534 578 * that aren't permanent get `cleanupAll`'d. */ 535 - export async function gc(opts: { dryRun?: boolean; idleDays?: number } = {}): Promise<GcResult> { 579 + export async function gc( 580 + opts: { 581 + dryRun?: boolean; 582 + idleDays?: number; 583 + fastFailWindowSec?: number; 584 + fastFailLimit?: number; 585 + } = {}, 586 + ): Promise<GcResult> { 536 587 const dryRun = !!opts.dryRun; 537 588 const globalIdleDays = opts.idleDays; 589 + const globalFastFailWindow = opts.fastFailWindowSec; 590 + const globalFastFailLimit = opts.fastFailLimit; 538 591 // First call to `listSessions` is intentionally throwaway — it has a 539 592 // side effect (`cleanupSocket`) on sessions whose daemon SIGKILL'd 540 593 // without writing an exit record, and those sessions are then *missing* ··· 630 683 : await listSessions(); 631 684 const respawned: GcResult["respawned"] = []; 632 685 const respawnFailed: GcResult["respawnFailed"] = []; 686 + const flapped: GcResult["flapped"] = []; 687 + const flappingSkipped: GcResult["flappingSkipped"] = []; 633 688 for (const s of afterStep15) { 634 689 if (s.metadata?.tags?.strategy !== "permanent") continue; 635 690 if (!isGone(s.status)) continue; 636 691 const ptyfileReread = !!s.metadata?.tags?.ptyfile; 692 + 693 + // Fast-fail classifier: was the previous respawn a fast crash? What's 694 + // the running counter? Should we flip to flapping? Runs before any 695 + // spawn so a session at the limit boundary flaps this tick instead 696 + // of respawning one more time. 697 + const decision = classifyFlapping( 698 + s, 699 + new Date(), 700 + globalFastFailWindow, 701 + globalFastFailLimit, 702 + ); 703 + 704 + if (decision.action === "skip-flapping") { 705 + flappingSkipped.push(s.name); 706 + continue; 707 + } 708 + 637 709 if (dryRun) { 710 + if (decision.action === "flap-now") { 711 + flapped.push({ 712 + name: s.name, 713 + counter: decision.counter, 714 + limit: decision.effectiveLimit, 715 + window: decision.effectiveWindow, 716 + }); 717 + continue; 718 + } 638 719 respawned.push({ name: s.name, ptyfileReread }); 639 720 continue; 640 721 } 722 + 723 + if (decision.action === "flap-now") { 724 + // Persist the flapping mark to on-disk metadata so subsequent 725 + // ticks see it. We update the metadata file directly instead of 726 + // going through updateTags — the session's daemon is gone, there's 727 + // no live connection to notify, and cleanupAll ordering constraints 728 + // in respawnPermanent don't apply here (we're NOT respawning). 729 + try { 730 + const meta = readMetadata(s.name); 731 + if (meta) { 732 + const merged: Record<string, string> = { 733 + ...(meta.tags ?? {}), 734 + ...decision.newBookkeeping, 735 + }; 736 + writeMetadata(s.name, { ...meta, tags: merged }); 737 + } 738 + } catch { 739 + // Best-effort — if we can't persist the flag now, the next tick 740 + // will recompute the same decision and try again. 741 + } 742 + try { 743 + appendEventSync(s.name, { 744 + session: s.name, 745 + type: "session_flapping", 746 + ts: new Date().toISOString(), 747 + counter: decision.counter, 748 + limit: decision.effectiveLimit, 749 + window: decision.effectiveWindow, 750 + }); 751 + } catch {} 752 + flapped.push({ 753 + name: s.name, 754 + counter: decision.counter, 755 + limit: decision.effectiveLimit, 756 + window: decision.effectiveWindow, 757 + }); 758 + continue; 759 + } 760 + 641 761 try { 642 - await respawnPermanent(s.name, s.metadata!); 762 + await respawnPermanent(s.name, s.metadata!, decision.newBookkeeping); 643 763 respawned.push({ name: s.name, ptyfileReread }); 644 764 } catch (err: any) { 645 765 respawnFailed.push({ name: s.name, error: err?.message ?? String(err) }); ··· 660 780 removed.push(s.name); 661 781 } 662 782 663 - return { removed, killedOrphanChildren, abandoned, respawned, respawnFailed }; 783 + return { 784 + removed, 785 + killedOrphanChildren, 786 + abandoned, 787 + respawned, 788 + respawnFailed, 789 + flapped, 790 + flappingSkipped, 791 + }; 664 792 } 665 793 666 794 /** Decide whether a permanent session is abandoned. Order: ··· 708 836 return { reason: "idle", idleDays: ageDays }; 709 837 } 710 838 839 + /** Decide whether a `strategy=permanent` session that's exited/vanished 840 + * should be respawned, marked flapping, or silently skipped because 841 + * it's already flapping. Reads three bookkeeping tags from the session: 842 + * - `strategy.last-respawn-at` (ISO ts): when gc last respawned it 843 + * - `strategy.consecutive-fast-fails` (int): running fast-fail counter 844 + * - `strategy.command-hash` (16-char hex): command fingerprint at last 845 + * respawn. If the current fingerprint differs, the operator edited 846 + * the pty.toml (or otherwise changed the command); reset the 847 + * counter and clear any stale `strategy.status=flapping`. 848 + * 849 + * Effective window/limit resolution: 850 + * per-session tag (strategy.fast-fail-window / -limit) 851 + * → global opt (CLI --fast-fail-window / --fast-fail-limit) 852 + * → DEFAULT_FAST_FAIL_WINDOW_SEC / DEFAULT_FAST_FAIL_LIMIT. 853 + * 854 + * Returned `newBookkeeping` MUST be merged onto the session's tags map 855 + * before/instead of respawn. The `flap-now` action never respawns; the 856 + * `respawn` action does; the `skip-flapping` action skips entirely. */ 857 + interface FlappingDecision { 858 + action: "respawn" | "flap-now" | "skip-flapping"; 859 + effectiveWindow: number; 860 + effectiveLimit: number; 861 + /** Fast-fail counter after this tick's classification. Only meaningful 862 + * for `respawn` (stamped on the session) and `flap-now` (the counter 863 + * that crossed the threshold, surfaced in the event payload). */ 864 + counter: number; 865 + /** Tag deltas to persist. Empty for `skip-flapping`. For `respawn`, 866 + * carries the fresh timestamp, counter, and command hash. For 867 + * `flap-now`, adds `strategy.status=flapping` on top. */ 868 + newBookkeeping: Record<string, string>; 869 + } 870 + 871 + function classifyFlapping( 872 + s: SessionInfo, 873 + now: Date, 874 + globalWindowSec: number | undefined, 875 + globalLimit: number | undefined, 876 + ): FlappingDecision { 877 + const tags = s.metadata?.tags ?? {}; 878 + 879 + const tagWindow = parseInt(tags["strategy.fast-fail-window"] ?? "", 10); 880 + const effectiveWindow = Number.isFinite(tagWindow) && tagWindow > 0 881 + ? tagWindow 882 + : (globalWindowSec !== undefined && globalWindowSec > 0 883 + ? globalWindowSec 884 + : DEFAULT_FAST_FAIL_WINDOW_SEC); 885 + 886 + const tagLimit = parseInt(tags["strategy.fast-fail-limit"] ?? "", 10); 887 + const effectiveLimit = Number.isFinite(tagLimit) && tagLimit > 0 888 + ? tagLimit 889 + : (globalLimit !== undefined && globalLimit > 0 890 + ? globalLimit 891 + : DEFAULT_FAST_FAIL_LIMIT); 892 + 893 + const command = s.metadata?.command ?? ""; 894 + const args = s.metadata?.args ?? []; 895 + const currentHash = commandFingerprint(command, args); 896 + const storedHash = tags["strategy.command-hash"]; 897 + const commandChanged = storedHash !== undefined && storedHash !== currentHash; 898 + 899 + // Command change wins over an existing flapping mark: the operator has 900 + // edited the pty.toml (or manually mutated the command), so give it a 901 + // fresh chance. `strategy.status` clears; counter resets to 0. 902 + if (tags["strategy.status"] === "flapping" && !commandChanged) { 903 + return { 904 + action: "skip-flapping", 905 + effectiveWindow, 906 + effectiveLimit, 907 + counter: parseInt(tags["strategy.consecutive-fast-fails"] ?? "0", 10) || 0, 908 + newBookkeeping: {}, 909 + }; 910 + } 911 + 912 + // Was the previous respawn a fast fail? Compare the exit timestamp 913 + // against the last-respawn stamp; anything under `window` seconds is 914 + // fast. If no prior stamp exists (never respawned by gc) or the exit 915 + // is missing (vanished session), treat as slow — the counter resets. 916 + const lastRespawnAt = tags["strategy.last-respawn-at"]; 917 + const exitedAt = s.metadata?.exitedAt; 918 + let liveMs: number | null = null; 919 + if (lastRespawnAt && exitedAt) { 920 + const lr = Date.parse(lastRespawnAt); 921 + const ex = Date.parse(exitedAt); 922 + if (Number.isFinite(lr) && Number.isFinite(ex)) liveMs = ex - lr; 923 + } 924 + const wasFastFail = liveMs !== null && liveMs >= 0 && liveMs < effectiveWindow * 1000; 925 + 926 + const prevCounter = parseInt(tags["strategy.consecutive-fast-fails"] ?? "0", 10) || 0; 927 + const nextCounter = commandChanged ? 0 : (wasFastFail ? prevCounter + 1 : 0); 928 + 929 + if (nextCounter >= effectiveLimit) { 930 + // Threshold crossed. Mark flapping, don't respawn. The counter goes 931 + // into the tags at its final value so subsequent listers can see how 932 + // deep the streak went. 933 + const bookkeeping: Record<string, string> = { 934 + "strategy.status": "flapping", 935 + "strategy.consecutive-fast-fails": String(nextCounter), 936 + "strategy.command-hash": currentHash, 937 + }; 938 + if (lastRespawnAt) bookkeeping["strategy.last-respawn-at"] = lastRespawnAt; 939 + return { 940 + action: "flap-now", 941 + effectiveWindow, 942 + effectiveLimit, 943 + counter: nextCounter, 944 + newBookkeeping: bookkeeping, 945 + }; 946 + } 947 + 948 + // Respawn. Stamp fresh bookkeeping. If we're clearing a stale flap 949 + // mark from a command change, drop `strategy.status` explicitly by 950 + // storing an empty string — updateTags treats that as a remove. 951 + const bookkeeping: Record<string, string> = { 952 + "strategy.last-respawn-at": now.toISOString(), 953 + "strategy.consecutive-fast-fails": String(nextCounter), 954 + "strategy.command-hash": currentHash, 955 + }; 956 + return { 957 + action: "respawn", 958 + effectiveWindow, 959 + effectiveLimit, 960 + counter: nextCounter, 961 + newBookkeeping: bookkeeping, 962 + }; 963 + } 964 + 711 965 /** Restart a `strategy=permanent` session whose daemon is gone. If the 712 966 * session was toml-managed (`ptyfile` + `ptyfile.session` tags), re-read 713 967 * the pty.toml so the new daemon picks up command/env edits since the 714 968 * last spawn. On any read error fall back to the stored metadata 715 969 * verbatim (last-known-good) so a temporarily-missing toml doesn't 716 970 * prevent restart. 971 + * 972 + * `bookkeepingOverlay` (optional) carries pty-internal tags that must 973 + * survive the pty.toml re-read: gc backoff state (`strategy.last-*`, 974 + * `strategy.command-hash`, `strategy.consecutive-fast-fails`). Passed 975 + * by `gc()` STEP-2; ignored by other callers. 717 976 * 718 977 * Lazy-imports `spawn.ts` so the `sessions.ts ↔ spawn.ts` cycle doesn't 719 978 * bite at module-init time. After spawn, appends a `session_respawn` 720 979 * event to the session's event log so consumers see the restart. */ 721 - async function respawnPermanent(name: string, metadata: SessionMetadata): Promise<void> { 980 + async function respawnPermanent( 981 + name: string, 982 + metadata: SessionMetadata, 983 + bookkeepingOverlay: Record<string, string> = {}, 984 + ): Promise<void> { 722 985 let command = metadata.command; 723 986 let args = metadata.args; 724 987 let displayCommand = metadata.displayCommand; ··· 750 1013 // error). Fall back to stored metadata — better to respawn with 751 1014 // last-known-good than to give up. 752 1015 } 1016 + } 1017 + 1018 + // Merge gc's backoff bookkeeping last so it survives the pty.toml 1019 + // overlay above. Callers pass an empty overlay when they aren't gc. 1020 + // If a command change is clearing a stale flap mark, the caller 1021 + // omits `strategy.status` from the overlay; we also clear any 1022 + // existing flag on the merged map so a rebuilt tags dict doesn't 1023 + // silently carry it forward from the previous metadata. 1024 + tags = { ...(tags ?? {}), ...bookkeepingOverlay }; 1025 + if (bookkeepingOverlay["strategy.status"] === undefined) { 1026 + delete tags["strategy.status"]; 753 1027 } 754 1028 755 1029 // Wipe stale socket/pid/events before respawn so spawnDaemon doesn't

+320

tests/gc-flapping.test.ts

··· 1 + // #54: fast-fail respawn cap. A crash-looping permanent session gets 2 + // flagged flapping and stopped after `strategy.fast-fail-limit` 3 + // consecutive fast failures within `strategy.fast-fail-window` seconds. 4 + // Auto-reset on command change; manual reset via `pty tag --rm`. 5 + 6 + import { describe, it, expect, afterEach, afterAll } from "vitest"; 7 + import * as fs from "node:fs"; 8 + import * as os from "node:os"; 9 + import * as path from "node:path"; 10 + import { fileURLToPath } from "node:url"; 11 + import { spawnSync } from "node:child_process"; 12 + import { createHash } from "node:crypto"; 13 + 14 + const __dirname = path.dirname(fileURLToPath(import.meta.url)); 15 + const nodeBin = process.execPath; 16 + const cliPath = path.join(__dirname, "..", "dist", "cli.js"); 17 + 18 + const testRoot = fs.mkdtempSync(path.join(os.tmpdir(), "pty-flap-")); 19 + afterAll(() => { 20 + fs.rmSync(testRoot, { recursive: true, force: true, maxRetries: 3, retryDelay: 100 }); 21 + }); 22 + 23 + let sessionDirs: string[] = []; 24 + 25 + function makeSessionDir(): string { 26 + const dir = fs.mkdtempSync(path.join(testRoot, "d-")); 27 + sessionDirs.push(dir); 28 + return dir; 29 + } 30 + 31 + let nameCounter = 0; 32 + function uniqueName(): string { 33 + return `fl${++nameCounter}${Math.random().toString(36).slice(2, 5)}`; 34 + } 35 + 36 + function runCli(sessionDir: string, ...args: string[]) { 37 + return spawnSync(nodeBin, [cliPath, ...args], { 38 + env: { ...process.env, PTY_SESSION_DIR: sessionDir }, 39 + encoding: "utf-8", 40 + timeout: 15000, 41 + }); 42 + } 43 + 44 + function readMeta(sessionDir: string, name: string): any { 45 + return JSON.parse(fs.readFileSync(path.join(sessionDir, `${name}.json`), "utf-8")); 46 + } 47 + 48 + function readEvents(sessionDir: string, name: string): any[] { 49 + const filePath = path.join(sessionDir, `${name}.events.jsonl`); 50 + try { 51 + return fs.readFileSync(filePath, "utf-8") 52 + .trimEnd().split("\n").filter((l) => l.length > 0).map((l) => JSON.parse(l)); 53 + } catch { 54 + return []; 55 + } 56 + } 57 + 58 + /** Write a synthetic exited-permanent metadata file. `respawnAt` seeds 59 + * `strategy.last-respawn-at`, `exitedAt` seeds `metadata.exitedAt` — 60 + * their delta drives the fast-fail classifier. */ 61 + function writeExitedPermanent( 62 + sessionDir: string, 63 + name: string, 64 + opts: { 65 + command?: string; 66 + args?: string[]; 67 + tags?: Record<string, string>; 68 + lastRespawnAt?: string; 69 + exitedAt?: string; 70 + counter?: number; 71 + commandHash?: string; 72 + status?: string; 73 + }, 74 + ): void { 75 + const command = opts.command ?? "sh"; 76 + const args = opts.args ?? ["-c", "exit 1"]; 77 + const tags: Record<string, string> = { 78 + strategy: "permanent", 79 + ...(opts.tags ?? {}), 80 + }; 81 + if (opts.lastRespawnAt !== undefined) tags["strategy.last-respawn-at"] = opts.lastRespawnAt; 82 + if (opts.counter !== undefined) tags["strategy.consecutive-fast-fails"] = String(opts.counter); 83 + if (opts.commandHash !== undefined) tags["strategy.command-hash"] = opts.commandHash; 84 + if (opts.status !== undefined) tags["strategy.status"] = opts.status; 85 + 86 + fs.writeFileSync(path.join(sessionDir, `${name}.json`), JSON.stringify({ 87 + command, args, displayCommand: command, 88 + cwd: os.tmpdir(), 89 + createdAt: new Date(Date.now() - 10 * 60_000).toISOString(), 90 + exitedAt: opts.exitedAt ?? new Date().toISOString(), 91 + exitCode: 1, 92 + tags, 93 + })); 94 + // Also write a stub events file so appendEventSync doesn't create 95 + // orphaned JSONL. Some listSessions paths key off it. 96 + const evPath = path.join(sessionDir, `${name}.events.jsonl`); 97 + if (!fs.existsSync(evPath)) fs.writeFileSync(evPath, ""); 98 + } 99 + 100 + function commandHash(command: string, args: string[]): string { 101 + const h = createHash("sha256"); 102 + h.update(command); 103 + h.update("\0"); 104 + h.update(args.join("\0")); 105 + return h.digest("hex").slice(0, 16); 106 + } 107 + 108 + afterEach(() => { 109 + for (const dir of sessionDirs) { 110 + try { 111 + for (const e of fs.readdirSync(dir)) { try { fs.unlinkSync(path.join(dir, e)); } catch {} } 112 + } catch {} 113 + } 114 + sessionDirs = []; 115 + }); 116 + 117 + describe("gc fast-fail respawn cap", () => { 118 + it("dry-run: below-limit fast fails preview as respawn (no state mutation)", () => { 119 + const dir = makeSessionDir(); 120 + const name = uniqueName(); 121 + // Session was respawned 5s ago, exited 1s later → fast fail (1s < 60s). 122 + // Prior counter 1 → this tick would make it 2, still below limit 3. 123 + const last = new Date(Date.now() - 5000).toISOString(); 124 + const exit = new Date(Date.now() - 4000).toISOString(); 125 + writeExitedPermanent(dir, name, { 126 + lastRespawnAt: last, exitedAt: exit, counter: 1, 127 + commandHash: commandHash("sh", ["-c", "exit 1"]), 128 + }); 129 + 130 + const r = runCli(dir, "gc", "--dry-run"); 131 + expect(r.status).toBe(0); 132 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 133 + expect(r.stdout).not.toContain("Would flap"); 134 + // Dry-run must not mutate tags on disk. 135 + const meta = readMeta(dir, name); 136 + expect(meta.tags["strategy.consecutive-fast-fails"]).toBe("1"); 137 + expect(meta.tags["strategy.status"]).toBeUndefined(); 138 + }); 139 + 140 + it("dry-run: at-limit tick previews Would flap and no respawn", () => { 141 + const dir = makeSessionDir(); 142 + const name = uniqueName(); 143 + const last = new Date(Date.now() - 5000).toISOString(); 144 + const exit = new Date(Date.now() - 4000).toISOString(); 145 + // Prior counter 2 → this tick makes it 3, hits default limit → flap. 146 + writeExitedPermanent(dir, name, { 147 + lastRespawnAt: last, exitedAt: exit, counter: 2, 148 + commandHash: commandHash("sh", ["-c", "exit 1"]), 149 + }); 150 + 151 + const r = runCli(dir, "gc", "--dry-run"); 152 + expect(r.status).toBe(0); 153 + expect(r.stdout).toMatch(new RegExp(`Would flap: ${name} \$3 fast-fails in 60s, limit 3\$`)); 154 + expect(r.stdout).not.toMatch(new RegExp(`Would respawn: ${name}`)); 155 + 156 + // No mutation on dry-run. 157 + const meta = readMeta(dir, name); 158 + expect(meta.tags["strategy.status"]).toBeUndefined(); 159 + expect(meta.tags["strategy.consecutive-fast-fails"]).toBe("2"); 160 + }); 161 + 162 + it("at-limit tick persists strategy.status=flapping and emits session_flapping", () => { 163 + const dir = makeSessionDir(); 164 + const name = uniqueName(); 165 + const last = new Date(Date.now() - 5000).toISOString(); 166 + const exit = new Date(Date.now() - 4000).toISOString(); 167 + writeExitedPermanent(dir, name, { 168 + lastRespawnAt: last, exitedAt: exit, counter: 2, 169 + commandHash: commandHash("sh", ["-c", "exit 1"]), 170 + }); 171 + 172 + const r = runCli(dir, "gc"); 173 + expect(r.status).toBe(0); 174 + expect(r.stdout).toContain(`Flapping: ${name} (3 fast-fails in 60s, limit 3)`); 175 + 176 + const meta = readMeta(dir, name); 177 + expect(meta.tags["strategy.status"]).toBe("flapping"); 178 + expect(meta.tags["strategy.consecutive-fast-fails"]).toBe("3"); 179 + 180 + const events = readEvents(dir, name); 181 + const flap = events.find((e) => e.type === "session_flapping"); 182 + expect(flap).toBeDefined(); 183 + expect(flap.counter).toBe(3); 184 + expect(flap.limit).toBe(3); 185 + expect(flap.window).toBe(60); 186 + }); 187 + 188 + it("already-flapping session is silently skipped on subsequent tick", () => { 189 + const dir = makeSessionDir(); 190 + const name = uniqueName(); 191 + writeExitedPermanent(dir, name, { 192 + status: "flapping", 193 + counter: 3, 194 + lastRespawnAt: new Date(Date.now() - 60_000).toISOString(), 195 + exitedAt: new Date(Date.now() - 55_000).toISOString(), 196 + commandHash: commandHash("sh", ["-c", "exit 1"]), 197 + }); 198 + 199 + const r = runCli(dir, "gc"); 200 + expect(r.status).toBe(0); 201 + expect(r.stdout).toContain(`Skipped (flapping): ${name}`); 202 + expect(r.stdout).not.toContain(`Respawned: ${name}`); 203 + 204 + // No new events beyond what was already there. 205 + const events = readEvents(dir, name); 206 + expect(events.filter((e) => e.type === "session_flapping").length).toBe(0); 207 + }); 208 + 209 + it("slow-fail (past window) resets the counter to 0", () => { 210 + const dir = makeSessionDir(); 211 + const name = uniqueName(); 212 + // Last respawn 10 minutes ago, exited 9 minutes ago → 60s live, past 213 + // the default 60s window (using an exit ~60m after respawn). 214 + const last = new Date(Date.now() - 10 * 60_000).toISOString(); 215 + const exit = new Date(Date.now() - 5 * 60_000).toISOString(); 216 + writeExitedPermanent(dir, name, { 217 + lastRespawnAt: last, exitedAt: exit, counter: 2, 218 + commandHash: commandHash("sh", ["-c", "exit 1"]), 219 + }); 220 + 221 + const r = runCli(dir, "gc", "--dry-run"); 222 + expect(r.status).toBe(0); 223 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 224 + expect(r.stdout).not.toContain("Would flap"); 225 + // (Counter reset is verified via the "no flap at same prior counter" 226 + // outcome — a fast fail at prior=2 would have flapped; slow-fail 227 + // continues to respawn.) 228 + }); 229 + 230 + it("command-hash change auto-resets counter and clears flapping mark", () => { 231 + const dir = makeSessionDir(); 232 + const name = uniqueName(); 233 + // Session is currently flagged flapping under an old command hash. 234 + // Metadata now reports a DIFFERENT command (operator edited pty.toml). 235 + // The classifier should notice the divergence and both reset the 236 + // counter and clear the flapping mark, letting gc respawn. 237 + const oldHash = commandHash("sh", ["-c", "old-command"]); 238 + writeExitedPermanent(dir, name, { 239 + command: "sh", args: ["-c", "exit 1"], // new command 240 + status: "flapping", counter: 3, 241 + lastRespawnAt: new Date(Date.now() - 5000).toISOString(), 242 + exitedAt: new Date(Date.now() - 4000).toISOString(), 243 + commandHash: oldHash, // stale hash 244 + }); 245 + 246 + const r = runCli(dir, "gc", "--dry-run"); 247 + expect(r.status).toBe(0); 248 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 249 + expect(r.stdout).not.toContain("Skipped (flapping)"); 250 + expect(r.stdout).not.toContain("Would flap"); 251 + }); 252 + 253 + it("per-session strategy.fast-fail-limit overrides the default", () => { 254 + const dir = makeSessionDir(); 255 + const name = uniqueName(); 256 + // Prior counter 1 + fast fail → this tick makes it 2. Default limit 257 + // 3 would let it respawn; per-session limit 2 flaps instead. 258 + writeExitedPermanent(dir, name, { 259 + lastRespawnAt: new Date(Date.now() - 5000).toISOString(), 260 + exitedAt: new Date(Date.now() - 4000).toISOString(), 261 + counter: 1, 262 + commandHash: commandHash("sh", ["-c", "exit 1"]), 263 + tags: { "strategy.fast-fail-limit": "2" }, 264 + }); 265 + 266 + const r = runCli(dir, "gc", "--dry-run"); 267 + expect(r.status).toBe(0); 268 + expect(r.stdout).toMatch(new RegExp(`Would flap: ${name} \$2 fast-fails in 60s, limit 2\$`)); 269 + }); 270 + 271 + it("--fast-fail-limit CLI flag applies to sessions without a per-session tag", () => { 272 + const dir = makeSessionDir(); 273 + const name = uniqueName(); 274 + // Prior counter 4 + fast fail → 5. CLI flag lifts limit to 10 → still respawn. 275 + writeExitedPermanent(dir, name, { 276 + lastRespawnAt: new Date(Date.now() - 5000).toISOString(), 277 + exitedAt: new Date(Date.now() - 4000).toISOString(), 278 + counter: 4, 279 + commandHash: commandHash("sh", ["-c", "exit 1"]), 280 + }); 281 + 282 + const r = runCli(dir, "gc", "--dry-run", "--fast-fail-limit=10"); 283 + expect(r.status).toBe(0); 284 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 285 + expect(r.stdout).not.toContain("Would flap"); 286 + }); 287 + 288 + it("per-session strategy.fast-fail-window overrides the default", () => { 289 + const dir = makeSessionDir(); 290 + const name = uniqueName(); 291 + // Live time = 30s. Default window 60s → fast fail. Per-session window 292 + // 10s → slow fail (30 > 10) → counter resets → respawn, no flap. 293 + writeExitedPermanent(dir, name, { 294 + lastRespawnAt: new Date(Date.now() - 30_000).toISOString(), 295 + exitedAt: new Date(Date.now() - 0).toISOString(), 296 + counter: 2, 297 + commandHash: commandHash("sh", ["-c", "exit 1"]), 298 + tags: { "strategy.fast-fail-window": "10" }, 299 + }); 300 + 301 + const r = runCli(dir, "gc", "--dry-run"); 302 + expect(r.status).toBe(0); 303 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 304 + expect(r.stdout).not.toContain("Would flap"); 305 + }); 306 + 307 + it("no prior last-respawn-at (first respawn ever) doesn't count as fast fail", () => { 308 + const dir = makeSessionDir(); 309 + const name = uniqueName(); 310 + // Session exited fresh — no last-respawn-at tag yet. Counter absent. 311 + writeExitedPermanent(dir, name, { 312 + exitedAt: new Date().toISOString(), 313 + }); 314 + 315 + const r = runCli(dir, "gc", "--dry-run"); 316 + expect(r.status).toBe(0); 317 + expect(r.stdout).toMatch(new RegExp(`Would respawn: ${name}`)); 318 + expect(r.stdout).not.toContain("Would flap"); 319 + }); 320 + });

Configure Feed

Configure Feed