+21

README.md

Reviewed

··· 56 56 cargo test -p trawler-core --release -- --ignored 57 57 ``` 58 58 59 59 + ### Data-integrity gates 60 60 + 61 61 + Two seeded gates (openspec capability `integrity-gates`) run in the default 62 62 + `cargo test --workspace` and make the data-safety promises executable: 63 63 + 64 64 + - **Rebuild equivalence** (`tests/rebuild_equivalence.rs`): after a random 65 65 + op stream with `GraphIndex` and the tantivy index maintained incrementally 66 66 + (exactly as the app maintains them), rebuilding both from the Loro doc 67 67 + alone must answer identically — backlinks, tags, properties, structure, 68 68 + and search hits. The comparator (`integrity::ObservableState`) is the 69 69 + working definition of "derived state"; extend it when adding a new derived 70 70 + structure. 71 71 + - **Crash consistency** (`tests/crash_safety.rs`): a child process applies 72 72 + and persists seeded edits, is hard-killed (`TerminateProcess` — no 73 73 + cleanup) at a random moment, and on reopen every acknowledged edit must be 74 74 + present with derived state matching a deterministic reference replay. 75 75 + 76 76 + Both print their seed; reproduce any failure exactly with 77 77 + `TRAWLER_TEST_SEED=<seed>`. Deeper sweeps are `#[ignore]`d alongside the 78 78 + other expensive tests. 79 79 + 59 80 ### Where your data lives 60 81 61 82 By default the graph directory is `%APPDATA%\trawler\graph`. Override it with

+5 -1

crates/trawler-core/Cargo.toml

Reviewed

··· 10 10 [features] 11 11 # Deterministic fixture-graph builder shared by tests and dev-server 12 12 # sessions (openspec change add-dev-automation, capability fixture-graphs). 13 13 - fixtures = [] 13 13 + # Also gates the integrity-gate test support (openspec change 14 14 + # add-integrity-gates): the seeded op generator and observable-state 15 15 + # comparator in `integrity`, which need `rand`. 16 16 + fixtures = ["dep:rand"] 14 17 15 18 [dependencies] 16 19 chrono = { version = "0.4.45", features = ["serde"] } 17 20 loro = "1.13.6" 21 21 + rand = { version = "0.10.2", optional = true } 18 22 serde = { version = "1.0.228", features = ["derive"] } 19 23 steel-core = "0.8.2" 20 24 tantivy = "0.26.1"

+174 -17

crates/trawler-core/src/index.rs

Reviewed

··· 126 126 IndexEvent::ChildrenChanged(parent) => self.sync_children(&outline, parent), 127 127 IndexEvent::BlockContentChanged(id) => self.sync_content(&outline, id), 128 128 IndexEvent::BlockPropertiesChanged(id) => self.sync_properties(&outline, id), 129 129 - IndexEvent::SubtreeDeleted(id) => self.remove_subtree(id), 129 129 + IndexEvent::SubtreeDeleted(id) => self.remove_subtree(&outline, id), 130 130 } 131 131 } 132 132 ··· 142 142 } 143 143 None => { 144 144 self.roots = outline.children(None); 145 145 - // Cheap to fully resync: bounded by page count, not block 146 146 - // count, so this stays O(pages) even for a 100k-block graph. 147 147 - self.pages_by_name.clear(); 148 148 - for &root in &self.roots { 149 149 - if let Ok(name) = outline.content(root) { 150 150 - self.pages_by_name.insert(name, root); 145 145 + self.refresh_page_names(outline); 146 146 + } 147 147 + } 148 148 + } 149 149 + 150 150 + /// Recompute `pages_by_name` from `roots` (in root order, so duplicate 151 151 + /// page names deterministically resolve to the *last* root — the same 152 152 + /// rule `rebuild` applies), then re-resolve every block that references 153 153 + /// a name whose page mapping changed. Without the re-resolution step, 154 154 + /// `[[Name]]` backlinks resolved at indexing time go stale when a page 155 155 + /// with that name is later created, deleted, renamed, or moved — the 156 156 + /// first bug the rebuild-equivalence gate caught (openspec change 157 157 + /// add-integrity-gates), and user-visible as backlinks pointing at the 158 158 + /// wrong node until a restart. 159 159 + /// 160 160 + /// Cost: O(pages) for the recompute (bounded by page count, not block 161 161 + /// count) plus O(referrers of changed names) for the re-resolution. 162 162 + fn refresh_page_names(&mut self, outline: &Outline) { 163 163 + let old = std::mem::take(&mut self.pages_by_name); 164 164 + for &root in &self.roots { 165 165 + if let Ok(name) = outline.content(root) { 166 166 + self.pages_by_name.insert(name, root); 167 167 + } 168 168 + } 169 169 + 170 170 + // Blocks whose reference targets may have changed: anything 171 171 + // referencing a changed name as a tag-fallback, or referencing the 172 172 + // page id that gained/lost that name. Collected first so the 173 173 + // relink pass below can borrow mutably. 174 174 + let mut affected: HashSet<TreeID> = HashSet::new(); 175 175 + { 176 176 + let mut names: HashSet<&String> = old.keys().collect(); 177 177 + names.extend(self.pages_by_name.keys()); 178 178 + for name in names { 179 179 + let before = old.get(name); 180 180 + let after = self.pages_by_name.get(name); 181 181 + if before == after { 182 182 + continue; 183 183 + } 184 184 + if let Some(set) = self.backlinks.get(&NodeId::tag(name.clone())) { 185 185 + affected.extend(set.iter().copied()); 186 186 + } 187 187 + for id in [before, after].into_iter().flatten() { 188 188 + if let Some(set) = self.backlinks.get(&NodeId::tree(*id)) { 189 189 + affected.extend(set.iter().copied()); 151 190 } 152 191 } 153 192 } 154 193 } 194 194 + for block in affected { 195 195 + self.relink_block_refs(outline, block); 196 196 + } 155 197 } 156 198 157 199 fn sync_content(&mut self, outline: &Outline, id: TreeID) { 200 200 + self.relink_block_refs(outline, id); 201 201 + 202 202 + // A page's own content is its name: a root rename can retarget 203 203 + // every `[[Name]]` reference in the graph, so refresh the name 204 204 + // table (and re-resolve affected referrers) without waiting for a 205 205 + // `ChildrenChanged(None)` event. 206 206 + if self.roots.contains(&id) { 207 207 + self.refresh_page_names(outline); 208 208 + } 209 209 + } 210 210 + 211 211 + /// Re-parse `id`'s content and re-resolve its outgoing references 212 212 + /// against the *current* page-name table: the reference half of 213 213 + /// [`sync_content`], without the page-name maintenance (so 214 214 + /// [`refresh_page_names`] can call it for affected referrers without 215 215 + /// recursing). 216 216 + fn relink_block_refs(&mut self, outline: &Outline, id: TreeID) { 158 217 if let Some(old_targets) = self.block_outgoing_refs.remove(&id) { 159 218 for target in old_targets { 160 219 self.unlink_backlink(&target, id); ··· 172 231 } 173 232 if !new_targets.is_empty() { 174 233 self.block_outgoing_refs.insert(id, new_targets); 175 175 - } 176 176 - 177 177 - // A page's own content is its name; keep pages_by_name in sync on 178 178 - // rename without waiting for a `ChildrenChanged(None)` event. 179 179 - if self.roots.contains(&id) { 180 180 - self.pages_by_name.retain(|_, &mut v| v != id); 181 181 - self.pages_by_name.insert(content, id); 182 234 } 183 235 } 184 236 ··· 212 264 213 265 /// Remove `root` and every descendant currently known to the index (via 214 266 /// our own `children` map, since by the time this is called the doc has 215 215 - /// already deleted them and can no longer be queried for them). 216 216 - fn remove_subtree(&mut self, root: TreeID) { 267 267 + /// already deleted them and can no longer be queried for them), then 268 268 + /// refresh the page-name table — deleting a page can retarget `[[Name]]` 269 269 + /// references (to another page with the same name, or back to the tag 270 270 + /// fallback). 271 271 + fn remove_subtree(&mut self, outline: &Outline, root: TreeID) { 272 272 + let deleted_a_root = self.roots.contains(&root); 217 273 let mut queue = vec![root]; 218 274 while let Some(id) = queue.pop() { 219 275 if let Some(targets) = self.block_outgoing_refs.remove(&id) { ··· 235 291 queue.extend(children); 236 292 } 237 293 self.roots.retain(|&r| r != id); 238 238 - self.pages_by_name.retain(|_, &mut v| v != id); 294 294 + } 295 295 + if deleted_a_root { 296 296 + self.refresh_page_names(outline); 239 297 } 240 298 } 241 299 ··· 509 567 let index = GraphIndex::rebuild(&doc); 510 568 511 569 assert_eq!(index.children.get(&page), Some(&vec![a, b, c])); 570 570 + } 571 571 + 572 572 + /// Regression tests for stale name resolution (openspec change 573 573 + /// add-integrity-gates, first gate catch): `[[Name]]` backlinks must 574 574 + /// re-resolve when the page owning that name is created, deleted, or 575 575 + /// renamed after the reference was indexed. 576 576 + #[test] 577 577 + fn page_lifecycle_retargets_existing_name_references() { 578 578 + let doc = doc_with_tree(); 579 579 + let outline = Outline::new(&doc); 580 580 + let mut index = GraphIndex::default(); 581 581 + 582 582 + let home = outline 583 583 + .create_block(None, Position::Index(0), "Home") 584 584 + .unwrap(); 585 585 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 586 586 + index.apply(&doc, IndexEvent::BlockContentChanged(home)); 587 587 + let referrer = outline 588 588 + .create_block(Some(home), Position::Index(0), "see [[Fishing]]") 589 589 + .unwrap(); 590 590 + index.apply(&doc, IndexEvent::ChildrenChanged(Some(home))); 591 591 + index.apply(&doc, IndexEvent::BlockContentChanged(referrer)); 592 592 + 593 593 + // Not yet created: resolves to the tag fallback. 594 594 + assert_eq!( 595 595 + index.backlinks_of(&NodeId::tag("Fishing")), 596 596 + HashSet::from([referrer]) 597 597 + ); 598 598 + 599 599 + // Page created after the reference: backlink retargets to the page. 600 600 + let fishing = outline 601 601 + .create_block(None, Position::Index(1), "Fishing") 602 602 + .unwrap(); 603 603 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 604 604 + index.apply(&doc, IndexEvent::BlockContentChanged(fishing)); 605 605 + assert_eq!( 606 606 + index.backlinks_of(&NodeId::tree(fishing)), 607 607 + HashSet::from([referrer]) 608 608 + ); 609 609 + assert!(index.backlinks_of(&NodeId::tag("Fishing")).is_empty()); 610 610 + assert_eq!(index, GraphIndex::rebuild(&doc)); 611 611 + 612 612 + // Page renamed: backlink falls back to the tag. 613 613 + outline.set_content(fishing, "Angling").unwrap(); 614 614 + index.apply(&doc, IndexEvent::BlockContentChanged(fishing)); 615 615 + assert_eq!( 616 616 + index.backlinks_of(&NodeId::tag("Fishing")), 617 617 + HashSet::from([referrer]) 618 618 + ); 619 619 + assert_eq!(index, GraphIndex::rebuild(&doc)); 620 620 + 621 621 + // Renamed back, then deleted: tag fallback again. 622 622 + outline.set_content(fishing, "Fishing").unwrap(); 623 623 + index.apply(&doc, IndexEvent::BlockContentChanged(fishing)); 624 624 + outline.delete_block(fishing).unwrap(); 625 625 + index.apply(&doc, IndexEvent::SubtreeDeleted(fishing)); 626 626 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 627 627 + assert_eq!( 628 628 + index.backlinks_of(&NodeId::tag("Fishing")), 629 629 + HashSet::from([referrer]) 630 630 + ); 631 631 + assert_eq!(index, GraphIndex::rebuild(&doc)); 632 632 + } 633 633 + 634 634 + /// Duplicate page names must resolve identically incrementally and on 635 635 + /// rebuild (both: last root in root order wins), including after the 636 636 + /// winning duplicate is deleted. 637 637 + #[test] 638 638 + fn duplicate_page_names_resolve_like_rebuild() { 639 639 + let doc = doc_with_tree(); 640 640 + let outline = Outline::new(&doc); 641 641 + let mut index = GraphIndex::default(); 642 642 + 643 643 + let first = outline 644 644 + .create_block(None, Position::Index(0), "Notes") 645 645 + .unwrap(); 646 646 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 647 647 + index.apply(&doc, IndexEvent::BlockContentChanged(first)); 648 648 + let referrer = outline 649 649 + .create_block(Some(first), Position::Index(0), "see [[Notes]]") 650 650 + .unwrap(); 651 651 + index.apply(&doc, IndexEvent::ChildrenChanged(Some(first))); 652 652 + index.apply(&doc, IndexEvent::BlockContentChanged(referrer)); 653 653 + 654 654 + let second = outline 655 655 + .create_block(None, Position::Index(1), "Notes") 656 656 + .unwrap(); 657 657 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 658 658 + index.apply(&doc, IndexEvent::BlockContentChanged(second)); 659 659 + assert_eq!(index, GraphIndex::rebuild(&doc)); 660 660 + 661 661 + outline.delete_block(second).unwrap(); 662 662 + index.apply(&doc, IndexEvent::SubtreeDeleted(second)); 663 663 + index.apply(&doc, IndexEvent::ChildrenChanged(None)); 664 664 + assert_eq!( 665 665 + index.backlinks_of(&NodeId::tree(first)), 666 666 + HashSet::from([referrer]) 667 667 + ); 668 668 + assert_eq!(index, GraphIndex::rebuild(&doc)); 512 669 } 513 670 514 671 /// spec scenario: "Incremental equals rebuild" — a randomized sequence

+513

crates/trawler-core/src/integrity.rs

Reviewed

··· 1 1 + //! Test support for the integrity gates (openspec change 2 2 + //! add-integrity-gates): a seeded random op generator and an 3 3 + //! observable-state comparator, shared by the rebuild-equivalence and 4 4 + //! crash-consistency tests (and reusable by future gates, e.g. the 5 5 + //! traced-revalidation gate planned in add-supertags). 6 6 + //! 7 7 + //! Determinism contract: the same seed applied to the same starting 8 8 + //! document produces the identical op stream — the crash gate's reference 9 9 + //! replay depends on this, so nothing here may consult clocks, map 10 10 + //! iteration order, or any entropy beyond the seeded `StdRng`. 11 11 + 12 12 + use std::collections::BTreeMap; 13 13 + 14 14 + use loro::TreeID; 15 15 + use rand::rngs::StdRng; 16 16 + use rand::{RngExt, SeedableRng}; 17 17 + 18 18 + use crate::graph::PropertyValue; 19 19 + use crate::index::{GraphIndex, IndexEvent}; 20 20 + use crate::outline::{Outline, Position}; 21 21 + use crate::search::SearchIndex; 22 22 + 23 23 + /// The seed for a gate run: `TRAWLER_TEST_SEED` if set (reproduce a CI 24 24 + /// failure locally), otherwise derived from the wall clock. Callers must 25 25 + /// print the returned seed so every failure is reproducible from its log 26 26 + /// alone (design D5). 27 27 + pub fn seed_from_env_or_entropy() -> u64 { 28 28 + match std::env::var("TRAWLER_TEST_SEED") { 29 29 + Ok(s) => s 30 30 + .parse() 31 31 + .unwrap_or_else(|_| panic!("TRAWLER_TEST_SEED must be a u64, got {s:?}")), 32 32 + Err(_) => { 33 33 + std::time::SystemTime::now() 34 34 + .duration_since(std::time::UNIX_EPOCH) 35 35 + .expect("clock after epoch") 36 36 + .subsec_nanos() as u64 37 37 + ^ 0x5EED_5EED 38 38 + } 39 39 + } 40 40 + } 41 41 + 42 42 + /// What one generated op did, in terms callers need to maintain derived 43 43 + /// state alongside the mutation: 44 44 + /// - feed every entry of `events` to [`GraphIndex::apply`]; 45 45 + /// - re-index (upsert) every block in `content_changed` in the search index; 46 46 + /// - remove every block in `deleted` from the search index. 47 47 + #[derive(Debug, Default)] 48 48 + pub struct StepOutcome { 49 49 + pub events: Vec<IndexEvent>, 50 50 + pub content_changed: Vec<TreeID>, 51 51 + pub deleted: Vec<TreeID>, 52 52 + /// Human-readable description of the op, for failure logs. 53 53 + pub description: String, 54 54 + } 55 55 + 56 56 + /// Seeded generator of a mixed, outliner-realistic op stream. Weights 57 57 + /// favor text edits (the real-world mix); every op goes through the real 58 58 + /// [`Outline`] mutation API. 59 59 + pub struct OpGenerator { 60 60 + pub seed: u64, 61 61 + rng: StdRng, 62 62 + live: Vec<TreeID>, 63 63 + } 64 64 + 65 65 + impl OpGenerator { 66 66 + pub fn new(seed: u64) -> Self { 67 67 + Self { 68 68 + seed, 69 69 + rng: StdRng::seed_from_u64(seed), 70 70 + live: Vec::new(), 71 71 + } 72 72 + } 73 73 + 74 74 + /// Apply one random op to `outline` and report what changed. Never 75 75 + /// fails: ops that turn out inapplicable (e.g. merge with no eligible 76 76 + /// pair) degrade to a create. 77 77 + pub fn step(&mut self, outline: &Outline) -> StepOutcome { 78 78 + if self.live.is_empty() { 79 79 + return self.create(outline, true); 80 80 + } 81 81 + match self.rng.random_range(0..100) { 82 82 + 0..=27 => { 83 83 + let make_page = self.rng.random_range(0..10) == 0; 84 84 + self.create(outline, make_page) 85 85 + } 86 86 + 28..=47 => self.edit_content(outline), 87 87 + 48..=55 => self.split(outline), 88 88 + 56..=62 => self.merge(outline), 89 89 + 63..=70 => self.set_prop(outline), 90 90 + 71..=74 => self.remove_prop(outline), 91 91 + 75..=84 => self.move_subtree(outline), 92 92 + 85..=91 => self.reorder(outline), 93 93 + _ => self.delete(outline), 94 94 + } 95 95 + } 96 96 + 97 97 + fn pick(&mut self) -> TreeID { 98 98 + self.live[self.rng.random_range(0..self.live.len())] 99 99 + } 100 100 + 101 101 + fn random_content(&mut self) -> String { 102 102 + match self.rng.random_range(0..7) { 103 103 + 0 => format!("plain note {}", self.rng.random_range(0..1000)), 104 104 + 1 => format!("tagged #topic{}", self.rng.random_range(0..8)), 105 105 + 2 => format!("see [[Page{}]] for details", self.rng.random_range(0..8)), 106 106 + 3 => format!( 107 107 + "due [[2026-07-{:02}]] hopefully", 108 108 + self.rng.random_range(1..29) 109 109 + ), 110 110 + 4 => { 111 111 + let id = self.pick(); 112 112 + format!("follow up on (({id}))") 113 113 + } 114 114 + 5 => format!( 115 115 + "mending the cod-end after haul {} #fishing", 116 116 + self.rng.random_range(0..50) 117 117 + ), 118 118 + _ => format!( 119 119 + "word{} word{} word{}", 120 120 + self.rng.random_range(0..30), 121 121 + self.rng.random_range(0..30), 122 122 + self.rng.random_range(0..30) 123 123 + ), 124 124 + } 125 125 + } 126 126 + 127 127 + fn create(&mut self, outline: &Outline, make_page: bool) -> StepOutcome { 128 128 + let parent = if make_page || self.live.is_empty() { 129 129 + None 130 130 + } else { 131 131 + Some(self.pick()) 132 132 + }; 133 133 + let content = if parent.is_none() { 134 134 + format!("Page{}", self.rng.random_range(0..8)) 135 135 + } else { 136 136 + self.random_content() 137 137 + }; 138 138 + let n = outline.children(parent).len(); 139 139 + let ix = self.rng.random_range(0..=n); 140 140 + let id = outline 141 141 + .create_block(parent, Position::Index(ix), &content) 142 142 + .expect("create_block"); 143 143 + self.live.push(id); 144 144 + StepOutcome { 145 145 + events: vec![ 146 146 + IndexEvent::ChildrenChanged(parent), 147 147 + IndexEvent::BlockContentChanged(id), 148 148 + ], 149 149 + content_changed: vec![id], 150 150 + deleted: vec![], 151 151 + description: format!("create {id} under {parent:?}: {content:?}"), 152 152 + } 153 153 + } 154 154 + 155 155 + fn edit_content(&mut self, outline: &Outline) -> StepOutcome { 156 156 + let id = self.pick(); 157 157 + let content = self.random_content(); 158 158 + outline.set_content(id, &content).expect("set_content"); 159 159 + StepOutcome { 160 160 + events: vec![IndexEvent::BlockContentChanged(id)], 161 161 + content_changed: vec![id], 162 162 + deleted: vec![], 163 163 + description: format!("edit {id}: {content:?}"), 164 164 + } 165 165 + } 166 166 + 167 167 + fn split(&mut self, outline: &Outline) -> StepOutcome { 168 168 + let id = self.pick(); 169 169 + let content = outline.content(id).expect("content"); 170 170 + let len = content.chars().count(); 171 171 + if len < 2 { 172 172 + return self.create(outline, false); 173 173 + } 174 174 + let offset = self.rng.random_range(1..len); 175 175 + let new_id = outline.split_block(id, offset).expect("split_block"); 176 176 + self.live.push(new_id); 177 177 + let parent = outline.parent(id); 178 178 + StepOutcome { 179 179 + events: vec![ 180 180 + IndexEvent::BlockContentChanged(id), 181 181 + IndexEvent::ChildrenChanged(parent), 182 182 + IndexEvent::BlockContentChanged(new_id), 183 183 + ], 184 184 + content_changed: vec![id, new_id], 185 185 + deleted: vec![], 186 186 + description: format!("split {id} at {offset} -> {new_id}"), 187 187 + } 188 188 + } 189 189 + 190 190 + fn merge(&mut self, outline: &Outline) -> StepOutcome { 191 191 + // Find a childless block with a previous sibling (the UI's merge 192 192 + // shape: backspace-at-start merges into the block above). Bounded 193 193 + // probe so a stream with no eligible pair degrades to create. 194 194 + for _ in 0..8 { 195 195 + let id = self.pick(); 196 196 + if !outline.children(Some(id)).is_empty() { 197 197 + continue; 198 198 + } 199 199 + let parent = outline.parent(id); 200 200 + let siblings = outline.children(parent); 201 201 + let Some(pos) = siblings.iter().position(|&s| s == id) else { 202 202 + continue; 203 203 + }; 204 204 + if pos == 0 { 205 205 + continue; 206 206 + } 207 207 + let into = siblings[pos - 1]; 208 208 + outline.merge_block(id, into).expect("merge_block"); 209 209 + self.live.retain(|&b| b != id); 210 210 + return StepOutcome { 211 211 + events: vec![ 212 212 + IndexEvent::BlockContentChanged(into), 213 213 + IndexEvent::SubtreeDeleted(id), 214 214 + IndexEvent::ChildrenChanged(parent), 215 215 + ], 216 216 + content_changed: vec![into], 217 217 + deleted: vec![id], 218 218 + description: format!("merge {id} into {into}"), 219 219 + }; 220 220 + } 221 221 + self.create(outline, false) 222 222 + } 223 223 + 224 224 + fn set_prop(&mut self, outline: &Outline) -> StepOutcome { 225 225 + let id = self.pick(); 226 226 + let key = ["due", "priority", "done", "rating"][self.rng.random_range(0..4)]; 227 227 + let value = match self.rng.random_range(0..4) { 228 228 + 0 => PropertyValue::Text(format!("v{}", self.rng.random_range(0..100))), 229 229 + 1 => PropertyValue::Number(self.rng.random_range(0..100) as f64), 230 230 + 2 => PropertyValue::Bool(self.rng.random_range(0..2) == 0), 231 231 + _ => PropertyValue::Date( 232 232 + chrono::NaiveDate::from_ymd_opt(2026, 7, self.rng.random_range(1..29)) 233 233 + .expect("valid generated date"), 234 234 + ), 235 235 + }; 236 236 + outline.set_property(id, key, &value).expect("set_property"); 237 237 + StepOutcome { 238 238 + events: vec![IndexEvent::BlockPropertiesChanged(id)], 239 239 + content_changed: vec![], 240 240 + deleted: vec![], 241 241 + description: format!("set-prop {id} {key}={value:?}"), 242 242 + } 243 243 + } 244 244 + 245 245 + fn remove_prop(&mut self, outline: &Outline) -> StepOutcome { 246 246 + let id = self.pick(); 247 247 + let key = ["due", "priority", "done", "rating"][self.rng.random_range(0..4)]; 248 248 + outline.remove_property(id, key).expect("remove_property"); 249 249 + StepOutcome { 250 250 + events: vec![IndexEvent::BlockPropertiesChanged(id)], 251 251 + content_changed: vec![], 252 252 + deleted: vec![], 253 253 + description: format!("remove-prop {id} {key}"), 254 254 + } 255 255 + } 256 256 + 257 257 + fn move_subtree(&mut self, outline: &Outline) -> StepOutcome { 258 258 + let id = self.pick(); 259 259 + let old_parent = outline.parent(id); 260 260 + let candidate = self.pick(); 261 261 + let new_parent = if candidate == id || is_descendant(outline, id, candidate) { 262 262 + None 263 263 + } else { 264 264 + Some(candidate) 265 265 + }; 266 266 + let n = outline.children(new_parent).len(); 267 267 + let ix = self.rng.random_range(0..=n); 268 268 + if outline 269 269 + .move_subtree(id, new_parent, Position::Index(ix)) 270 270 + .is_err() 271 271 + { 272 272 + return self.create(outline, false); 273 273 + } 274 274 + StepOutcome { 275 275 + events: vec![ 276 276 + IndexEvent::ChildrenChanged(old_parent), 277 277 + IndexEvent::ChildrenChanged(new_parent), 278 278 + ], 279 279 + content_changed: vec![], 280 280 + deleted: vec![], 281 281 + description: format!("move {id} from {old_parent:?} to {new_parent:?}"), 282 282 + } 283 283 + } 284 284 + 285 285 + fn reorder(&mut self, outline: &Outline) -> StepOutcome { 286 286 + let id = self.pick(); 287 287 + let parent = outline.parent(id); 288 288 + let siblings = outline.children(parent); 289 289 + if siblings.len() < 2 { 290 290 + return self.create(outline, false); 291 291 + } 292 292 + let other = siblings[self.rng.random_range(0..siblings.len())]; 293 293 + if other == id || outline.reorder_sibling(id, Position::After(other)).is_err() { 294 294 + return self.create(outline, false); 295 295 + } 296 296 + StepOutcome { 297 297 + events: vec![IndexEvent::ChildrenChanged(parent)], 298 298 + content_changed: vec![], 299 299 + deleted: vec![], 300 300 + description: format!("reorder {id} after {other}"), 301 301 + } 302 302 + } 303 303 + 304 304 + fn delete(&mut self, outline: &Outline) -> StepOutcome { 305 305 + let id = self.pick(); 306 306 + let parent = outline.parent(id); 307 307 + let mut subtree = vec![id]; 308 308 + let mut stack = outline.children(Some(id)); 309 309 + while let Some(cur) = stack.pop() { 310 310 + subtree.push(cur); 311 311 + stack.extend(outline.children(Some(cur))); 312 312 + } 313 313 + if outline.delete_block(id).is_err() { 314 314 + return self.create(outline, false); 315 315 + } 316 316 + self.live.retain(|b| !subtree.contains(b)); 317 317 + StepOutcome { 318 318 + events: vec![ 319 319 + IndexEvent::SubtreeDeleted(id), 320 320 + IndexEvent::ChildrenChanged(parent), 321 321 + ], 322 322 + content_changed: vec![], 323 323 + deleted: subtree, 324 324 + description: format!("delete {id} (+ subtree)"), 325 325 + } 326 326 + } 327 327 + } 328 328 + 329 329 + fn is_descendant(outline: &Outline, ancestor: TreeID, id: TreeID) -> bool { 330 330 + let mut stack = outline.children(Some(ancestor)); 331 331 + while let Some(cur) = stack.pop() { 332 332 + if cur == id { 333 333 + return true; 334 334 + } 335 335 + stack.extend(outline.children(Some(cur))); 336 336 + } 337 337 + false 338 338 + } 339 339 + 340 340 + /// Everything the graph *answers*, flattened into an ordered key → value 341 341 + /// map so two states can be compared and the first divergence named. The 342 342 + /// keys cover: root order and page names, per-parent child order, every 343 343 + /// block's content, backlinks per node, tag membership, per-block property 344 344 + /// values, and search hits for a deterministic content-derived term sample. 345 345 + /// Deliberately not a struct-equality check — representation may change, 346 346 + /// answers may not (spec: "observable identity"). 347 347 + #[derive(Debug, Clone, PartialEq, Eq)] 348 348 + pub struct ObservableState { 349 349 + entries: BTreeMap<String, String>, 350 350 + } 351 351 + 352 352 + /// How many content-derived terms the search sample includes. 353 353 + const SEARCH_TERM_SAMPLE: usize = 24; 354 354 + 355 355 + impl ObservableState { 356 356 + /// Capture the observable state of `outline` as answered by `graph` 357 357 + /// and `search`. The caller chooses whether those indexes were 358 358 + /// maintained incrementally or rebuilt — that choice is exactly what 359 359 + /// the rebuild-equivalence gate compares. 360 360 + pub fn capture(outline: &Outline, graph: &GraphIndex, search: &SearchIndex) -> Self { 361 361 + let mut entries = BTreeMap::new(); 362 362 + 363 363 + entries.insert( 364 364 + "roots".to_string(), 365 365 + graph.roots.iter().map(id_str).collect::<Vec<_>>().join(","), 366 366 + ); 367 367 + for (name, id) in &graph.pages_by_name { 368 368 + entries.insert(format!("page-name:{name}"), id.to_string()); 369 369 + } 370 370 + for (parent, children) in &graph.children { 371 371 + entries.insert( 372 372 + format!("children:{parent}"), 373 373 + children.iter().map(id_str).collect::<Vec<_>>().join(","), 374 374 + ); 375 375 + } 376 376 + 377 377 + // Block content for every block reachable from the index's view of 378 378 + // the tree (the crash gate compares text, not just structure). 379 379 + let mut all_blocks: Vec<TreeID> = graph.roots.clone(); 380 380 + let mut stack: Vec<TreeID> = graph.roots.clone(); 381 381 + while let Some(id) = stack.pop() { 382 382 + for &child in graph.children.get(&id).map(Vec::as_slice).unwrap_or(&[]) { 383 383 + all_blocks.push(child); 384 384 + stack.push(child); 385 385 + } 386 386 + } 387 387 + let mut terms = std::collections::BTreeSet::new(); 388 388 + for &id in &all_blocks { 389 389 + if let Ok(content) = outline.content(id) { 390 390 + for word in content.split_whitespace() { 391 391 + let word: String = word 392 392 + .chars() 393 393 + .filter(|c| c.is_ascii_alphanumeric()) 394 394 + .collect::<String>() 395 395 + .to_ascii_lowercase(); 396 396 + if (4..=12).contains(&word.len()) { 397 397 + terms.insert(word); 398 398 + } 399 399 + } 400 400 + entries.insert(format!("content:{id}"), content); 401 401 + } 402 402 + } 403 403 + 404 404 + for (node, blocks) in &graph.backlinks { 405 405 + entries.insert(format!("backlink:{node}"), sorted_ids(blocks)); 406 406 + } 407 407 + for (tag, blocks) in &graph.tags { 408 408 + entries.insert(format!("tag:{tag}"), sorted_ids(blocks)); 409 409 + } 410 410 + for (date, blocks) in &graph.dates { 411 411 + entries.insert(format!("date:{date}"), sorted_ids(blocks)); 412 412 + } 413 413 + for (key, by_block) in &graph.properties { 414 414 + for (block, value) in by_block { 415 415 + entries.insert( 416 416 + format!("prop:{key}:{block}"), 417 417 + crate::properties::encode(value), 418 418 + ); 419 419 + } 420 420 + } 421 421 + 422 422 + for term in terms.into_iter().take(SEARCH_TERM_SAMPLE) { 423 423 + let hits = search 424 424 + .search(&term, 10_000) 425 425 + .unwrap_or_else(|e| panic!("search for sampled term {term:?} failed: {e}")); 426 426 + let set: std::collections::HashSet<TreeID> = 427 427 + hits.into_iter().map(|h| h.block).collect(); 428 428 + entries.insert(format!("search:{term}"), sorted_ids(&set)); 429 429 + } 430 430 + 431 431 + Self { entries } 432 432 + } 433 433 + 434 434 + /// The first divergent entry between two captures, as a readable 435 435 + /// message — `None` when equal. Keys are compared in order, so the 436 436 + /// report is deterministic. 437 437 + pub fn first_divergence(&self, other: &Self) -> Option<String> { 438 438 + let keys: std::collections::BTreeSet<&String> = 439 439 + self.entries.keys().chain(other.entries.keys()).collect(); 440 440 + for key in keys { 441 441 + match (self.entries.get(key), other.entries.get(key)) { 442 442 + (Some(a), Some(b)) if a == b => {} 443 443 + (a, b) => { 444 444 + return Some(format!( 445 445 + "first divergence at {key:?}:\n left: {a:?}\n right: {b:?}" 446 446 + )) 447 447 + } 448 448 + } 449 449 + } 450 450 + None 451 451 + } 452 452 + } 453 453 + 454 454 + fn id_str(id: &TreeID) -> String { 455 455 + id.to_string() 456 456 + } 457 457 + 458 458 + fn sorted_ids(set: &std::collections::HashSet<TreeID>) -> String { 459 459 + let mut v: Vec<String> = set.iter().map(TreeID::to_string).collect(); 460 460 + v.sort(); 461 461 + v.join(",") 462 462 + } 463 463 + 464 464 + #[cfg(test)] 465 465 + mod tests { 466 466 + use super::*; 467 467 + use loro::LoroDoc; 468 468 + 469 469 + fn doc_with_tree() -> LoroDoc { 470 470 + let doc = LoroDoc::new(); 471 471 + doc.get_tree(crate::storage::OUTLINE_TREE) 472 472 + .enable_fractional_index(0); 473 473 + doc 474 474 + } 475 475 + 476 476 + /// Same seed, same starting state → byte-identical op streams and 477 477 + /// resulting docs. The crash gate's reference replay depends on this. 478 478 + #[test] 479 479 + fn generator_is_deterministic_for_a_seed() { 480 480 + let run = |seed: u64| { 481 481 + let doc = doc_with_tree(); 482 482 + doc.set_peer_id(42).unwrap(); 483 483 + let outline = Outline::new(&doc); 484 484 + let mut generator = OpGenerator::new(seed); 485 485 + let mut descriptions = Vec::new(); 486 486 + for _ in 0..300 { 487 487 + descriptions.push(generator.step(&outline).description); 488 488 + } 489 489 + (descriptions, doc.get_deep_value()) 490 490 + }; 491 491 + 492 492 + let (desc_a, state_a) = run(7); 493 493 + let (desc_b, state_b) = run(7); 494 494 + assert_eq!(desc_a, desc_b); 495 495 + assert_eq!(state_a, state_b); 496 496 + 497 497 + let (desc_c, _) = run(8); 498 498 + assert_ne!(desc_a, desc_c, "different seeds should differ"); 499 499 + } 500 500 + 501 501 + #[test] 502 502 + fn first_divergence_names_the_differing_key() { 503 503 + let a = ObservableState { 504 504 + entries: BTreeMap::from([("k".into(), "1".into())]), 505 505 + }; 506 506 + let b = ObservableState { 507 507 + entries: BTreeMap::from([("k".into(), "2".into())]), 508 508 + }; 509 509 + let msg = a.first_divergence(&b).unwrap(); 510 510 + assert!(msg.contains("\"k\"")); 511 511 + assert!(a.first_divergence(&a).is_none()); 512 512 + } 513 513 + }

+2

crates/trawler-core/src/lib.rs

Reviewed

··· 4 4 pub mod fixtures; 5 5 pub mod graph; 6 6 pub mod index; 7 7 + #[cfg(feature = "fixtures")] 8 8 + pub mod integrity; 7 9 pub mod outline; 8 10 pub mod properties; 9 11 pub mod query;

+30

crates/trawler-core/src/outline.rs

Reviewed

··· 200 200 Ok(()) 201 201 } 202 202 203 203 + /// Remove a property from a block, if set. A no-op when the key is 204 204 + /// absent. 205 205 + pub fn remove_property(&self, id: TreeID, key: &str) -> LoroResult<()> { 206 206 + let meta = self.tree().get_meta(id)?; 207 207 + let props = meta.ensure_mergeable_map("properties")?; 208 208 + props.delete(key)?; 209 209 + Ok(()) 210 210 + } 211 211 + 203 212 /// All properties currently set on a block. 204 213 pub fn properties( 205 214 &self, ··· 343 352 344 353 let props = outline.properties(block).unwrap(); 345 354 assert_eq!(props.get("due"), Some(&due)); 355 355 + } 356 356 + 357 357 + #[test] 358 358 + fn remove_property_deletes_and_tolerates_absent_key() { 359 359 + use crate::graph::PropertyValue; 360 360 + 361 361 + let doc = doc_with_tree(); 362 362 + let outline = Outline::new(&doc); 363 363 + let page = outline.create_block(None, Position::Index(0), "").unwrap(); 364 364 + let block = outline 365 365 + .create_block(Some(page), Position::Index(0), "task") 366 366 + .unwrap(); 367 367 + 368 368 + outline 369 369 + .set_property(block, "due", &PropertyValue::Bool(true)) 370 370 + .unwrap(); 371 371 + outline.remove_property(block, "due").unwrap(); 372 372 + assert!(outline.properties(block).unwrap().is_empty()); 373 373 + 374 374 + // Removing a key that isn't set must not error. 375 375 + outline.remove_property(block, "never-set").unwrap(); 346 376 } 347 377 348 378 #[test]

+222

crates/trawler-core/tests/crash_safety.rs

Reviewed

··· 8 8 //! tasks.md 2.8. 9 9 10 10 use std::fs; 11 11 + use std::io::{BufRead, BufReader, Write}; 11 12 use std::path::PathBuf; 13 13 + use std::process::{Command, Stdio}; 14 14 + use std::sync::mpsc; 15 15 + use std::time::Duration; 12 16 13 17 use trawler_core::index::GraphIndex; 18 18 + use trawler_core::integrity::{seed_from_env_or_entropy, ObservableState, OpGenerator}; 14 19 use trawler_core::outline::{Outline, Position}; 20 20 + use trawler_core::search::SearchIndex; 15 21 use trawler_core::storage::GraphStorage; 16 22 17 23 fn temp_dir(name: &str) -> PathBuf { ··· 129 135 assert_eq!(actual, expected_content); 130 136 fs::remove_dir_all(&dir).ok(); 131 137 } 138 138 + 139 139 + // --------------------------------------------------------------------------- 140 140 + // Real-process-death crash gate (openspec change add-integrity-gates, spec 141 141 + // "Acknowledged edits survive real process death"). 142 142 + // 143 143 + // The parent test re-invokes this same test binary as a "victim" child 144 144 + // (env-var-gated), lets it apply seeded ops — acknowledging each persisted 145 145 + // op on stdout — and hard-kills it at a random moment. `Child::kill` on 146 146 + // Windows is `TerminateProcess`: immediate, no signal handlers, no cleanup, 147 147 + // no destructors — exactly the no-warning death the guarantee is about 148 148 + // (tasks 3.4). On reopen, every acknowledged op must be present and derived 149 149 + // state must match a deterministic reference replay of the same seed. 150 150 + // 151 151 + // Reproduce a failure: `TRAWLER_TEST_SEED=<seed> cargo test -p trawler-core 152 152 + // --test crash_safety killed_process`. 153 153 + // --------------------------------------------------------------------------- 154 154 + 155 155 + /// Peer id pinned in both the victim and the reference replay so generated 156 156 + /// `peer@counter` block ids line up exactly. 157 157 + const VICTIM_PEER_ID: u64 = 7777; 158 158 + /// Upper bound on victim ops — the parent kills long before this on any 159 159 + /// realistic machine; if the victim finishes first, the iteration still 160 160 + /// verifies (state == reference at full length). 161 161 + const VICTIM_MAX_OPS: usize = 600; 162 162 + 163 163 + const VICTIM_DIR_ENV: &str = "TRAWLER_CRASH_VICTIM_DIR"; 164 164 + const VICTIM_SEED_ENV: &str = "TRAWLER_CRASH_VICTIM_SEED"; 165 165 + 166 166 + /// The victim body. As a normal test (env unset) this is a no-op pass; run 167 167 + /// with the env vars set it becomes the process the parent kills. 168 168 + #[test] 169 169 + fn crash_victim_process_entry() { 170 170 + let Ok(dir) = std::env::var(VICTIM_DIR_ENV) else { 171 171 + return; 172 172 + }; 173 173 + let seed: u64 = std::env::var(VICTIM_SEED_ENV) 174 174 + .expect("victim seed env") 175 175 + .parse() 176 176 + .expect("victim seed u64"); 177 177 + 178 178 + let storage = GraphStorage::create(&dir).expect("victim create graph"); 179 179 + storage.doc().set_peer_id(VICTIM_PEER_ID).expect("pin peer"); 180 180 + let outline = Outline::new(storage.doc()); 181 181 + let mut generator = OpGenerator::new(seed); 182 182 + 183 183 + let stdout = std::io::stdout(); 184 184 + let mut out = stdout.lock(); 185 185 + writeln!(out, "VICTIM-READY").unwrap(); 186 186 + out.flush().unwrap(); 187 187 + 188 188 + for i in 0..VICTIM_MAX_OPS { 189 189 + generator.step(&outline); 190 190 + storage.persist_update().expect("victim persist"); 191 191 + // Printed strictly after persist returned: this line IS the 192 192 + // acknowledgment the guarantee covers. 193 193 + writeln!(out, "VICTIM-ACK {i}").unwrap(); 194 194 + out.flush().unwrap(); 195 195 + } 196 196 + } 197 197 + 198 198 + /// Replay `ops` generator steps from `seed` on a fresh in-memory doc with 199 199 + /// the victim's pinned peer id — the deterministic reference for what the 200 200 + /// victim's graph must contain after `ops` acknowledged ops. 201 201 + fn reference_doc(seed: u64, ops: usize) -> loro::LoroDoc { 202 202 + let doc = loro::LoroDoc::new(); 203 203 + doc.set_peer_id(VICTIM_PEER_ID).expect("pin peer"); 204 204 + doc.get_tree(trawler_core::storage::OUTLINE_TREE) 205 205 + .enable_fractional_index(0); 206 206 + let outline = Outline::new(&doc); 207 207 + let mut generator = OpGenerator::new(seed); 208 208 + for _ in 0..ops { 209 209 + generator.step(&outline); 210 210 + } 211 211 + doc 212 212 + } 213 213 + 214 214 + /// Observable state of `doc`, with all derived state rebuilt from scratch 215 215 + /// (the reopened victim graph and the reference replay are compared on 216 216 + /// identical footing). 217 217 + fn observable(doc: &loro::LoroDoc, search_dir: &PathBuf) -> ObservableState { 218 218 + let _ = fs::remove_dir_all(search_dir); 219 219 + let graph = GraphIndex::rebuild(doc); 220 220 + let search = SearchIndex::open_or_create(search_dir, doc).expect("search index"); 221 221 + let outline = Outline::new(doc); 222 222 + let state = ObservableState::capture(&outline, &graph, &search); 223 223 + fs::remove_dir_all(search_dir).ok(); 224 224 + state 225 225 + } 226 226 + 227 227 + /// One kill iteration: spawn the victim, kill after `delay_ms`, count the 228 228 + /// acknowledged prefix, reopen, and require the observable state to equal 229 229 + /// the reference replay at `acks` ops — or `acks + 1`, covering a kill in 230 230 + /// the window after a persist became durable but before its acknowledgment 231 231 + /// line was written. 232 232 + fn kill_iteration(seed: u64, delay_ms: u64) { 233 233 + println!("crash gate: seed={seed} delay={delay_ms}ms"); 234 234 + let dir = temp_dir(&format!("kill-{seed}")); 235 235 + 236 236 + let mut child = Command::new(std::env::current_exe().expect("current exe")) 237 237 + .args([ 238 238 + "crash_victim_process_entry", 239 239 + "--exact", 240 240 + "--nocapture", 241 241 + "--test-threads=1", 242 242 + ]) 243 243 + .env(VICTIM_DIR_ENV, &dir) 244 244 + .env(VICTIM_SEED_ENV, seed.to_string()) 245 245 + .stdout(Stdio::piped()) 246 246 + // Victim panics are diagnosed via the acks/reference mismatch (with 247 247 + // the seed printed), not stderr — keep CI logs quiet. 248 248 + .stderr(Stdio::null()) 249 249 + .spawn() 250 250 + .expect("spawn victim"); 251 251 + 252 252 + // A reader thread owns the pipe; the main thread owns the kill. Lines 253 253 + // arrive over a channel so a hung victim can't hang the test. 254 254 + let stdout = child.stdout.take().expect("victim stdout"); 255 255 + let (tx, rx) = mpsc::channel::<String>(); 256 256 + let reader = std::thread::spawn(move || { 257 257 + for line in BufReader::new(stdout).lines().map_while(Result::ok) { 258 258 + if tx.send(line).is_err() { 259 259 + break; 260 260 + } 261 261 + } 262 262 + }); 263 263 + 264 264 + // Wait for READY so the kill timer measures op time, not process/test- 265 265 + // harness startup. 266 266 + loop { 267 267 + match rx.recv_timeout(Duration::from_secs(60)) { 268 268 + // ends_with, not ==: libtest prints `test <name> ... ` without a 269 269 + // newline before the test body runs, so READY arrives glued to 270 270 + // that prefix. 271 271 + Ok(line) if line.trim_end().ends_with("VICTIM-READY") => break, 272 272 + Ok(_) => continue, 273 273 + Err(_) => { 274 274 + let _ = child.kill(); 275 275 + panic!("victim never became ready (seed={seed})"); 276 276 + } 277 277 + } 278 278 + } 279 279 + 280 280 + std::thread::sleep(Duration::from_millis(delay_ms)); 281 281 + child.kill().expect("kill victim"); 282 282 + child.wait().expect("reap victim"); 283 283 + 284 284 + // Drain everything the victim flushed before death. A final line 285 285 + // without a newline still counts: it was written after its persist 286 286 + // returned, so the edit is acknowledged. 287 287 + let mut acks: usize = 0; 288 288 + while let Ok(line) = rx.recv_timeout(Duration::from_secs(10)) { 289 289 + if let Some(n) = line.trim().strip_prefix("VICTIM-ACK ") { 290 290 + let i: usize = n.parse().expect("ack index"); 291 291 + acks = acks.max(i + 1); 292 292 + } 293 293 + } 294 294 + reader.join().expect("reader thread"); 295 295 + println!("crash gate: seed={seed} killed after {acks} acknowledged ops"); 296 296 + 297 297 + // The graph must reopen cleanly regardless of where the kill landed. 298 298 + let recovered = GraphStorage::open(&dir) 299 299 + .unwrap_or_else(|e| panic!("reopen after kill failed (seed={seed}, acks={acks}): {e}")); 300 300 + let recovered_state = observable(recovered.doc(), &temp_dir(&format!("kill-{seed}-search-a"))); 301 301 + 302 302 + // Accept ref(acks) or ref(acks+1): a persist can become durable in the 303 303 + // instant before its acknowledgment line is flushed. 304 304 + let ref_at_acks = observable( 305 305 + &reference_doc(seed, acks), 306 306 + &temp_dir(&format!("kill-{seed}-search-b")), 307 307 + ); 308 308 + if recovered_state == ref_at_acks { 309 309 + fs::remove_dir_all(&dir).ok(); 310 310 + return; 311 311 + } 312 312 + let ref_past_acks = observable( 313 313 + &reference_doc(seed, acks + 1), 314 314 + &temp_dir(&format!("kill-{seed}-search-c")), 315 315 + ); 316 316 + if recovered_state == ref_past_acks { 317 317 + fs::remove_dir_all(&dir).ok(); 318 318 + return; 319 319 + } 320 320 + 321 321 + let divergence = recovered_state 322 322 + .first_divergence(&ref_at_acks) 323 323 + .unwrap_or_default(); 324 324 + panic!( 325 325 + "recovered graph matches neither {acks} nor {} acknowledged ops \ 326 326 + (seed={seed}, delay={delay_ms}ms)\nvs ref({acks}): {divergence}", 327 327 + acks + 1 328 328 + ); 329 329 + } 330 330 + 331 331 + /// Fast tier: a handful of kills at staggered delays, in the default 332 332 + /// `cargo test --workspace`. 333 333 + #[test] 334 334 + fn killed_process_loses_no_acknowledged_edit() { 335 335 + let base = seed_from_env_or_entropy(); 336 336 + for (i, delay_ms) in [25u64, 60, 120, 220].into_iter().enumerate() { 337 337 + kill_iteration(base.wrapping_add(i as u64), delay_ms); 338 338 + } 339 339 + } 340 340 + 341 341 + /// Deep tier: many kills across a wide delay sweep. 342 342 + /// `cargo test -p trawler-core --test crash_safety -- --ignored`. 343 343 + #[test] 344 344 + #[ignore = "deep tier: many kill iterations, run explicitly"] 345 345 + fn killed_process_deep_sweep() { 346 346 + let base = seed_from_env_or_entropy(); 347 347 + for i in 0..20u64 { 348 348 + // Deterministic-but-varied delays derived from the seed, spanning 349 349 + // "barely started" to "hundreds of persisted ops". 350 350 + let delay_ms = 10 + (base.wrapping_mul(31).wrapping_add(i * 97) % 400); 351 351 + kill_iteration(base.wrapping_add(i), delay_ms); 352 352 + } 353 353 + }

+101

crates/trawler-core/tests/rebuild_equivalence.rs

Reviewed

··· 1 1 + //! Rebuild-equivalence gate (openspec change add-integrity-gates, spec 2 2 + //! "Rebuild equivalence is continuously verified"): after a seeded random 3 3 + //! op stream with indexes maintained incrementally — exactly as the app 4 4 + //! maintains them — rebuilding all derived state from the Loro doc alone 5 5 + //! must answer identically. 6 6 + //! 7 7 + //! Reproduce a failure: every run prints its seed; rerun with 8 8 + //! `TRAWLER_TEST_SEED=<seed> cargo test -p trawler-core --test rebuild_equivalence`. 9 9 + 10 10 + use std::fs; 11 11 + use std::path::PathBuf; 12 12 + 13 13 + use trawler_core::index::GraphIndex; 14 14 + use trawler_core::integrity::{seed_from_env_or_entropy, ObservableState, OpGenerator}; 15 15 + use trawler_core::outline::Outline; 16 16 + use trawler_core::search::SearchIndex; 17 17 + use trawler_core::storage::GraphStorage; 18 18 + 19 19 + fn temp_dir(name: &str) -> PathBuf { 20 20 + let dir = 21 21 + std::env::temp_dir().join(format!("trawler-rebuild-eq-{name}-{}", std::process::id())); 22 22 + let _ = fs::remove_dir_all(&dir); 23 23 + dir 24 24 + } 25 25 + 26 26 + /// Run one equivalence round: `ops` generated ops from `seed`, incremental 27 27 + /// maintenance on one pair of indexes, from-scratch rebuild on another, 28 28 + /// compare observable state. Panics with the seed and first divergence on 29 29 + /// mismatch. 30 30 + fn equivalence_round(seed: u64, ops: usize, label: &str) { 31 31 + println!("rebuild-equivalence [{label}]: seed={seed} ops={ops}"); 32 32 + 33 33 + let graph_dir = temp_dir(&format!("{label}-graph-{seed}")); 34 34 + let storage = GraphStorage::create(&graph_dir).expect("create graph"); 35 35 + let outline = Outline::new(storage.doc()); 36 36 + 37 37 + let incremental_search_dir = temp_dir(&format!("{label}-inc-search-{seed}")); 38 38 + let incremental_search = 39 39 + SearchIndex::open_or_create(&incremental_search_dir, storage.doc()).expect("search index"); 40 40 + let mut incremental_graph = GraphIndex::default(); 41 41 + 42 42 + let mut generator = OpGenerator::new(seed); 43 43 + for i in 0..ops { 44 44 + let outcome = generator.step(&outline); 45 45 + for event in &outcome.events { 46 46 + incremental_graph.apply(storage.doc(), *event); 47 47 + } 48 48 + for &block in &outcome.content_changed { 49 49 + let content = outline.content(block).unwrap_or_default(); 50 50 + incremental_search 51 51 + .upsert_block(block, &content) 52 52 + .unwrap_or_else(|e| panic!("seed={seed} op#{i} upsert failed: {e}")); 53 53 + } 54 54 + for &block in &outcome.deleted { 55 55 + incremental_search 56 56 + .remove_block(block) 57 57 + .unwrap_or_else(|e| panic!("seed={seed} op#{i} remove failed: {e}")); 58 58 + } 59 59 + } 60 60 + 61 61 + // The rebuilt side: derived state reconstructed from the doc alone. 62 62 + let rebuilt_graph = GraphIndex::rebuild(storage.doc()); 63 63 + let rebuilt_search_dir = temp_dir(&format!("{label}-rebuilt-search-{seed}")); 64 64 + let rebuilt_search = 65 65 + SearchIndex::open_or_create(&rebuilt_search_dir, storage.doc()).expect("rebuilt search"); 66 66 + 67 67 + let incremental_state = 68 68 + ObservableState::capture(&outline, &incremental_graph, &incremental_search); 69 69 + let rebuilt_state = ObservableState::capture(&outline, &rebuilt_graph, &rebuilt_search); 70 70 + 71 71 + if let Some(divergence) = incremental_state.first_divergence(&rebuilt_state) { 72 72 + panic!( 73 73 + "incremental derived state diverged from rebuild \ 74 74 + (seed={seed}, ops={ops}, label={label})\n{divergence}" 75 75 + ); 76 76 + } 77 77 + 78 78 + fs::remove_dir_all(&graph_dir).ok(); 79 79 + fs::remove_dir_all(&incremental_search_dir).ok(); 80 80 + fs::remove_dir_all(&rebuilt_search_dir).ok(); 81 81 + } 82 82 + 83 83 + /// Fast tier: runs in the default `cargo test --workspace`. One round — 84 84 + /// each run draws a fresh seed, so coverage accumulates across runs while 85 85 + /// the per-run cost stays around ten seconds (dominated by tantivy's 86 86 + /// per-edit commit, which is the app's real write path). 87 87 + #[test] 88 88 + fn incremental_index_maintenance_equals_rebuild() { 89 89 + let base = seed_from_env_or_entropy(); 90 90 + equivalence_round(base, 250, "fast"); 91 91 + } 92 92 + 93 93 + /// Deep tier: `cargo test -p trawler-core --test rebuild_equivalence -- --ignored`. 94 94 + #[test] 95 95 + #[ignore = "deep tier: large op counts, run explicitly"] 96 96 + fn incremental_index_maintenance_equals_rebuild_deep() { 97 97 + let base = seed_from_env_or_entropy(); 98 98 + for i in 0..4 { 99 99 + equivalence_round(base.wrapping_add(i), 1500, "deep"); 100 100 + } 101 101 + }

+12 -11

openspec/changes/add-integrity-gates/tasks.md

Reviewed

··· 1 1 ## 1. Shared infrastructure (trawler-core, fixtures feature) 2 2 3 3 - - [ ] 1.1 Seeded op generator: mixed weighted stream (create/split/merge/move/indent/outdent/reorder/text-edit-with-refs-and-tags/set-remove-prop/delete) over a fixture graph; seed from env or entropy, always printed; same seed → same stream 4 4 - - [ ] 1.2 Observable-state comparator: capture backlinks per node, tag membership, block properties, full tree traversal (parent/children/order), and search results for a deterministic content-derived term sample; equality with a readable first-divergence diff 3 3 + - [x] 1.1 Seeded op generator: mixed weighted stream (create/split/merge/move/indent/outdent/reorder/text-edit-with-refs-and-tags/set-remove-prop/delete) over a fixture graph; seed from env or entropy, always printed; same seed → same stream 4 4 + - [x] 1.2 Observable-state comparator: capture backlinks per node, tag membership, block properties, full tree traversal (parent/children/order), and search results for a deterministic content-derived term sample; equality with a readable first-divergence diff 5 5 6 6 ## 2. Rebuild-equivalence gate 7 7 8 8 - - [ ] 2.1 Property test: apply N generated ops with incremental index maintenance → comparator snapshot → rebuild GraphIndex + search index from the doc → comparator snapshot → assert equal; fast tier (few seeds, moderate N) in default `cargo test` 9 9 - - [ ] 2.2 `#[ignore]`d deep tier: large N, many seeds, alongside the existing expensive tests 10 10 - - [ ] 2.3 Failure output includes seed, N, and first divergent comparator entry (design D5) 8 8 + - [x] 2.1 Property test: apply N generated ops with incremental index maintenance → comparator snapshot → rebuild GraphIndex + search index from the doc → comparator snapshot → assert equal; fast tier (few seeds, moderate N) in default `cargo test` 9 9 + - [x] 2.2 `#[ignore]`d deep tier: large N, many seeds, alongside the existing expensive tests 10 10 + - [x] 2.3 Failure output includes seed, N, and first divergent comparator entry (design D5) 11 11 + - Gate immediately caught a real bug: `[[Name]]` backlinks went stale when a page with that name was later created/renamed/deleted. Fixed in `GraphIndex` (`refresh_page_names` re-resolution; duplicate names now resolve last-root-wins identically to rebuild) with regression tests `page_lifecycle_retargets_existing_name_references` and `duplicate_page_names_resolve_like_rebuild`. 11 12 12 13 ## 3. Crash-consistency gate 13 14 14 14 - - [ ] 3.1 Victim mode: env-var-gated child path in the test binary that opens a graph dir, loops apply-op → `persist_update` → flush acknowledgment line (op index + digest) to stdout 15 15 - - [ ] 3.2 Kill test: parent spawns victim, hard-kills after a random delay, collects acknowledged prefix, reopens, asserts all acknowledged effects present + doc loads + derived state matches a reference replay of the acknowledged prefix; fast tier with a handful of iterations 16 16 - - [ ] 3.3 `#[ignore]`d deep tier: many kill iterations across varied delays 17 17 - - [ ] 3.4 Confirm kill semantics on Windows are cleanup-free (`Child::kill` = TerminateProcess) and document in the test header 15 15 + - [x] 3.1 Victim mode: env-var-gated child path in the test binary that opens a graph dir, loops apply-op → `persist_update` → flush acknowledgment line (op index) to stdout (digest dropped — the reference replay compares full observable state, which subsumes it) 16 16 + - [x] 3.2 Kill test: parent spawns victim, hard-kills after a random delay, collects acknowledged prefix, reopens, asserts all acknowledged effects present + doc loads + derived state matches a reference replay of the acknowledged prefix (or prefix+1: durable-persist-before-ack-flush window); fast tier kills at 25/60/120/220ms, landing mid-stream (~11–140 acked ops) 17 17 + - [x] 3.3 `#[ignore]`d deep tier: 20 kill iterations across seed-derived delays (10–410ms) 18 18 + - [x] 3.4 Confirm kill semantics on Windows are cleanup-free (`Child::kill` = TerminateProcess) and document in the test header 18 19 19 20 ## 4. Verification and docs 20 21 21 21 - - [ ] 4.1 `cargo clippy --workspace --all-targets -- -D warnings` and `cargo test --workspace` pass; fast tiers add acceptable time to the default run 22 22 - - [ ] 4.2 README development-commands section: document the gates and how to run the deep tiers; note the comparator checklist as the definition of "derived state" 22 22 + - [x] 4.1 `cargo clippy --workspace --all-targets -- -D warnings` and `cargo test --workspace` pass; fast tiers add acceptable time to the default run (equivalence ~13s single round — fresh seed per run accumulates coverage; crash gate ~2s) 23 23 + - [x] 4.2 README development-commands section: document the gates and how to run the deep tiers; note the comparator checklist as the definition of "derived state"