Embeddable hybrid graph-vector database in OCaml
reports
+62
+38
reports/10-12-25.md
+38
reports/10-12-25.md
···
1
+
# 10/12/25
2
+
3
+
## initial vector implementation with brute-force knn
4
+
5
+
Got the initial vector storage working, putting directly into lmdb maps with two auxiliary maps for linking vectors to their owning node and tag.
6
+
for now went with 3 map structures:
7
+
8
+
1. `vectors`: `vector_id -> vector_data` (raw float32 bigstring)
9
+
2. `vector_index`: `(node_id, vector_tag_id, vector_id) -> empty` - compound key for range scans getting all vectors for a node, optionally filtered by tag
10
+
3. `vector_owners`: `vector_id -> (node_id, vector_tag_id)` - reverse lookup to find owning node and tag given a vector_id
11
+
12
+
then wrote a super simple brute-force knn implementation that scans the entire vector map, computes euclidean/cosine/dot product distances, and returns the k nearest to a given query vector. right now there's
13
+
no defining the dimensions of a certain vector tag, so query vectors aren't validated for correct length (and you can write whatever you want to the vector map) - need to think about best way to do this! given the rest of the database's focus on type + memory safety with the capnproto stuff we should definitely not leave this as a footgun
14
+
15
+
a lot of this (including the core 3 map structures) will probably need to be refactored when the hnsw slab stuff is implemented, this is probably the next major task.
16
+
17
+
## result type refactoring
18
+
19
+
One big thing wrong with the code was various option types being mixed with result and exceptions/error types. this was a pain to handle and debug and had a ton of holes where error handling was completely missing. switched to using results everywhere and `let*`/`let+` for chaining results, along with some well defined error types and small utility functions.
20
+
21
+
## small fixes
22
+
23
+
1) fixed the O(N) issue described in 28-11-25, now extracting edge src/dst directly from the composite keys with no need for a map and second edge fetch
24
+
2) thought more about it and im 99% sure we won't need a batch api. internally it's all memory mapped so batching isn't important, and over the wire we'll be using promise pipelining and network requests will be batched up at the capnp client level. in certain edge cases (e.g. migrating a full database from one server to another) we might need a batch api, but that's not worth considering for this project i think!
25
+
3) thought more about the hard coded map size, there were 4 options:
26
+
1) large default (e.g. 10GB)
27
+
2) user supplied parameter
28
+
3) auto resize on Map_full
29
+
4) large default + user override
30
+
virtual address space is super cheap, the actual memory is only allocated when needed so can safely go with 4 i think! treating it like an upper bound on the actual database size on disk. resize on Map_full is a little bit of a pain because it can't be done while any transaction is ongoing, wasnt sure if dealing with that was worth the complexity
31
+
32
+
## next stuff!
33
+
34
+
- still need to work on the standalone binary/capnp interface generator
35
+
- then will write a design doc/plan for the hnsw slab stuff. properly formalise the plan and what it should all look like
36
+
- and then implementing hnsw!!
37
+
- after all that is done, can go into the other stuff mentioned in 28-11-25 - i.e. filter adjacency queries by properties, patch operations on properties, and graph algorithms (pathfinding/centrality etc)
38
+
- then can start on the extensions. i think first task should be making the research paper graph usecase with a nice demo, this will hopefully also reveal any errors/bugs/missing features/etc
+24
reports/28-11-25.md
+24
reports/28-11-25.md
···
1
+
2
+
# 28/11/25
3
+
4
+
general TODO list after supervisor meeting, in order of priority. with notes as to whether they're important for the MVP (or project at all).
5
+
6
+
- research - does capnproto just solve batch operations via promise pipelining? either way probably dont need batch operations in the internal library
7
+
- need to test database reopening, persistence across close/open
8
+
- can i just delete the List.filter_map in the scan_adjacency_index???? no more O(N)??
9
+
- standalone binary/capnp interface generator
10
+
- vectors important
11
+
- growing (?) map size ( or just letting user configure )
12
+
- scan pretty important
13
+
- edge deletion cascade
14
+
- patch maybe not important? can just fully read and write now, can optimise later
15
+
- filter adjacency queries by properties - unimportant
16
+
- add big warning for mmap-ed bigstrings, users MUST COPY!@!!
17
+
- refactor stuff to use result - constrain exceptions to scenarios where some invariant has been bypassed. this is worst case exceptions should not be thrown everywhere - https://dev.realworldocaml.org/error-handling.html
18
+
- schema validation on writes, currently registering a schema is completely useless because it's never checked
19
+
- graph algorithms would be cool, pathfinding/centrality etc
20
+
- do deepdive into whether zero-copy actually matters for use cases. based on initial testing/reasoning I think there's only savings when a node is of size > ~2Kb. this super depends on usecase - for small social graphs the added complexity for zero-copy wont be worth it, the overhead will dominate and reduce performance. for graphs with large nodes, e.g. a property storing the full text of a research paper, and/or selective access of properties, or high-frequency access to primitive properties, zero-copy will be a win
21
+
- pretty crucial extension for testing - make the actual research paper graph usecase
22
+
23
+
24
+
flesh out agent memory usecase this is dope