Monorepo for Tangled
blog/posts: bobbin intro
Lewis: May this revision serve well! <lewis@tangled.org>
+200
+200
blog/posts/bobbin.md
+200
blog/posts/bobbin.md
···
1
+
---
2
+
atroot: true
3
+
template:
4
+
slug: bobbin
5
+
title: Introducing Bobbin
6
+
subtitle: A diskless, API-only AppView for Tangled
7
+
date: 2026-07-13
8
+
authors:
9
+
- name: Lewis
10
+
email: lewis@tangled.org
11
+
handle: oyster.cafe
12
+
draft: true
13
+
---
14
+
15
+
Hello! It is I, Lewis, with my first Tangled post.
16
+
At Tangled my job involves a lot of Research & Development, by
17
+
which I mean fucking around and finding out. Let me tell you about one
18
+
such finding out, which turned into this new Tangled API.
19
+
20
+
Bobbin is a read-only, API-only
21
+
[AppView](https://atproto.com/guides/glossary#app-view)
22
+
for Tangled. It serves the read side of the `sh.tangled.*`
23
+
[lexicons](https://atproto.com/specs/lexicon)
24
+
over [XRPC](https://atproto.com/specs/xrpc), so a
25
+
client can render git repositories, issues, pulls,
26
+
comments, stars, & follows, without having to ping N
27
+
[PDSes](https://atproto.com/guides/glossary#pds-personal-data-server)
28
+
for their data. It has no permanent storage. Everything it knows is only
29
+
ever in RAM and gets backfilled from upstream every restart.
30
+
31
+
> I, along with many others, find the term "AppView" to be too
32
+
> abstract or perhaps even unfortunately-named. In atproto, datasets
33
+
> are distributed across Personal Data Servers (PDSes) that users each
34
+
> own, so an AppView is a single API through which one can query
35
+
> a dataset, in contrast to all downstream clients each maintaining a
36
+
> connection with every PDS. An AppView is one of the many programs in
37
+
> use to reduce the strain on PDSes, so that a PDS is relatively cheap and
38
+
> resource-efficient to run.
39
+
40
+
Let me list some problems that I find Tangled's AppView to have, such
41
+
that the journey of discovery that led me to Bobbin makes sense:
42
+
43
+
1. Tangled's AppView has no API; it doubles as a web client via SSR.
44
+
2. It is brittle in the sense that it's not able to backfill
45
+
[atproto](https://atproto.com) data easily.
46
+
3. Its architecture from the ground up is inherently built for the kind
47
+
of web program that assumes *it* is the source of truth of data.
48
+
4. Because of all of the above, and the fact that there *is* some
49
+
private data such as email->user [DID](https://atproto.com/specs/did)
50
+
ties, there ought to only be one AppView for Tangled. This means that
51
+
latency to Tangled is vaguely a function of distance from the server
52
+
in Stockholm.
53
+
5. It is hungry for RAM during a RAM crisis.
54
+
55
+
Flipping to the inverse of all of these items would yield me my ideal
56
+
AppView. An API-only, easily-backfillable, easily-distributable,
57
+
easy-on-RAM program.
58
+
59
+
Why "API-only"? Once it's easy to retrieve Tangled data via generic
60
+
XRPC API, more programs can be built on top of it; programs such
61
+
as alternative frontends, third-party clients, one's own targeted
62
+
dashboards, in my case it was to enable a good CLI (though I suppose
63
+
that just comes under alternative frontends). The existing Tangled
64
+
AppView being its own web client means there's nothing extendable to
65
+
build on, unless one is a scraping wizard; one has to figure out their
66
+
own way to go grab the Tangled dataset from the world of PDSes in the
67
+
wild.
68
+
69
+
While I was mulling how to go about creating such a program for fun, I
70
+
have also been having a crisis about where to store state, after I've
71
+
woken up in cold sweats wondering if Tranquil PDS' homegrown embedded
72
+
DB could fail in new mystical ways. How could I avoid this next program
73
+
having to write any durable, fallible state? Besides, AppViews are
74
+
downstream of PDSes, which means the more we try and hold onto state in
75
+
an AppView, the more chance there is that it'll diverge from reality.
76
+
All it takes to have an AppView be wildly incorrect is missing a single
77
+
record, so it is of upmost importance to have an AppView be able to
78
+
reconcile what it thinks it knows with what's actually true.
79
+
80
+
Being the simpleton that I am, I think that there is no better way
81
+
to make a program whose only job is to cache and serve records than
82
+
to make it never touch disk in the first place. I would want it to
83
+
simply backfill all of its data from scratch on every restart. In that
84
+
case, reconciliation with reality is just a restart away and happens
85
+
automatically! Was that feasible? What if such a backfill took a whole
86
+
day every time? That would sure make deployments slow, would it anyhow
87
+
be worth it?
88
+
89
+
My first tests of this concept gave promising results, I
90
+
threw together a little server that could in fact backfill
91
+
all of Tangled's atproto dataset in around 15 minutes, using
92
+
[Hydrant](https://tangled.org/did:plc:6v3ul2ptnqctyxwkz5ti4amn) as its
93
+
aggregator. I decided that an acceptable backfill time would be under
94
+
5 minutes or less, so that it would be feasible to switch out running
95
+
instances on a blue/green deploy within a reasonable warming timeframe.
96
+
97
+
> Hydrant is an aggregator for arbitrary atproto datasets that works
98
+
> by tailing the live [firehose](https://atproto.com/specs/event-stream),
99
+
> backfilling every [repo](https://atproto.com/guides/glossary#data-repo)
100
+
> on the network by pulling their
101
+
> [CARs](https://atproto.com/specs/repository#car-file-serialization),
102
+
> and serving it back over a websocket. All Bobbin has to do is open a
103
+
> connection, ask for cursor 0, and (re)build its index off the resulting
104
+
> barrage.
105
+
>
106
+
> Massive gratitude for
107
+
> [Dawn](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae) for having
108
+
> made Hydrant! Then for hearing my struggles and going and optimizing it
109
+
> also for fun!
110
+
111
+
Once I reached backfill in 5 minutes, I moved the goalpost for fun, that
112
+
the acceptable backfill time should now be 90 seconds or less. This is
113
+
around the number we are approaching today, depending on how warm the
114
+
upstream Hydrant instance is; ranging from 30 seconds in best-case
115
+
scenario to 20 minutes in a disaster where Hydrant is also pulling
116
+
records from PDSes from scratch.
117
+
118
+
> Yes, you too can spin these up & order the entirety of Tangled in 20
119
+
> minutes or less, or you get a refund. (Of €0.)
120
+
121
+
There's a little conundrum that I encountered: what to have the
122
+
program serve while the backfill is still warming? Do I bother
123
+
having the program serve requests while aggregate/linked data is
124
+
possibly incorrect? I think not. So do I have a program serving
125
+
nothing in the backfill time that is possibly 20 minutes at
126
+
worst? I probably could have done that, but instead we connect to
127
+
[Slingshot](https://tangled.org/did:plc:c7mc2fn47ihdihul4vjwsuy3/tree/main/slingshot)
128
+
for point-lookups at least and immediately get accurate
129
+
data in the meantime, even if the program itself hasn't technically seen
130
+
that data yet.
131
+
132
+
> Slingshot is a wonderful edge cache for atproto records
133
+
> & identities. Bobbin asks it for single record and identity
134
+
> lookups, so that a freshly-born Bobbin can answer individual
135
+
> queries accurately before its own index has warmed up. Slingshot
136
+
> is part of the [microcosm](https://www.microcosm.blue/) project,
137
+
> a bundle of community-run atproto infrastructure. Thanks
138
+
> [Fig](https://tangled.org/did:plc:hdhoaan3xa3jiuq4fg4mefid) for making
139
+
> it!
140
+
141
+
The program that was turning into Bobbin was able to hold all of
142
+
Tangled's atproto dataset in around 200MB of RAM, which was good, but
143
+
some compression led to 100MB. There is likely more I could compress
144
+
that is yet to be done. "Wow, such a small amount of data for the entire
145
+
social layer of a whole platform!" you remark, but on the contrary
146
+
remember that this is the *smallest* the dataset will likely ever be
147
+
from today onwards. Space-saving is of the essence.
148
+
149
+
Bobbin uses Slingshot as a record/identity resolver in real time for
150
+
anything it doesn't already hold in RAM or hasn't necessarily received
151
+
from Hydrant. I love Fig's work, don't get me wrong, but my lean on
152
+
Slingshot instead of just writing the pings myself was a cop-out to save
153
+
time; if there's one major architectural change yet to come to Bobbin,
154
+
it's inlining that away such that Bobbin never has to ping Slingshot and
155
+
can figure out faster ways of getting the same data.
156
+
157
+
One would be tempted to throw in an embedded instance of Hydrant to
158
+
their AppView program itself, and have the serving of a particular
159
+
atproto dataset be just a thin wrapper over the Hydrant data; but
160
+
I want it to be easy to spin up possibly hundreds of instances of my
161
+
program. Having hundreds of Hydrants all pinging PDSes all the time
162
+
would be quite wasteful. One Hydrant instance can serve hundreds of
163
+
Bobbins without a sweat.
164
+
165
+
Since we don't have a Hydrant/relay equivalent for Knot-related data,
166
+
Bobbin proxies Knot-related requests straight through to a given knot. I
167
+
think a knotstream aggregator is in order, perhaps even just a plugin to
168
+
Hydrant that has it serving both. I think Bobbin and Knot Mirror should
169
+
merge into one service, but that's just my opinion.
170
+
171
+
With a "stateless" AppView, we don't have to worry about databases and
172
+
their woes: migrations, production massaging when things are slightly
173
+
off compared to the "true" dataset, et cetera. Any time we want to
174
+
change/add functionality, all we have to do is simply restart
175
+
Bobbin and the operation is done. Another benefit is that Bobbin lends
176
+
itself to "cloud-native deployments" as the corpos say. Tangled's own
177
+
flagship instance runs on a CloudFlare worker.
178
+
179
+
All this being said, I'm not necessarily married to the idea that Bobbin
180
+
never touches disk, I think that one cute improvement could be to simply
181
+
dump RAM-stored-data contents as a snapshot once every hour and backfill
182
+
the difference at startup; but for now I think there's headroom to
183
+
continue down this path, considering Bobbin can healthily run with 200MB
184
+
given to it at time of writing. Additionally, my original requirement of
185
+
being easy on RAM is still fulfilled for now, but the moment it becomes
186
+
too heavy is another point at which Bobbin will offload parts of itself
187
+
to disk dynamically during runtime. How do I define "too heavy", you
188
+
ask? I look inside my heart, which tells me that 1GB of RAM is wasteful,
189
+
10GB is unacceptable. Bobbin also must serve users, whose each API
190
+
request puts more pressure on the RAM beyond just storing the Tangled
191
+
dataset.
192
+
193
+
There you have it! Now there's this fun AppView with no disk storage,
194
+
sub-90-second backfill, that serves API requests as a little engine
195
+
that could. With it, anyone can build programs using the Tangled dataset
196
+
more easily.
197
+
198
+
There is an instance of Bobbin on
199
+
[api.tangled.org](https://api.tangled.org), and hopefully you'd find it
200
+
easy to host your own.