tangled-core/knotmirror/db/ at dwn/pdsls-localinfra · willow.sh/tangled-core · Tangled

willow.sh / tangled-core

0

Monorepo for Tangled

0

tangled-core / knotmirror / db /

at dwn/pdsls-localinfra 6 files

Seongmin Lee knotmirror/db: use `DBTX` interface 6w ago

knotmirror/db: use `DBTX` interface Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror/db: use `DBTX` interface Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror: key repos by repo_did + disk migration Lewis: May this revision serve well! <lewis@tangled.org>

3 months ago

migrations_list.go

knotmirror: key repos by repo_did + disk migration Lewis: May this revision serve well! <lewis@tangled.org>

3 months ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago

knotmirror/db: use `DBTX` interface Signed-off-by: Seongmin Lee <git@boltless.me>

1 month ago