core/knotmirror/xrpc/ at sl/comment · bdbch.com/core · Tangled

bdbch.com / core

0

Monorepo for Tangled

0

core / knotmirror / xrpc /

at sl/comment 1 folder 15 files

oppiliappan knotmirror/xrpc: add missing author info to GitTempGetTree_LastCommit 2mo ago

knotmirror: performant language indexer `git.listLanguages` has been one of the method that fails most often. Opening multiple git repos simultaneously can eaily cause OOM and language indexing itself usually takes super long. So several changes: - use `gitea.CatFileBatch` instead of go-git to avoid OOM - skip files larger than 16KB - sync HEAD ref language stats in knotmirror db - cache language stats info by commits (30d TTL) When syncing HEAD ref language stats, we do indexing on background. KnotMirror maintains internal "repo_stats_update" queue and right after `doResync` is done, enqueue the language stat indexing job so we can pre-index the language stats of HEAD ref. It's ok to spam this queue because all later events will be eventually ignored as we are resolving HEAD lazily. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

git_get_archive.go

knotmirror/xrpc: add missing author info to GitTempGetTree_LastCommit seems to have been lost after the recent changes to use `git cat-file --batch`. Signed-off-by: oppiliappan <me@oppi.li>

2 months ago

git_get_blob.go

knotmirror: prefer `cat-file --buffer` for frequent endpoints ideally we should use this everywhere and completely remove the go-git dependency. go-git consumes a lot of memory for large repos because it loads pack index into heap memory. This is preferred way of other go based git forges like go-git. lots of code are copied from go-git implementation and slightly modified to match tangled's data model. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

git_get_branch.go

knotmirror: temp git xrpc to use repodid Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

knotmirror: temp git xrpc to use repodid Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

git_get_tree.go

knotmirror/xrpc: add missing author info to GitTempGetTree_LastCommit seems to have been lost after the recent changes to use `git cat-file --batch`. Signed-off-by: oppiliappan <me@oppi.li>

2 months ago

git_list_branches.go

knotmirror: temp git xrpc to use repodid Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

git_list_commits.go

knotmirror: temp git xrpc to use repodid Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

git_list_languages.go

*: fmt, remove a couple little frontend bits Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

git_list_tags.go

knotmirror: temp git xrpc to use repodid Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago

knotmirror: add inflight api Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

knotmirror/xrpc: add more prometheus metrics Signed-off-by: Anirudh Oppiliappan <anirudh@tangled.org>

2 months ago

appview/repo: detect readme file from appview Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

knotmirror: prefer `cat-file --buffer` for frequent endpoints ideally we should use this everywhere and completely remove the go-git dependency. go-git consumes a lot of memory for large repos because it loads pack index into heap memory. This is preferred way of other go based git forges like go-git. lots of code are copied from go-git implementation and slightly modified to match tangled's data model. Signed-off-by: Seongmin Lee <git@boltless.me>

2 months ago

sync_request_crawl.go

knotmirror: key repos by repo_did + disk migration Lewis: May this revision serve well! <lewis@tangled.org>

3 months ago

*: fmt, remove a couple little frontend bits Lewis: May this revision serve well! <lewis@tangled.org>

2 months ago