Git backed by object storage because you can't stop me
git
object-storage
kefka
docs: add git smart HTTP protocol implementation plan
Captures the design decisions and rollout plan for adding HTTP smart
protocol support to objgitd alongside the existing git:// transport.
Signed-off-by: Xe Iaso <me@xeiaso.net>
+176
+176
docs/plans/git-http-protocol.md
+176
docs/plans/git-http-protocol.md
···
1
+
# Plan: Git Smart HTTP protocol for objgitd
2
+
3
+
## Context
4
+
5
+
`objgitd` currently serves repositories out of a Tigris/S3-backed billy filesystem
6
+
over the **git:// (TCP)** protocol only (`cmd/objgitd/daemon.go`). git:// is
7
+
unauthenticated, hard to put behind a normal HTTP proxy, and not what most hosts
8
+
(or CI systems) expect. We want clients to be able to `clone`/`fetch`/`push` over
9
+
**HTTP smart protocol** (`https://host/repo.git`), which is the standard transport
10
+
and the natural place to bolt on auth later via middleware.
11
+
12
+
go-git v6's server commands are already HTTP-ready: `transport.UploadPack` and
13
+
`transport.ReceivePack` expose `AdvertiseRefs bool` and `StatelessRPC bool` knobs
14
+
that map exactly onto the two-endpoint smart-HTTP flow, and `transport.AdvertiseRefs`
15
+
emits the `# service=...\n` + flush "smart reply" prefix when told to. So this is
16
+
almost entirely HTTP plumbing around the same backend the git:// daemon already uses.
17
+
18
+
Decisions from the user:
19
+
20
+
- **No auth now** — expose a plain `http.Handler` so auth can be added as wrapping
21
+
middleware later. Push stays gated by the existing global `-allow-push` flag.
22
+
- **Run both transports.** HTTP is the new primary (`-http-bind`, default on);
23
+
git:// becomes opt-in via `-git-bind` (if unset, the git:// listener is not started).
24
+
- **Rename** the existing git:// files to `git_protocol.go` / `git_protocol_test.go`.
25
+
26
+
Code/test conventions follow the `xe-go:xe-go-style` and `xe-go:go-table-driven-tests`
27
+
skills (flagenv→flag, kebab-case flags, slog with `"err"` key, errgroup server
28
+
lifecycle, `t.Helper()` helpers, table-driven subtests with `tt`).
29
+
30
+
## Files
31
+
32
+
| Action | Path | Purpose |
33
+
| ------ | ----------------------------------------------------------------- | --------------------------------------- |
34
+
| Rename | `cmd/objgitd/daemon.go` → `cmd/objgitd/git_protocol.go` | git:// server (unchanged logic) |
35
+
| Rename | `cmd/objgitd/daemon_test.go` → `cmd/objgitd/git_protocol_test.go` | git:// tests (unchanged) |
36
+
| Create | `cmd/objgitd/http.go` | smart-HTTP handler on `*daemon` |
37
+
| Create | `cmd/objgitd/http_test.go` | table-driven HTTP clone/push tests |
38
+
| Edit | `cmd/objgitd/main.go` | flags + errgroup two-listener lifecycle |
39
+
40
+
The `daemon` struct stays the shared backend (`fs`, `loader`, `allowPush`,
41
+
`loadOrInit`). git:// methods live in `git_protocol.go`; HTTP methods in `http.go`.
42
+
Use `git mv` for the renames so history is preserved.
43
+
44
+
## http.go design
45
+
46
+
Make `*daemon` implement `http.Handler`. Dispatch on URL **suffix** (the same way
47
+
`git-http-backend` does), since repo paths are multi-segment and end in `.git`
48
+
(e.g. `/foo/bar.git/info/refs`) — `http.ServeMux` wildcards can't capture a
49
+
variable-depth prefix before a fixed suffix.
50
+
51
+
```go
52
+
func (d *daemon) ServeHTTP(w http.ResponseWriter, r *http.Request) {
53
+
p := r.URL.Path
54
+
switch {
55
+
case r.Method == http.MethodGet && strings.HasSuffix(p, "/info/refs"):
56
+
d.handleInfoRefs(w, r, strings.TrimSuffix(p, "/info/refs"))
57
+
case r.Method == http.MethodPost && strings.HasSuffix(p, "/git-upload-pack"):
58
+
d.handleRPC(w, r, transport.UploadPackService, strings.TrimSuffix(p, "/git-upload-pack"))
59
+
case r.Method == http.MethodPost && strings.HasSuffix(p, "/git-receive-pack"):
60
+
d.handleRPC(w, r, transport.ReceivePackService, strings.TrimSuffix(p, "/git-receive-pack"))
61
+
default:
62
+
http.NotFound(w, r)
63
+
}
64
+
}
65
+
```
66
+
67
+
### Reference discovery — `GET /{repo}/info/refs?service=git-(upload|receive)-pack`
68
+
69
+
- Read `service` from the query; reject anything that isn't the two known services
70
+
with `400`.
71
+
- Resolve the repo, mirroring the git:// switch in `git_protocol.go`:
72
+
- `git-upload-pack`: `d.loader.Load(&url.URL{Path: repo})`; on
73
+
`transport.ErrRepositoryNotFound` → `404`.
74
+
- `git-receive-pack`: if `!d.allowPush` → `403`; else `d.loadOrInit(repo)`
75
+
(preserves git://'s "first push creates the repo" behavior — the advertise GET
76
+
happens before the POST, so it must create here too).
77
+
- Set headers **before** writing the body:
78
+
- `Content-Type: application/x-<service>-advertisement`
79
+
- `Cache-Control: no-cache`
80
+
- Emit the advertisement by calling the matching server command with
81
+
`AdvertiseRefs: true, StatelessRPC: true` (StatelessRPC=true makes
82
+
`AdvertiseRefs` write the `# service=...` smart-reply prefix). Reader is `nil`
83
+
(the advertise path returns before touching it); wrap the writer with
84
+
`ioutil.WriteNopCloser(w)`:
85
+
```go
86
+
transport.UploadPack(r.Context(), st, nil, ioutil.WriteNopCloser(w),
87
+
&transport.UploadPackRequest{AdvertiseRefs: true, StatelessRPC: true,
88
+
GitProtocol: r.Header.Get("Git-Protocol")})
89
+
```
90
+
(and the `ReceivePack` equivalent for receive-pack).
91
+
92
+
### RPC — `POST /{repo}/git-(upload|receive)-pack`
93
+
94
+
- Resolve repo with the same load / loadOrInit / 403 / 404 rules as above.
95
+
- Body handling: if `Content-Encoding: gzip`, wrap `r.Body` in `gzip.NewReader`
96
+
(git clients gzip the upload-pack request). Wrap the (possibly decompressed)
97
+
body in `io.NopCloser` for the `io.ReadCloser` arg.
98
+
- Headers before body: `Content-Type: application/x-<service>-result`,
99
+
`Cache-Control: no-cache`.
100
+
- Dispatch with `StatelessRPC: true` (and `AdvertiseRefs: false`) so the command
101
+
skips advertisement and reads the negotiation straight from the POST body:
102
+
```go
103
+
transport.UploadPack(r.Context(), st, body, ioutil.WriteNopCloser(w),
104
+
&transport.UploadPackRequest{StatelessRPC: true,
105
+
GitProtocol: r.Header.Get("Git-Protocol")})
106
+
```
107
+
- Pre-dispatch failures use real status codes; once the command starts writing,
108
+
errors are only `slog.Error(... "err", err)`-logged (status already sent).
109
+
110
+
### Helpers / notes
111
+
112
+
- Reuse `ioutil` = `github.com/go-git/go-git/v6/utils/ioutil` for `WriteNopCloser`
113
+
— no local nop-closer type needed.
114
+
- `transport.UploadPackService`/`ReceivePackService` are the string constants
115
+
`"git-upload-pack"`/`"git-receive-pack"`, so they match both the `?service=`
116
+
value and the URL suffixes with no conversion friction.
117
+
- Log each request with slog (service, repo path, remote) like the git:// `handle`.
118
+
119
+
## main.go changes
120
+
121
+
- Replace the `bind` flag:
122
+
- `gitBind := flag.String("git-bind", "", "TCP address for the git:// protocol; empty disables it")`
123
+
- `httpBind := flag.String("http-bind", ":8080", "TCP address for the git smart-HTTP protocol; empty disables it")`
124
+
- Keep `-bucket`, `-allow-push`, `-slog-level`.
125
+
- Validate at least one of `gitBind`/`httpBind` is non-empty, else `slog.Error` + exit.
126
+
- Replace the single `d.Serve(ctx, ln)` call with an `errgroup.WithContext(ctx)`
127
+
(`golang.org/x/sync/errgroup`, already in go.mod — promote to direct via
128
+
`go mod tidy`) running, conditionally:
129
+
- **git://**: if `*gitBind != ""`, `net.Listen` then `d.Serve(gCtx, ln)`.
130
+
- **HTTP**: if `*httpBind != ""`, build `srv := &http.Server{Handler: d}`, run
131
+
`srv.Serve(ln)` in one goroutine and a second goroutine that waits on
132
+
`gCtx.Done()` then `srv.Shutdown(context.WithTimeout(...10s))`. Treat
133
+
`http.ErrServerClosed` as clean shutdown.
134
+
- `slog.Info("objgitd listening", "git_bind", *gitBind, "http_bind", *httpBind, "bucket", ...)`.
135
+
136
+
## Tests — http_test.go (table-driven)
137
+
138
+
Same package, so reuse `runGit`/`tryGit` from `git_protocol_test.go`. Drive a real
139
+
git client against `httptest.NewServer(d)` over a `memfs`-backed `daemon`. Guard
140
+
with the existing `exec.LookPath("git")` skip.
141
+
142
+
Cases (one table, `tt`, subtests via `t.Run(tt.name, ...)`):
143
+
144
+
- **push then clone round-trips** (`allowPush: true`): push a seed commit to
145
+
`<ts.URL>/test.git`, assert `/test.git/config` exists in the memfs, clone it
146
+
back, `rev-parse HEAD` is non-empty.
147
+
- **push creates repo on demand** (`allowPush: true`): push to a path that does
148
+
not exist yet succeeds and creates the bare repo (HTTP parity with
149
+
`TestDaemonPushCreatesRepo`).
150
+
- **push rejected when disabled** (`allowPush: false`): push fails and
151
+
`/test.git/config` is never created.
152
+
- **fetch from missing repo 404s**: clone of a nonexistent path fails.
153
+
154
+
Per-case fields: `name`, `allowPush bool`, plus a `wantErr`-style expectation for
155
+
the push/clone outcome. Helper (e.g. `newHTTPServer(t, allowPush) (*httptest.Server, billy.Filesystem)`)
156
+
marked `t.Helper()`. Note: `httptest` serves plain HTTP, so the git client needs
157
+
no TLS config; `GIT_TERMINAL_PROMPT=0` is already set by `tryGit`.
158
+
159
+
## Verification
160
+
161
+
1. `gofmt`/`goimports` clean; `go build ./...`.
162
+
2. `go test ./cmd/objgitd/...` — both git:// and HTTP suites pass (git:// tests
163
+
unchanged after rename).
164
+
3. `go mod tidy` leaves `golang.org/x/sync` as a direct dep, no other churn.
165
+
4. Manual smoke against a real bucket:
166
+
```
167
+
objgitd -bucket $BUCKET -http-bind :8080 -allow-push
168
+
git clone http://localhost:8080/smoke.git # 404/empty until first push
169
+
(cd repo && git push http://localhost:8080/smoke.git main) # creates + pushes
170
+
git clone http://localhost:8080/smoke.git verify # round-trips
171
+
```
172
+
Optionally add `-git-bind :9418` and confirm both transports serve the same repo.
173
+
174
+
```
175
+
176
+
```