The first version of narwal was correct and slow, and we could not have told you which parts were slow or why. So we measured: a thousand sequential pushes, a thousand concurrent clones, and then the same push test again with two nodes instead of one. Every number below came off the live host at gitwal.io, a single t3.small behind an ALB, with S3 as the log.

Correctness held everywhere. Performance did not, in four specific ways.

What the pushes showed

A thousand pushes, one commit each, advancing the same ref: 1000 landed, no failures, no retries, no orphans. Latency was remarkably flat — p50 788 ms with a standard deviation of 50 ms — and it stayed flat as the log grew. Quartile medians ran 763, 782, 791, 804 ms from the first entry to the thousandth, so the index compare-and-swap does not get more expensive as the entry list gets longer.

One number is worth calling out: 1001 pushes produced exactly 1000 WAL entries. The extra push re-sent a ref that was already current, and the host correctly did nothing. Redelivery is a no-op, under load, without a special case.

Where the clones stopped

The clone storm is where it got interesting. We swept concurrency — 10, 25, 50 workers — and throughput did not move:

workers   throughput   p50        p90
10        3.0/s        2,922 ms   3,465 ms
25        3.2/s        6,730 ms   7,581 ms
50        3.2/s       10,891 ms  14,708 ms

Flat throughput, latency scaling linearly with concurrency. That is a queue in front of a single server, not a saturated machine — and CloudWatch agreed: 38% CPU average, burst credits never depleted. A thousand clones took 310 seconds at 3.2/s, with an implied serialized service time of 310 ms each.

The cause was in our own code. Cache::lock was an exclusive per-repository mutex, and it was held across the freshness sync and the entire upload-pack subprocess. Two people cloning the same repository on the same node took turns. No instance size would have changed that.

What a second node did

The two-node run was meant to test compare-and-swap contention, which one node structurally cannot produce — the per-repo mutex serializes writers before they ever reach S3.

With two nodes racing on one index.json, clients still saw zero CAS conflicts and zero lost updates: retry amplification was 1.0049, because the server retries the CAS internally and almost always wins inside its six attempts. The contention was real, but it showed up as latency, not errors — throughput fell from 3.29/s to 0.58/s, 5.7× worse with twice the hardware, at 11–15% CPU on both boxes. Serialization had simply moved from a local mutex to S3 round trips.

Two other things surfaced only under that racing. Two orphans appeared — packs written to S3 whose index CAS never landed, harmless because the index never names them, but the first we had seen. And one push failed with an HTTP 500.

That 500 was a real bug. S3 answers a raced conditional write with 409 ConditionalRequestConflict, which is not the same as 412 PreconditionFailed: 412 means the precondition was evaluated and did not hold, 409 means the write collided with another conditional write and was cleanly rejected. We only recognized 412. The 409 fell through as an unclassified error and became a 500 — an opaque failure, in a design whose rule is that clients always get guidance.

The cold-rebuild cost

One more, found by accident. A single info/refs probe against a repository with about 950 WAL entries hung past 120 seconds, because the fetch path must bring the cache fresh before serving, and that meant replaying every entry. Worse, it starved the concurrent push loop: throughput dropped from 1.26/s to 0.27/s and one push took 72 seconds. A cold cache in front of a long log is expensive, and it competes with writers.

What shipped

Eleven changes, each pointing at something above.

Cache generations and read leases are the answer to the clone ceiling. A repository is now a sequence of immutable directories: {name}.git is generation 0, and a rebuild that cannot replace the current directory installs a sibling {name}.git.gN. A reader takes a lease on the generation whose recorded etag matches the index it already loaded, and that lease — not a lock — keeps the directory alive. Readers share a generation and never touch the writer mutex, so warm fetches overlap. Writers still take the lock, which refuses new leases while it drains, so readers cannot starve a push. The freshness rule is untouched: no read begins until its generation matches the S3 index.

Streaming transfers under a process budget stop memory scaling with transfer size. Receive-pack streams through validation into index-pack, upload-pack streams straight into the response, and a node-wide budget caps concurrent git children. A saturated node refuses immediately with a retryable busy instead of queueing work it cannot start. That budget and the leases solve different problems: the budget bounds total git processes, the leases remove per-repository serialization.

WAL checkpoints bound the rebuild. An index entry can be marked a checkpoint, and replay starts there instead of at entry zero — the direct answer to the cold-rebuild cost above.

409 is now contention. A conditional-write conflict is classified distinctly from a precondition failure and routed into the existing CAS retry path, surfacing as a retryable busy refusal with guidance rather than a 500.

Fleet membership, rendezvous routing, gossip, and replica prewarm give the fleet a view of itself. Nodes heartbeat into nodes/{id}.json at the bucket root, read with the same expire-on-read rule as per-repo presence. Rendezvous hashing ranks nodes per repository so pushes tend to land where the cache is warm — a preference, never a gate; any node still accepts a push, and S3 remains the arbiter. A publish gossips the new index etag to healthy peers and fans prewarm out to the top-ranked replicas. Both are hints. Neither is a freshness proof: a fetch still reads index.json.

Idle eviction finally reclaims disk. A background sweep removes a repository that no writer holds, that no lease points at, and that has been idle past the interval — the lease count overrides the idle stamp, because a live upload-pack is reading that exact directory.

Provenance and rewind make the log answer the questions a log is for: who moved this ref, what the repository looked like after any entry, and the difference between two of them. Rewind publishes a new entry restoring an earlier state, with a dry run that reports what it would do.

And the fleet is pinned out of us-east-1e, which does not offer the instance type we run.

What is still open

The monitor page reads one S3 object per WAL entry, serially. On a repository with 327 entries that took 86 seconds — 265 ms per entry. The HTML page now caps at the last twenty entries, which hides the cost rather than removing it.

Per-push cost still grows with the number of refs in a repository: every push rewrites the whole index.json and the presence walk grows with the directory. In the concurrent-ref test the per-100-refs interval roughly doubled — 30s, 69s, 136s — which is a quadratic total. There is a ticket.

And the clone-throughput claim above is the honest one to check next. The lease design provably does not take the writer mutex, and its tests hold two leases on one repository at once. Whether that turns into the throughput the serialization analysis predicts is a measurement we have not repeated yet.