gitwal

remote:

Architecture: where a push and a fetch actually go.

remote:

Smart HTTP only. Stock git is the engine. S3 holds the write-ahead log and is the source of truth. The bare repo on local disk is a cache that may be deleted at any moment and rebuilt from the WAL.

01The shape

One process per node. HTTP is an adapter and sequences nothing. Ordering lives in host.

git client https http route · credential host policy · order meta repo.json · ACL wal index.json · entries cache sync · ingest · apply S3 source of truth bare repo disposable replay pkt Origin

02Routing

Path is parsed once. Site pages never touch Host. Everything else parses a flat RepoName first.

RequestGoes toAuth
GET /static landingnone
GET /setupstatic token scriptnone
GET /AGENTS.mdstatic operating manualnone
GET /architecturethis pagenone
GET /blog · /blog/{slug}blog, embedded postsnone
GET /healthzALB probenone
GET /{name}Host::describesame as fetch
GET /{name}/monitorHost::monitorsame as fetch
GET/POST /{name}/settingsadmin JSONtoken on ACL
GET /{name}/info/refs?service=…advertise (upload or receive)per service
POST /{name}/git-upload-packHost::fetchpublic: anon · private: listed hash
POST /{name}/git-receive-packHost::pushwell-formed token, then ACL
/owner/name.gitrejected400, names the flat form

Never 401

No WWW-Authenticate ever leaves this host, so Git will not prompt and will not send URL-embedded Basic. Clients send the credential proactively.

Two independent checks

Discovery and the pack POST authorize separately. Neither trusts the other. A listed push hash implies fetch.

404 is the shrug

A private repo you are not on and a name that does not exist return the same 404 body. Existence is not probeable.

03A push

Discovery is served from index.json without touching disk. Then the POST runs this order.

CAS: compare-and-swap

Every read of index.json comes back with the ETag S3 gave that exact version. A write sends it back as If-Match: <etag>, which means replace the object only if it is still the version I read. If another node published in between, the ETag no longer matches, S3 refuses with 412, and nothing is overwritten. The loser re-reads and retries, up to NARWAL_CAS_ATTEMPTS before it gives up with Busy.

That single conditional write is what makes concurrent pushes safe with no lock, no leader, and no database. Two nodes can race; exactly one wins, and the loser learns it lost instead of silently clobbering the winner.

The first write of an object uses the other half of the same idea: If-None-Match: *, meaning create only if nothing is there. That is how a repository name is claimed and how WAL entry objects stay immutable.

client parse · authz steps 1–5 sync · ingest index-pack pack PUT create-only index CAS commit point pack after this, nothing may fail the push
  1. Parse framing into commands.The only path that returns 400. Everything after this is 200 with band-2 guidance and per-ref ng, because Git will not display a non-2xx POST body.
  2. Enforce NARWAL_MAX_PACK_BYTES.Before the body is trusted.
  3. Require a well-formed token.Shape is nwl_ plus 64 hex. Only the sha256 is ever stored.
  4. Reject ref names that fail git check-ref-format.The name is not claimed by a bad request.
  5. Authorize the existing repo, or defer if the name is unowned.An unclaimed name is decided later, at step 9.
  6. cache.sync against the current index.Loaded with WalStore::load, never the coalescing load_shared. An empty snapshot always rebuilds the bare repo. Failure here is fatal to the request.
  7. cache.ingest, alongside the create-only pack PUT.git index-pack, skipped for an empty pack. Digest is sha256 of the in-memory body. Client faults become ng. If ingest fails after the PUT, that pack object is an orphan.
  8. Ref preconditions against that snapshot.Skipped when the push is an exact redelivery of the current tip. A stale competing tip is ng.
  9. meta.claim with If-None-Match: *.The first valid receive-pack POST wins the name. Repositories start private.
  10. Re-run access against whatever metadata now exists.Same-token retries converge; a different-token loser gets ng.
  11. wal.publish.commit pointMint a ULID entry key, create-only json PUT, pack PUT if not already written, then a bounded index CAS. Duplicate delivery with a matching tip writes no entry. The push is not acknowledged until the pack PUT and this CAS both succeed.
  12. cache.apply.best effortUpdate the local bare repo and geometrically repack. Failure is logged; the push stays successful.
  13. maybe_compact.best effortUnder the same cache lock, when the published index has at least NARWAL_COMPACT_AFTER entries.
  14. Presence write.best effortSnapshotted from pack file lengths and the index etag, then spawned and not awaited. A failed presence write never fails Git.

04A fetch or clone

Two requests, each authorized on its own. Neither writes to the WAL.

GET info/refs service=upload-pack POST upload-pack wants · haves access repo.json cache.sync etag compare upload-pack under the lock presence, not awaited packfile
  1. GET info/refs?service=git-upload-pack.Load repo.json, run access for fetch, cache.sync, queue a presence write, then git upload-pack --http-backend-info-refs. The lock is held through sync and upload-pack.
  2. POST git-upload-pack.Authorize from headers before reading anything, read at most NARWAL_MAX_FETCH_BYTES+1, reauthorize, cache.sync, queue presence, then git upload-pack --stateless-rpc with GIT_PROTOCOL copied from the Git-Protocol header.

Public fetch is anonymous. Private fetch needs a listed hash. Unauthorized or malformed private fetch is the same 404 as a missing name; malformed credentials on a public name are 403. Upload-pack is never served until the cache is fresh against the S3 index ETag.

05Cache sync

The bare repo under {NARWAL_DATA}/repos/{name}.git is disposable. narwal.etag next to it records which index built it.

index handle etag from S3 etag == narwal.etag? serve disk HEAD must be live delete · replay every entry, digest-checked empty packs still update refs repack · midx bitmaps, under the lock etag written last hit miss, or IndexHandle::empty()

A miss deletes the directory and replays the whole WAL rather than patching it, so leftover ingest from a rejected push cannot survive. The rebuilt directory is renamed into place after the etag file is written. Compacted local packs are never written back to S3.

06Compaction

The node that just published is the node that compacts. No scheduler, no elected primary.

  1. Decide.NARWAL_COMPACT_AFTER entries or more, at least two, at least one ref. 0 disables it.
  2. Build and verify a pack.git pack-objects --revs --stdout over the tips, then index it into a throwaway {name}.git.verify with --strict and walk rev-list --objects from every tip. A thin or truncated pack fails here, before anything references it.
  3. CAS the prefix away.One fresh entry that creates every tip from the zero oid, replacing the compacted prefix and keeping everything published since. Replaced keys are appended to superseded, so the monitor does not read them as orphans.
  4. Apply locally.So this node's narwal.etag matches the index it just wrote.

A compaction that loses its CAS, or whose prefix moved, writes no index at all. It never drops a concurrently published entry.

07Where this lives on AWS

Terraform in infra/. Everything above is one binary on an EC2 node; the only thing two nodes share is the bucket.

default VPC · public subnets · us-east-1 Auto Scaling group · ELB health git client Route 53 name ALB :443 ACM · TLS 1.3 :80 redirects idle timeout 4000s health check /healthz EC2 node · AL2023 narwal.service on :8080 http · host · meta · wal · cache EBS gp3 · /var/lib/narwal warm cache EC2 node · AL2023 identical binary its own disk, its own etag cold until it syncs S3 bucket versioned · SSE-S3 public access blocked repos/{name}/index.json wal/{ulid}.pack · .json the narwal binary 443 IAM role, repos/* only
WhatRuns onNotes
http · host · meta · wal · cacheone narwal process per EC2 nodesystemd unit, Restart=always, listens on :8080
bare repo cacheEBS gp3 root volume, per nodeNARWAL_DATA=/var/lib/narwal. Deleted on instance refresh; rebuilt from the WAL
WAL, repo.json, presenceone versioned S3 bucketThe only shared state. SSE-S3, public access blocked, incomplete multipart aborted after a day
TLS and routingALB in the default VPC's public subnetsACM cert, Route 53 apex and www, HTTP redirected to HTTPS, health check /healthz
capacityAuto Scaling group + launch templateELB health checks, rolling refresh at 100% min healthy, so max_size is one above desired
the deployed binaryan object in the same bucketUser data pulls it at boot; a new filemd5 is a new launch template version
node credentialsIAM instance profileGetObject/PutObject on repos/*, ListBucket conditioned on that prefix, plus SSM for access

Nodes share nothing

No RDS, no EFS, no ElastiCache. Two nodes agree only because they both CAS the same index.json, and each one notices a new etag on its next request.

No bootstrap secret

There is no Secrets Manager token to hand a node. A node needs the bucket and nothing else, so replacing one is not a credential event.

Idle timeout 4000s

A large clone can hold one HTTP response open for a long time. The ALB default would cut it; this one does not.

us-east-1e is excluded

It does not offer the node instance type, and an ASG spanning every default subnet would eventually try to launch there and stall a refresh.

remote:

Watch it happen on a repo you own: /<name>/monitor shows the S3 index etag against this node's cache, every WAL entry with its size and sha256, and which nodes are warm.

curl -fsSL https://gitwal.io/setup | bash
git push https://gitwal.io/my-project.git main
open https://gitwal.io/my-project/monitor
remote:

Full operating manual: /AGENTS.md. Project notes: blog.