Shipping Pages: the deadlock we almost deployed

How it was built

Each agent owned an explicit set of files, and no two agents wrote the same one. A foundation pass (schema, shared types, configuration) landed first, so nobody built on a moving base. We have written about the rails we put around a coding agent; this was their largest test so far.

Writing the code was never the constraint. Knowing whether it worked was. So the second half of the project was verification, and wherever we could, each layer was run by someone, or something, other than the author of the code under test.

What each layer caught

The two browser bugs are the kind a unit test walks straight past. Opening one view-settings sheet blanked the whole page. And table copy and paste had a race: a refetch that arrived out of order blanked a cell mid-operation, and the paste then wrote nulls where values should have been. Both needed a real browser, real timing and a real server.

The deadlock we almost deployed

Dhaga has no migration tool. The schema ships as idempotent DDL that each instance checks at startup (background here). When it changes, the first instance to boot replays it under a Postgres advisory lock. In the enterprise build, that replay turns on row-level security for every tenant table:

ALTER TABLE ... ENABLE ROW LEVEL SECURITY;

It runs unconditionally because it is idempotent: on a table that already has RLS it changes nothing. Nothing in the data, that is. It still takes an ACCESS EXCLUSIVE lock on the table, every time.

Pages added indexes to tables that already held rows. A plain CREATE INDEX blocks writes while it builds, so these used CREATE INDEX CONCURRENTLY, and to make sure they existed they were built at boot, right after the replay.

Now start two instances at once, which is what a normal deploy does:

1. Instance A is replaying. It holds ACCESS EXCLUSIVE locks on the tables it has passed, and moves on to the next.
2. Instance B is building one of the new indexes on a table A has not reached yet. A concurrent build holds a lock on its table throughout, and before it finishes it waits for older transactions in the database to end. A's replay is one of them.
3. A waits for B's lock. B waits for A's transaction. Postgres detects the cycle and aborts one of them.

If the replay lost, it retried a few times, and when it ran out of attempts the schema bootstrap cached the failure. Every request on that instance that needed the tenant schema then failed until the instance was recycled.

The fix had two parts:

• The concurrent builds left boot. A background job that already runs on a schedule builds them, one statement at a time, outside any transaction. It skips its run while a replay holds the advisory lock.
• A failed bootstrap is retried, not kept. Only success is cached now, and a failure is tried again a few seconds later at the earliest.

Why did our tests miss it? The runner's tests used a mocked connection pool. They proved the statements ran in order and that a lock timeout was retried. They could not show what two processes do to each other's locks. Tests written by the author of a mechanism encode the author's model of it, and concurrency is exactly what that model leaves out.

A preview is not a sandbox

A pull request's preview build runs the same boot path as production, including applying schema changes, so a change can reach real infrastructure before anyone has reviewed it. We could not make that go away cheaply, so we made it safe:

• DDL is strictly additive, idempotent and append-only. Nothing a preview runs can drop, rename or rewrite, and a new chunk goes at the end.
• After each preview, we read the live database back inside a read-only transaction and check the shape we expected is there. The check itself cannot write.

BEGIN TRANSACTION READ ONLY;
SELECT column_name, data_type FROM information_schema.columns WHERE table_name = '...';
ROLLBACK;

The build that ran out of memory

Vercel's builder had 2 cores and 8 GB. Next's built-in type check runs after Turbopack compiles, while the Turbopack worker is still writing its persistent cache. On a codebase this size the two together passed 8 GB, and the container killed the build without an error from either tool.

• Type-check first, in its own process. next typegen writes the route types, tsc checks everything, and only then does next build run, with Next's own check skipped on that path alone. A bare next build still checks itself.
• Turn off Turbopack's persistent build cache on Vercel. It had grown past the size Vercel keeps, so every build paid to write a cache the next one threw away. Without it, Vercel's own build cache fits its limit again.

Postgres under row-level security

Every page read goes through a predicate for "pages this user can read". We measured on a local Postgres with synthetic data, connected as the app's own role with RLS forced, because the superuser's query plan is a different plan.

An OR with a subquery arm scans everyone. One arm of the access predicate was id IN (SELECT ...). Inside an OR, Postgres cannot turn that into a join or an index lookup, so it checked the subquery against every row of the table, every tenant's, before RLS discarded most of them. Written as = ANY (ARRAY (SELECT ...)), the subquery runs once and becomes an array of ids the planner can combine with the other arms' indexes.

-- before: checked row by row, no index for this arm
... OR id IN (SELECT ...)
-- after: computed once, then an index lookup
... OR id = ANY (ARRAY (SELECT ...))

Under RLS, only leakproof conditions become index conditions. Postgres will not run a query's own condition ahead of the policy unless every function in it is marked leakproof, because an unvetted function could reveal hidden rows, in an error message for example. lower(), LIKE, ILIKE, the trigram operators and even COALESCE are not leakproof, so an expression index on lower(...) cannot serve a tenant's query. We measured one before dropping the idea.

A row compared with itself. A column reference was rendered bare inside a subquery, so it bound to the inner table instead of the outer one, and the condition compared a row with itself. It was always false and no index could answer it, so every page open scanned the whole table. Qualifying the reference took a page open from 91 ms to 3 ms.

These are medians of six runs on local synthetic data, noisy by about 25% either way, and production plans may differ. Only two new indexes went in, both partial; the rest was rewriting queries.

Who has an account here?

An invite form's response showed whether an email address already had a Dhaga account. That is account enumeration. Now every invitation gets the same response, and it stays pending, granting no access, until it is accepted, whether or not the person has an account.

The day a cleanup moved a repository

Not a Pages bug, but it happened during the build. To free disk space, a cleanup moved a package manager's store to another drive with robocopy /MOVE. That store keeps directory junctions back into every project that uses it, and robocopy followed them. It moved a whole repository and three other projects to the other drive, removing each source after copying it.

Nothing was lost: the copy was complete, and we recovered from it and from git backups. What we changed:

• Never /MOVE, /MIR or /PURGE on a tree that may contain links without /XJ (skip junctions) and a dry run with /L first.
• Relocate a cache through the tool's own configuration, not by moving its directory.
• Keep backup pushes on hidden refs. A push to refs/backup/... is off the machine, and because it is not a branch it starts no preview build, which, as above, would apply DDL to production.
• A PreToolUse hook now blocks the dangerous forms before an agent can run them.

Process that held up

• Hard stops before anything irreversible. A real email to real users had a strict dry run, a quota check, and a pause for a person to confirm the recipient count.
• A test must fail when its fix is reverted. If it still passes, it is testing its own mock.
• Status reports have to be checkable. Twice, an agent reported something that was not true. A cross-check against the real state caught it both times.

Three takeaways

1. The author's tests prove the author's model. Ask someone else to read the mechanism with one question: what if two of these run at once?
2. Know which environments touch real state. A preview that can change real infrastructure changes what "safe to push" means. Make those writes additive, and check them with something that cannot write.
3. Measure as the role your app uses. Under row-level security, the plan you see as a superuser is not the plan your users get.