The staging environment lie: why PR previews replaced ours

Every stack we inherit has one. staging.clientname.be, or test., or — in the honest shops — staging2.. It was stood up in the first month of the project as the place where things get checked before they go live, and by month eighteen it's a machine everybody has an opinion about and nobody trusts. Someone asks "is that fix on staging?" in a channel, three people answer differently, and the actual verification happens by pasting a screenshot from a local dev server.

We stopped building them. Not from a principle we read somewhere — from noticing that across a decade of client projects the staging conversation is always the same three sentences, and that all three are symptoms of one design decision: there is exactly one of it, and it is shared.

Staging fails in three ways, and they compound

It's blocked. Two features are in flight, three if the client is paying for it. Only one can be on staging. So the second developer waits, or deploys over the first, or — the realistic outcome — merges both to a staging branch that exists nowhere else in the workflow and now has to be kept in sync with main by hand. The moment you have a branch whose only purpose is "the thing that's on staging," you have invented a second integration point that nobody tests and everybody has to remember.

It's stale. The database was restored from production eight months ago and has been mutated by every test since. The product catalogue is half-real. The user your client logs in with has a cart from March. Nobody dares refresh it from production, because refreshing it destroys whatever test data the current round of UAT depends on. So staging drifts, and every bug found there gets triaged twice: once for the bug, once for "is that real or is it just staging?"

It's unattributable. Ask what commit is running on it. On a good day someone knows. On a normal day the answer is a range — "everything up to Tuesday, plus Jasper's fix, unless the deploy failed." A green check on an environment whose contents you can't name is not evidence of anything, and once a team learns that, they stop looking at it. That's the real cost. Not that staging is broken — that people quietly route around it and the last gate before production becomes vibes.

These compound, because each one's obvious fix makes another worse. Refreshing the database fixes staleness and destroys the UAT dataset. Locking staging to one branch fixes attribution and doubles the queue. Adding staging2 fixes contention for exactly one more person, and now you have two servers rotting instead of one. Every one of these fixes is really a fix for the environment being shared, applied to something that isn't. It's worth naming the pattern out loud, because it's the thing that decides the architecture: contention, drift and provenance are not three problems, they're one problem counted three times. One environment, many changes.

"Preview environment" means three different things

Before the argument for previews, some precision, because the word covers three setups and only one of them replaces staging.

A preview build. The site is rendered from the branch and served statically. This is what most hosting platforms hand you for free, and for marketing pages that don't need a server it's genuinely enough — the pages are the same for every visitor, so a static render is the product. It stops being enough the second anything writes.

A preview app on the shared database. Each PR gets its own app pods, all pointed at the one staging database. This is the setup that looks like it solved the problem and didn't. Two PRs both run a migration against the same schema, and now the answer to "why does PR #412 crash" is "because PR #418 dropped a column." You've moved the contention from the application tier to the data tier, where it's harder to see and much more annoying.

A full instance. Its own pods, its own database seeded from a known snapshot, its own workers and queues, its own URL, its own credentials for anything outside itself. This is the one that replaces staging, because it's the only one where the answer to "what's running here?" is a commit sha and nothing else.

The difference between the second and third is the entire engineering problem, and it lives almost entirely in one place.

The database is the entire problem

The stateless half is a solved and boring exercise. A namespace, a deployment, a service, an ingress with a wildcard certificate, an image tagged with the commit sha. If you already run applications on Kubernetes, per-PR app pods are an afternoon and some templating. Nobody's preview strategy fails there.

It fails on the database, and there are three ways to get one, which we'd rank in this order:

Migrate an empty database. Fast, cheap, and worth very little. An empty schema proves your migrations run; it proves nothing about the feature, because there's nothing to look at. Reviewers open the preview, see an empty list, and go back to reading the diff. Fine as a smoke test in CI. Not an environment.

Restore a dump per PR. Correct and honest, and for a small database it's the right answer — a 500 MB Postgres dump restores in under a minute and you're done. It stops scaling somewhere around the point where the restore takes longer than the reviewer's attention span, which for a real ERP database is well before you'd like.

Copy a template database. This is the one that made previews practical for us. Postgres will copy an existing database at the file level, which is enormously faster than replaying a logical dump:

CREATE DATABASE "pr_412" TEMPLATE "seed_2026_09" OWNER app;

A 20 GB database that takes eleven minutes to pg_restore copies in well under a minute this way, because Postgres isn't parsing SQL and rebuilding indexes — it's copying files and writing a couple of catalogue rows. The one operational catch is worth knowing before it bites you: no other session may be connected to the template while the copy runs, so the nightly job that rebuilds seed_2026_09 has to own it exclusively, and anything that idles a connection against it will make preview creation fail with a message that reads like a permissions error and isn't.

On Postgres running in the cluster this fits the existing shape: previews bootstrap from a snapshot of the seed cluster rather than a live restore, and the nightly rebuild becomes a scheduled job like any other. Which produces a side effect we didn't plan and now consider one of the better arguments for the whole approach: the seed rebuild is a restore drill that runs every night and that nobody has to remember to do. We make a monthly ritual of restore drills precisely because a backup nobody has restored is a hypothesis. Wire your preview seed to the backup pipeline and the hypothesis gets tested daily, by a robot, with a failure that pages someone — because the first developer to open a PR the next morning gets no database.

If your database is genuinely too large to copy per PR — hundreds of gigabytes of document attachments, mostly — the escape hatch is a CSI volume snapshot rather than a logical copy, or a seed built from a subset: recent records, one company, one warehouse. A subset is a real trade-off and we'd rather make it deliberately than discover it as a cost surprise.

The seed is a product, and it needs an owner

The moment you build this, you learn that the seed dataset is the most valuable artefact in the pipeline and the one nobody has been assigned.

Two rules we hold to.

Anonymise in the pipeline that builds the seed, never in the preview. A preview URL is guessable, gets pasted into a GitHub comment, and lives on a domain your client's staff will happily open on a train. Real names, real e-mail addresses and real VAT numbers do not belong there, and "we scrub it after restore" describes a window during which the real data was live. Windows like that become permanent the first time a scrub step fails quietly.

UPDATE res_partner SET
  name   = 'Partner ' || id,
  email  = 'seed+' || id || '@example.invalid',
  phone  = NULL,
  mobile = NULL,
  vat    = NULL
WHERE id NOT IN (SELECT partner_id FROM seed_keep_real);

The seed_keep_real exception exists because a handful of records genuinely have to survive — your own company, the test bank account, the two suppliers whose EDI codes the integration matches on. Make that list explicit and reviewable rather than encoding it as a WHERE id > 500 that someone will not understand in a year. The same instinct that keeps an ERP backend off the public internet entirely applies here: preview environments are the most exposed copy of your client's data you will ever create, so the data in them should be worth nothing.

The seed must contain the awkward records. A dataset with three clean orders and a well-behaved customer proves nothing, and that's how a preview becomes theatre. Ours carry the cancelled order with a partial refund, the customer with a Dutch delivery address and a Belgian VAT number, the product with no image, the invoice with a credit note against it, and the one name with an apostrophe in it that breaks a template roughly once a year. Every time production produces a shape that broke something, it earns a permanent seat in the seed. That list is the accumulated memory of the project, and it's worth more than any individual test in the suite.

Everything that talks to the outside world

The other half of the work is stopping forty environments from doing forty times as much damage as one.

The rule we apply without exception: every outbound integration in a preview is either a sandbox credential or a black hole, and the failure mode of a missing configuration value is "refuse to start," not "fall back to the default." Fail closed. A preview that boots with an unset SMTP host and quietly picks up the production relay from a base config is not a hypothetical; it's an afternoon of apologies.

• E-mail goes to a catch-all mailbox, never to the address on the record. It's the one integration where the sandbox is more useful than the real thing anyway — every message a preview would have sent, in one inbox, reviewable in order. It also keeps your sending reputation out of it, which matters when the transactional platform and the marketing platform are deliberately separate systems and only one of them has a reputation you can't rebuild.
• SMS is dropped or routed to one internal number. This isn't only about the cost per message: an appointment confirmation that answers back sent from a preview to a real customer is a support incident, and it will arrive at 03:00 because a test fixture used now().
• Payments use the provider's test keys, and the preview's own configuration must not be able to hold a live key at all. Scope the secret path, don't rely on the value.
• Inbound webhooks are the genuinely hard one, and we won't pretend otherwise. A payment provider will not deliver to forty ephemeral URLs. Two workable answers: replay recorded payloads into the preview from a fixture set — which is what we do most of the time, and it's also just a better test — or route by a reference prefix, where the preview registers pr412- as a prefix on a shared receiver that forwards matching events on. The second is more faithful and more machinery; pick it only when the integration under review is the webhook.

Credentials for all of this come from the same mechanism as everything else — a preview namespace gets an ExternalSecret pointed at a preview/ path, never the production one. The secrets flow we trust does the enforcement here: the preview service account simply cannot read the production path, so "someone copied the wrong value into an env var" stops being a class of mistake.

Teardown is the feature, not the cleanup

The reason previews don't rot is not that they're new. It's that they die.

The environment's lifecycle is the pull request's lifecycle, and that has to be literal: opened creates it, pushed updates it, closed or merged destroys it — namespace, database, DNS, secrets, all of it. That single property is what fixes all three of staging's failures at once, and it's why "we'll clean them up periodically" is not a smaller version of this idea. It's the old idea again. A preview stack without automatic teardown becomes a staging server with forty of them, and the second-order problem — which of these is still needed? — is worse than the one you started with, because now nobody owns any of them.

On this website's own pipeline the switch is a label, so previews are opt-in per PR rather than a tax on every dependency bump:

case ",$CI_COMMIT_PULL_REQUEST_LABELS," in
  *,preview,*) ;;
  *) echo "PR has no 'preview' label — no instance to announce, skipping"; exit 0 ;;
esac
 
PREVIEW_URL="https://$CI_COMMIT_PULL_REQUEST-web.apps.codeagency.cloud"

The PR number is the whole identity: the namespace, the hostname, the database name and the teardown key all derive from it, which means there's no state anywhere mapping environments to branches that can drift out of sync. The pipeline then posts the URL as a comment on the PR itself and pings the channel, because an environment nobody is told about is an environment nobody opens. Everything about how that pipeline is built — self-hosted runners in the cluster, a shared remote cache, no per-minute billing — is described in the CI setup behind this site; the preview instance is the part of it that changed how the work actually gets reviewed.

One belt-and-braces detail from experience: also put a TTL on the namespace, independent of the PR state. Seven days without a deploy and it goes, whatever GitHub thinks. The closed webhook will be missed eventually — an outage, a rate limit, a PR closed while the runner was down — and the cost of a missed teardown compounds silently until someone reads a bill.

The arithmetic, honestly

This is not free, and the pitch that it is comes from vendors who bill for it.

The recurring cost is idle capacity. A preview environment is used for perhaps forty minutes across the three days it exists and sits idle for the rest, which makes it exactly the workload where over-requesting hurts most, because there are n of them at once. Previews should be requests-light and burstable, and they're the best possible argument for right-sizing requests and limits from real numbers rather than copying production's values — a preview that reserves production's memory footprint will happily eat a node for a feature branch nobody has opened yet.

Storage is where the honest number lives. A 20 GB seed copied per PR, times a dozen open pull requests, is 240 GB of block storage that exists because your team is busy. That's a real line on a real invoice, and it's the constraint that decides whether you copy the whole database or a subset. It's also the number that tells you to cap concurrent previews — we do, per project, and the cap has never actually been hit, which is the correct outcome for a safety limit.

Set against that, the shared staging server was not free either. It was an always-on node, a database, a certificate, and — the part that never appears on the invoice — a recurring share of a senior person's attention every time it broke, drifted or blocked someone. Previews shift spend from a permanent fixed cost to a variable one that tracks how much work is actually in flight, which is the right shape for an agency and, in our experience, roughly a wash on the infrastructure bill.

What previews don't fix

They are not a test suite. A preview environment tells you whether the software behaves like software; it says nothing about whether it stays correct next month. That's what a real test suite with fixtures and CI gates is for, and previews make it more important rather than less, because a reviewer clicking happily through a working feature is exactly the kind of positive signal that lets a regression through. Both run on the same PR, and they catch different things: the suite catches the thing you thought of, the preview catches the thing you didn't.

They don't validate a deploy either. A preview proves the image boots and the migration applies against seed data. It does not prove the rollout is safe against a live database with real concurrency and a schema two versions behind — that's a property of the deploy strategy you chose, and no preview will tell you your rolling update needed to be a blue-green.

And they don't replace watching production. Previews shorten the loop before a merge; the metrics and alerts on the other side of it are still the only thing that knows what real traffic does to the change.

The one long-lived environment we kept

Because the honest version of this argument has an exception in it.

Two situations still deserve a permanent, stable environment. The first is user acceptance and training before a go-live: a client's staff need weeks on a fixed URL, building muscle memory in data that persists between sessions, and an environment that vanishes when a PR merges is precisely the wrong shape for that. The second is a third-party integrator who needs a stable endpoint to develop against for months — a logistics partner, an accounting package, anyone who needs to hand their IP allowlist a hostname that will still exist in March.

What makes those work is not that they're long-lived. It's that each has one purpose, one named owner, and a documented lifespan — the training environment is scheduled to die two weeks after go-live, and it does. That's the actual lesson from a decade of rotting staging servers: the box was never the problem. Making one box be the review environment, the demo environment, the integration environment and the training environment simultaneously was the problem, and no amount of discipline survives four jobs and one hostname.

The short version

One shared staging server has one environment and many changes, and every symptom people complain about — the queue, the drift, the untrusted green check — follows from that arithmetic rather than from anyone's discipline. Give each change its own environment and the symptoms go away, not because the tooling is better but because the arithmetic changed.

The work is not in the app tier; it's in the database copy, a seed dataset somebody owns and keeps ugly on purpose, integrations that fail closed rather than falling back to production, and teardown that is automatic and belt-and-braced with a TTL. Get those four right and the review conversation moves from "looks fine in the diff" to "I clicked through it," which is a different and much better sentence.

This is how we build and host every custom application we ship, on our own clusters, including this website — the post you're reading was reviewed on an instance of its own before it merged. If you're currently maintaining a staging server that nobody trusts, the first move isn't the Kubernetes work. It's writing down what that server is actually for, and noticing how many answers you get.