Expired, then deleted: the credential dates nobody is watching

The job ran every night at half past two and exited zero every night at half past two. Its log said it had started, said it had finished, and in between said nothing at all, which is what a loop over an empty result set looks like. Six weeks later a bookkeeper asked why the accounting platform had no invoices since July, three days before a VAT deadline.

What we found was worse than a wrong password. The API key that export authenticated with had been created with an expiration date — set a year earlier by someone being careful — and when the date passed, a housekeeping cron in Odoo did exactly what it is written to do and deleted the record. The key did not start returning "expired". It started returning "no such key", which is indistinguishable from "somebody typo'd the secret", which is indistinguishable from "this was never provisioned in this environment". The audit trail for a credential that no longer exists is an empty query.

Nothing in the stack was down. Uptime was 100% for the month. Every dashboard was green, because every dashboard was measuring whether things answered, and everything answered beautifully.

Expiry is an outage you scheduled and then forgot

A credential with a date on it is a timer you wound up and put in a drawer. That is genuinely a security control — it bounds how long a leaked key is useful — but it is only a control if somebody is going to act on it. Set an expiry with no renewal process behind it and you have not reduced risk; you have converted a security risk into an availability risk, and moved the incident to a date of your choosing that nobody wrote down.

The reason these outages hurt more than their size deserves is that they are unattributable. Nothing changed. No deploy went out, no config was edited, no dependency was bumped. The last commit touching that code path is from March, and the change that broke production was made by the passage of time. Every debugging instinct a team has is trained on "what did we change?", and here the honest answer is nothing, which sends people looking in the wrong place for a day.

It also means they cluster. A key provisioned during a project kickoff, a certificate issued the same week, a signing profile generated the same month — all with one-year lifetimes, all landing within a fortnight of each other, twelve months later, usually while the person who created them is on holiday.

Three ways a credential dies, and only one of them is loud

It fails hard. Every call returns 401, the error rate goes vertical, the error tracker lights up, somebody is looking at it inside ten minutes. This is the good case and it is the one people design for, which is why it is almost never the one that costs money.

It fails softly. The credential sits on a path that is already wrapped in a try/catch, usually for a good reason — a visitor should not see an ERP outage, a batch should not abort halfway and leave the run half-applied. So the job exits zero, the queue accepts, the webhook returns 200, and the failure is one warning line in a log nobody tails. This is the expensive one. The system is not down; it is lying, and it will keep lying until a human notices a second-order effect — a report with no rows, a bookkeeper three days from a deadline.

It disappears. Rarer, much nastier, and increasingly common as platforms clean up after themselves. Odoo's vacuum job removes lapsed user API keys. A rotating OAuth refresh token is invalidated the moment its successor is issued, so one failed refresh, retried, kills the grant permanently. Google's refresh tokens go stale if an integration sits unused for six months — which is exactly what a quarterly export does. Kubernetes projected service-account tokens are bound and short-lived by design, and the kubelet refreshes them for you right up until something reads one off disk and caches it in a client that was written before that was true.

The soft failure is the one worth engineering against, because it is the only one where the gap between "broken" and "noticed" is measured in weeks. Everything below exists to close that gap.

The register: expiry as data, in git, not in somebody's head

Before any monitoring, an inventory. We keep it in the repository next to the code it authenticates, because a wiki page describing credentials rots at exactly the same rate as the credentials do.

The rule for what goes in it: anything that can stop working while the code stays the same. That list is longer than most teams expect. From a real audit of one mid-sized client stack:

That last row is not a joke. We have been called in twice for "the whole platform is down" incidents that were an unpaid domain renewal on a mailbox nobody reads.

The register itself is boring on purpose — a typed file, reviewed like any other change:

export type Credential = {
  /** Stable id; becomes the metric label, so never rename it casually. */
  id: string
  system: string
  /** Where the value lives. Never the value itself. */
  source: string
  /** ISO date. `null` = deliberately non-expiring, see `rationale`. */
  expiresOn: string | null
  /** A person, not a team alias — teams do not renew certificates. */
  owner: string
  /** How to renew it, step by step. A link, not a sentence. */
  runbook: string
  /** What breaks, in business terms, so triage can skip a discovery phase. */
  blastRadius: string
}
 
export const credentials: Credential[] = [
  {
    id: "odoo-api-key-export",
    system: "Odoo (production)",
    source: "infisical:/integrations#ODOO_API_KEY",
    expiresOn: null, // deliberate: see ADR — scoped user, no expiry, audited
    owner: "fabio",
    runbook: "docs/runbooks/odoo-api-key.md",
    blastRadius: "Nightly accounting export writes nothing, and exits 0.",
  },
  {
    id: "apple-distribution-cert",
    system: "Apple Developer",
    source: "1password:Engineering/Apple Distribution",
    expiresOn: "2027-02-11",
    owner: "fabio",
    runbook: "docs/runbooks/ios-signing.md",
    blastRadius: "No iOS release can be built. Installed apps unaffected.",
  },
]

Two fields do the heavy lifting, and neither is the date.

owner is a person. Every version of this we have seen fail had "platform team" in that column, and a credential owned by everybody is renewed by nobody. blastRadius is what a colleague reads at 22:00 while deciding whether this is a tomorrow problem — writing it down once, calmly, is worth more than the twenty minutes of archaeology it replaces.

And expiresOn: null is a legitimate, documented answer. More on that below.

Make it a metric, because a calendar reminder is not monitoring

A date in a file is a document. What you want is a number that goes down, in the same system as everything else you watch, so that expiry competes for attention on equal terms with disk space and error rates. Ours goes into the metrics stack we already run as a single gauge, emitted by a CronJob that runs hourly and does nothing clever:

import { credentials } from "../ops/credentials.ts"
 
const DAY = 86_400_000
 
/**
 * Days until each credential lapses, pushed as a gauge. Negative means it
 * already has. Non-expiring entries are reported as a large constant rather
 * than omitted — a credential that stops being reported must mean the job
 * broke, not that someone deleted a row.
 */
const lines = credentials.map((c) => {
  const days = c.expiresOn
    ? Math.floor((Date.parse(c.expiresOn) - Date.now()) / DAY)
    : 3650
  const labels = `id="${c.id}",system="${c.system}",owner="${c.owner}"`
  return `credential_expiry_days{${labels}} ${days}`
})
 
await fetch(`${process.env.PUSHGATEWAY_URL}/metrics/job/credential_expiry`, {
  method: "POST",
  body: `${lines.join("\n")}\n`,
})

Then the part that makes it real, which is an alert that fires long before anyone is under pressure:

groups:
  - name: credentials
    rules:
      - alert: CredentialExpiringSoon
        expr: credential_expiry_days < 30
        for: 1h
        labels: { severity: ticket }
        annotations:
          summary: "{{ $labels.id }} expires in {{ $value }} days ({{ $labels.owner }})"
 
      - alert: CredentialExpiringUrgently
        expr: credential_expiry_days < 7
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "{{ $labels.id }} expires in {{ $value }} days — renew today"

Thirty days is not arbitrary. It is roughly the shortest notice on which a renewal that needs somebody else — a client's IT department, a certificate authority, a payment provider's support queue — can be completed without anybody rushing. Seven days is when it stops being a ticket and starts being someone's afternoon.

Where the system can tell you the date itself, do not copy it into the register — read it. Declared dates drift from real ones the first time somebody renews out of band:

SELECT rolname, rolvaliduntil
FROM pg_roles
WHERE rolvaliduntil IS NOT NULL
  AND rolvaliduntil < now() + interval '30 days';

For anything serving TLS, the blackbox exporter already gives you probe_ssl_earliest_cert_expiry for free, and it is measuring the certificate the world actually receives rather than the one you believe you deployed — which is the difference that matters when an ingress is serving a stale secret. cert-manager on the ingress makes public TLS a non-issue nine times in ten; the tenth is a DNS-01 challenge that started failing in month two of a ninety-day certificate, silently, with seventy days of runway that nobody was counting. Note that Let's Encrypt stopped sending expiry-notification e-mails in 2025, so if that was your safety net, it is gone.

A health check that never authenticates proves nothing

Readiness probes answer "is this process willing to accept traffic". They do not answer "can this process still do its job", and for anything that talks to a system you do not own, those are different questions with different answers for weeks at a time.

What closes the gap is a synthetic check that performs a real authenticated call against every external dependency, on a schedule, and reports auth failures as their own class of problem:

type Check = { id: string; run: () => Promise<void> }
 
const checks: Check[] = [
  {
    id: "odoo",
    // authenticate() is the real login path, not a ping — an unauthenticated
    // version endpoint stays healthy for months after the key is gone.
    run: async () => void (await odoo.authenticate()),
  },
  { id: "object-storage", run: () => storage.headBucket() },
  { id: "sms-gateway", run: () => sms.balance() },
]
 
export async function runChecks() {
  for (const check of checks) {
    try {
      await check.run()
      report(check.id, "ok")
    } catch (error) {
      // 401/403 is a credential problem and pages a human. A timeout is the
      // provider having a bad morning and waits for the next run.
      report(check.id, isAuthError(error) ? "auth_failed" : "unreachable")
    }
  }
}

The distinction in that catch block is the whole point. Treating every integration failure as one alert means the class that never self-heals gets buried under the class that always does, and after a fortnight of transient noise the alert is muted. An authentication failure is never transient. It should route differently, page differently, and read differently.

This is also the strongest practical argument for putting a typed API layer in front of the ERP: one service owns the credential, so there is one place to check, one place to rotate, and one place where a 401 is recognised as a 401 rather than being flattened into a generic upstream error four call-sites deep. Three applications each holding their own copy of the ERP's API surface means three renewals, and the one nobody remembers is always the one running the nightly job.

Non-expiring is a valid choice — if you pay for it elsewhere

This is the part where we disagree with the reflex. An expiry date is not free security. It is a control whose entire value depends on a renewal process, and if no such process exists, what you have bought is a guaranteed future outage in exchange for a hypothetical reduction in the useful lifetime of a leak nobody detected.

So we set expiry dates when — and only when — one of these is true:

• Renewal is automated end to end. cert-manager, workload identity, the Fulcio certificates behind keyless image signing that live for minutes. Nobody has to remember anything, so short lifetimes are pure upside.
• The credential is short-lived by nature. A presigned URL, a session token, a one-time link. Expiry is the feature.
• The renewal is already on somebody's calendar for another reason. An annual mobile release cycle drags the signing material along with it.

For everything else — a scoped machine account reading one model in an ERP, say — we set no expiry and pay for it with the controls that actually work unattended: least privilege on the account, secrets that never enter git or CI logs via Infisical and ESO, an audit trail on use, and a rehearsed revocation path so that a suspected leak is a ten-minute rotation rather than a project. The honest statement of the trade-off is that we would rather be able to kill a credential in ten minutes than have it kill itself in a year while we are not looking.

One caveat worth being blunt about: "no expiry" is a decision that has to be written down with its rationale, in the register, next to the thing. Otherwise the next engineer sees a permanent key, assumes negligence, sets a date on it to be safe — and we are back at the first paragraph of this post.

Renewal is a runbook, and the first rehearsal is the expensive one

Every renewal we have watched go badly went badly for the same reason: the first time anyone performed the procedure was under time pressure, on the day it expired, using a browser session they did not have.

So the register entry points at a runbook, and the runbook is written the first time the credential is created, when everything is fresh and nothing is urgent. It names where the account lives, who has access, what has to be updated afterwards, and — the step that is always missing — what to verify to know it worked. "Rotate the key" is not done until a synthetic check has gone green with the new value in production.

Two habits make the difference between a document and a capability.

Rehearse the ones that are hard to rehearse. iOS signing is the canonical example: a yearly cadence, a fiddly console, and total failure to ship when it lapses. Renewing it three months early costs nothing — the new certificate simply starts working — and it converts a cliff into a routine. Mobile is where this bites hardest, because an app left alone genuinely does expire in a way a web service does not.

Attach it to a ritual you already keep. We look at the expiry board in the same monthly slot as the restore drill, for the same reason: both are checks on a promise that is invisible until the day it is not kept. For clients on a care plan this sits in the plan explicitly, with the list of what is being watched — partly so it gets done, mostly so that "who renews the certificate" has an answer that was agreed before it mattered.

A note on dates themselves, since we have written about this recently: an expiry is an instant, not a wall-clock time, and the number of "it expired a day early" confusions caused by comparing a UTC expiry against a local midnight is non-trivial. Store and compare them as instants, render them in a zone you name — the same discipline every other date on the system needs.

Two things that are not credentials but behave exactly like them

DKIM keys never expire, which is precisely why nobody rotates them. There is no forcing function at all, so a selector published in 2019 is still signing mail today. Rotation is cheap when the DNS and the sending platform are things you control — publish a second selector, switch, retire the first — and it belongs on the same yearly list as everything above.

Dependency versions age like certificates. The base image with no tzdata update, the library whose fixed CVE never shipped. That one is genuinely solved by automation rather than vigilance, which is why Renovate runs on everything we host and why we would rather spend the human attention on the credentials no bot can renew.

The short version

An integration does not usually break because someone changed it. It breaks because time passed and something it depended on had a date on it.

• Write the register down — every credential, certificate and token that can stop working while the code stays still. In git, with a named human owner and a plain-language blast radius.
• Turn dates into a gauge, alert at thirty days and at seven, and read the real expiry from the system wherever the system will tell you.
• Probe with a real authenticated call, and route auth failures separately from transient ones. A health check that never uses the credential is measuring the wrong thing.
• Only set an expiry you can automate or you have diarised. Otherwise set none deliberately, write down why, and compensate with scope, audit and a fast revocation path.
• Write the runbook while it's calm, rehearse the awkward ones early, and hang the review off a ritual you already keep.

None of this is sophisticated. It is a list, a number and an alert — perhaps a day of work — and the thing it buys you is that the next credential to lapse becomes a ticket with three weeks of warning instead of six quiet weeks that end with somebody else's deadline. We build the applications that hold these integrations, run the clusters and hosting the certificates live in, and maintain the Odoo behind them — which is how we learned that the most expensive outages are the ones where nothing happened at all.